初始化项目，由ModelHub XC社区提供模型

Model: vandijklab/pythia-160m-c2s
Source: Original Platform

.gitattributes

@@ -0,0 +1,50 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*.tfevents* filter=lfs diff=lfs merge=lfs -text
+*.db* filter=lfs diff=lfs merge=lfs -text
+*.ark* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
+ 
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.gguf* filter=lfs diff=lfs merge=lfs -text
+*.ggml filter=lfs diff=lfs merge=lfs -text
+*.llamafile* filter=lfs diff=lfs merge=lfs -text
+*.pt2 filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,149 @@
+---
+license: cc-by-4.0
+datasets:
+- vandijklab/immune-c2s
+language:
+- en
+tags:
+- pytorch
+- causal-lm
+- scRNA-seq
+---
+# Overview
+
+This is the the Pythia-160m model developed by EleutherAI fine-tuned using Cell2Sentence on *full* scRNA-seq cells.
+Cell2Sentence is a novel method for adapting large language models to single-cell transcriptomics. 
+We transform single-cell RNA sequencing data into sequences of gene names ordered by expression level, termed "cell sentences". 
+For more details, we refer to the paper linked below. 
+This model was trained on the immune tissue dataset from [Domínguez et al.](https://www.science.org/doi/10.1126/science.abl5197) 
+using 8 A100 40GB GPUs for approximately 20 hours on the following tasks:
+1. conditional cell generation
+2. unconditional cell generation
+3. cell type prediction
+
+## Cell2Sentence Links:  
+GitHub: <https://github.com/vandijklab/cell2sentence-ft> (Note: Codebase has Apache 2.0 license, weights shared on HuggingFace are CC-by-4.0)
+Paper: <https://www.biorxiv.org/content/10.1101/2023.09.11.557287v3>
+
+## Pythia Links:  
+GitHub: <https://github.com/EleutherAI/pythia>  
+Paper: <https://arxiv.org/abs/2304.01373>  
+Hugging Face: <https://huggingface.co/EleutherAI/pythia-160m>
+
+# Evaluation
+
+This model was evaluated on KNN classification and Gromov-Wasserstein (GW) distance. 
+The label for a generated cell is the corresponding cell type used in its corresponding prompt for generation. 
+Ground truth cells were sampled with replacement from a held out test dataset. 
+The generated cells are converted to expression vectors using the method described in the paper. 
+For complete details on the experiments, we refer to the paper.
+
+| Model              | k=3 NN (&#8593;) | k=5 NN (&#8593;) | k=10 NN (&#8593;) | k=25 NN (&#8593;) | GW (&#8595;) |
+| :---- | :---: | :---: | :---: | :---: | :----: |
+| scGEN              | 0.2376 | 0.2330 | 0.2377 | 0.2335 | 315.9505   | 
+| scVI               | 0.2436 | 0.2400 | 0.2425 | 0.2348 | 302.1285   |
+| scDiffusion        | 0.2335 | 0.2288 | 0.2368 | 0.2306 | 72.0208    |
+| scGPT              | 0.1838 | 0.1788 | 0.1811 | 0.1882 | 2989.8066  |
+| **C2S (Pythia-160m)**  | **0.2588** | **0.2565**  | **0.2746**      | **0.2715**    | **54.3040**         | 
+
+# Sample Code
+
+We provide an example of how to use the model to conditionally generate a cell equipped with a post-processing function to remove duplicate and invalid genes. 
+In order to generate full cells, the `max_length` generation parameter should be changed to 9200. 
+However, we recommend using an A100 GPU for inference speed and memory capacity if full cell generation is required. 
+Unconditional cell generation and cell type prediction prompts are included as well, but we do not include an example cell sentence to format the prompt. 
+We refer to the paper and GitHub repository for instructions on how to transform expression vectors into cell sentences.
+
+```
+import json
+import re
+from collections import Counter
+from typing import List
+
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+
+def post_process_generated_cell_sentences(
+    cell_sentence: str, 
+    gene_dictionary: List
+):
+    """
+    Post-processing function for generated cell sentences. 
+    Invalid genes are removed and ranks of duplicated genes are averaged.
+
+    Arguments:
+        cell_sentence:              generated cell sentence string
+        gene_dictionary:            list of gene vocabulary (all uppercase)
+
+    Returns:
+        post_processed_sentence:    generated cell sentence after post processing steps
+    """
+    generated_gene_names = cell_sentence.split(" ")
+    generated_gene_names = [generated_gene.upper() for generated_gene in generated_gene_names]
+
+    #--- Remove nonsense genes ---#
+    generated_gene_names = [gene_name for gene_name in generated_gene_names if gene_name in gene_dictionary]
+
+    #--- Average ranks ---#
+    gene_name_to_occurrences = Counter(generated_gene_names)  # get mapping of gene name --> number of occurrences
+    post_processed_sentence = generated_gene_names.copy()  # copy of generated gene list
+
+    for gene_name in gene_name_to_occurrences:
+        if gene_name_to_occurrences[gene_name] > 1 and gene_name != replace_nonsense_string:
+            # Find positions of all occurrences of duplicated generated gene in list
+            # Note: using post_processed_sentence here; since duplicates are being removed, list will be
+            #   getting shorter. Getting indices in original list will no longer be accurate positions
+            occurrence_positions = [idx for idx, elem in enumerate(post_processed_sentence) if elem == gene_name]
+            average_position = int(sum(occurrence_positions) / len(occurrence_positions))
+
+            # Remove occurrences
+            post_processed_sentence = [elem for elem in post_processed_sentence if elem != gene_name]
+
+            # Reinsert gene_name at average position
+            post_processed_sentence.insert(average_position, gene_name)
+    
+    return post_processed_sentence
+
+genes_path = "pbmc_vocab.json"
+
+with open(vocab_path, "r") as f:
+    gene_dictionary = json.load(f)
+
+model_name = "vandijklab/pythia-160m-c2s"
+
+model = AutoModelForCausalLM.from_pretrained(
+        model_name,
+        torch_dtype=torch.float16, 
+        attn_implementation="flash_attention_2"
+        ).to(torch.device("cuda"))
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+cell_type = "T Cell"
+ccg = f"Enumerate the genes in a {cell_type} cell with nonzero expression, from highest to lowest."
+
+# Prompts for other forms a generation.
+# ucg = "Display a cell's genes by expression level, in descending order."
+# cellsentence = "CELL_SENTENCE"
+# ctp = "Identify the cell type most likely associated with these highly expressed genes listed in descending order. "
+#  + cellsentence +
+#  "Name the cell type connected to these genes, ranked from highest to lowest expression."
+
+tokens = tokenizer(ccg, return_tensors='pt')
+input_ids = tokens['input_ids'].to(torch.device("cuda"))
+attention_mask = tokens['attention_mask'].to(torch.device("cuda"))
+
+with torch.no_grad():
+    outputs = model.generate(
+        input_ids=input_ids,
+        attention_mask=attention_mask,
+        do_sample=True,
+        max_length=1024,
+        top_k=50,
+        top_p=0.95,
+    )
+
+output_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+cell_sentence = "".join(re.split(r"\?|\.|:", output_text)[1:]).strip()
+processed_genes = post_process_generated_cell_sentences(cell_sentence, gene_dictionary)
+```

config.json

@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "/data/training-runs/EleutherAI/pythia-160m-2500genes-default_all-local292-2024-01-26_19-37-28/checkpoint-4000",
+  "architectures": [
+    "GPTNeoXForCausalLM"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "classifier_dropout": 0.1,
+  "eos_token_id": 0,
+  "hidden_act": "gelu",
+  "hidden_dropout": 0.0,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 9200,
+  "model_type": "gpt_neox",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "rope_scaling": null,
+  "rotary_emb_base": 10000,
+  "rotary_pct": 0.25,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.37.1",
+  "use_cache": false,
+  "use_parallel_residual": true,
+  "vocab_size": 50304
+}

configuration.json

@@ -0,0 +1 @@
+{"framework": "pytorch", "task": "text-generation", "allow_remote": true}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.37.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:533d44516b8f0a8149a87ef935266a2d77df7d860c41c296cb5252629905de44
+size 649308728

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c24618a1b3e6a38167beff1c72cffd126c3a66254347304b50547d12c5f25624
+size 2113710

tokenizer_config.json

@@ -0,0 +1,9 @@
+{
+  "add_prefix_space": false,
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "name_or_path": "EleutherAI/gpt-neox-20b",
+  "special_tokens_map_file": "/admin/home-hailey/.cache/huggingface/hub/models--EleutherAI--gpt-neox-20b/snapshots/4e49eadb5d14bd22f314ec3f45b69a87b88c7691/special_tokens_map.json",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}