初始化项目，由ModelHub XC社区提供模型

Model: jmichaelov/parc-pythia-seed0 Source: Original Platform
2026-07-24 19:22:11 +08:00
commit 635de48edc
9 changed files with 251028 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+parc_eval_plot.png filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,128 @@
+---
+license: apache-2.0
+datasets:
+- Skylion007/openwebtext
+language:
+- en
+base_model:
+- EleutherAI/pythia-160m
+library_name: transformers
+---
+
+# Parc-Pythia (Seed 0)
+
+The Parc (Parallel Architecture) models are a set of autoregressive language models (of the Pythia, Mamba, and RWKV architectures) of roughly the same size trained in parallel on the same data (2B tokens of OpenWebText) for the same number of steps, with 6 runs of training each (based on a different random seed). The Parc models were designed to allow for more direct and fine-grained analysis of training dynamics across and within architectures.
+
+## Model Details
+
+### Model Description
+
+
+- **Developed by:** James Michaelov
+- **Model type:** Pythia (autoregressive transformer)
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+
+### Model Sources
+
+#### Base Model
+
+- **Repository:** [EleutherAI/pythia-160m](https://huggingface.co/EleutherAI/pythia-160m)
+- **Paper:** [Biderman et al. (2023)](https://proceedings.mlr.press/v202/biderman23a.html)
+
+
+## Model Use
+
+The Parc models are intended for research use and generally not suitable for deployment. They are pretrained on a subset of OpenWebText, which is not well-documented, and thus it possible that they are trained on (and may generate) harmful, offensive, or otherwise inappropriate text, especially as they are not fine-tuned in any way. For the same reason, there is no guarantee that they will generate accurate or truthful text. Rather than fine-tuning our models, we instead recommend fine-tuning the original Pythia, Mamba, RWKV models, as they are trained on many times more data, and thus are likely to have substantially better performance.
+
+## How to Get Started with the Model
+
+Example code for generation:
+
+```
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained(
+  "jmichaelov/parc-pythia-seed0",
+  revision="checkpoint-4000"
+)
+
+tokenizer = AutoTokenizer.from_pretrained(
+  "jmichaelov/parc-pythia-seed0",
+  revision="checkpoint-4000"
+)
+
+inputs = tokenizer("The Parc language models", return_tensors="pt")
+tokens = model.generate(**inputs)
+tokenizer.decode(tokens[0])
+```
+
+## Training Details
+
+### Training Data
+
+* [OpenWebText](https://huggingface.co/datasets/Skylion007/openwebtext): An open replication of the WebText corpus (on which GPT-2 was trained).
+
+### Training Procedure
+
+* Context Length: 1024 tokens
+* Effective batch size: 512 (batch size * gradient accumulation)
+* Total training steps: 4000
+* Total Tokens = 4000 * 512 * 1024 = 2,097,152,000
+
+
+#### Training Hyperparameters
+* Warmup Steps: 100
+* Weight Decay: 0.1
+* Learning Rate: 6e-4
+* Learning Rate Scheduler: Cosine
+* Precision: `float32`
+
+
+## Evaluation
+
+#### Testing Data
+
+* [ARC (Easy)](https://huggingface.co/datasets/allenai/ai2_arc)
+* [BLiMP](https://huggingface.co/datasets/nyu-mll/blimp)
+* [LAMBADA (OpenAI version)](https://huggingface.co/datasets/EleutherAI/lambada_openai)
+* [SciQ](https://huggingface.co/datasets/allenai/sciq)
+* [SWAG](https://huggingface.co/datasets/allenai/swag)
+
+All evaluations were carried out using the [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness).
+
+#### Metrics
+
+* Accuracy: The standard metric for the benchmarks used.
+
+### Results
+
+![Results](https://huggingface.co/jmichaelov/parc-pythia-seed0/resolve/main/parc_eval_plot.png)
+
+
+## Environmental Impact
+
+- **Hardware Type:** GPUs: NVIDIA A100 80GB; CPUs: AMD EPYC (7713, 7643, or 7513)
+- **Hours used:** ~42hrs (on average)
+- **Infrastructure Details:** Massachusetts Green High-Performance Computing Center
+- **Carbon Emitted:** ~0.8kg (upper bound based on the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) from [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700) and the carbon efficiency of 0.0231kg/KWh reported for the data center by [Sharma et al., (2017)](https://ieeexplore.ieee.org/document/7994556/)).
+
+## Citation
+
+If you use any of the Parc models, please cite our forthcoming NeurIPS paper where we introduce them:
+
+**BibTeX:**
+
+```
+@inproceedings{michaelov_language_2025,
+    title = {Language {Model} {Behavioral} {Phases} are {Consistent} {Across} {Scale} and {Architecture}},
+    author = {Michaelov, James A. and Levy, Roger P. and Bergen, Benjamin K.},
+    booktitle = {Advances in {Neural} {Information} {Processing} {Systems}},
+    volume = {38},
+    year = {2025}
+}
+```
+
+**APA:**
+
+Michaelov, J. A., Levy, R. P., & Bergen, B. K. (2025). Language Model Behavioral Phases are Consistent Across Scale and Architecture. *Advances in Neural Information Processing Systems, 38*.
--- a/config.json
+++ b/config.json
@@ -0,0 +1,31 @@
+{
+  "architectures": [
+    "GPTNeoXForCausalLM"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "classifier_dropout": 0.1,
+  "eos_token_id": 0,
+  "hidden_act": "gelu",
+  "hidden_dropout": 0.0,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 2048,
+  "model_type": "gpt_neox",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "partial_rotary_factor": 0.25,
+  "rope_scaling": null,
+  "rope_theta": 10000,
+  "rotary_emb_base": 10000,
+  "rotary_pct": 0.25,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.51.3",
+  "use_cache": true,
+  "use_parallel_residual": true,
+  "vocab_size": 50277
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.51.3"
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53e376d4fe6ab2f176762471eba39940b20ebeb2609b806a98c4a22028819d16
+size 649142840
--- a/parc_eval_plot.png
+++ b/parc_eval_plot.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:390bab190302d675380645d57f2cb96c10947b8e0fe092d9f4f212eddb1a18d1
+size 362396
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,215 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}