初始化项目，由ModelHub XC社区提供模型

Model: EleutherAI/pythia-160m-data-seed3
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,170 @@
+---
+language:
+  - en
+tags:
+  - pytorch
+  - causal-lm
+  - pythia
+  - polypythias
+license: apache-2.0
+datasets:
+  - EleutherAI/pile
+  - EleutherAI/pile-preshuffled-seeds
+library_name: transformers
+arxiv: 2503.09543
+---
+
+# PolyPythias
+
+This model is part of the **PolyPythias** suite, an extension of the [Pythia](https://github.com/EleutherAI/pythia) project providing 45 additional training runs across 5 model sizes with 9 different random seeds each. These models enable systematic study of training stability and reproducibility in language models.
+
+## Paper
+
+**[PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs](https://arxiv.org/abs/2503.09543)**
+
+Oskar van der Wal, Pietro Lesci, Max Muller-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem Zuidema, and Stella Biderman. *ICLR 2025*.
+
+## Model Details
+
+| Size | Parameters | Layers | Model Dim | Heads | Original Model |
+|------|------------|--------|-----------|-------|----------------|
+| 14M  | 14M        | 6      | 128       | 4     | [pythia-14m](https://huggingface.co/EleutherAI/pythia-14m) |
+| 31M  | 31M        | 6      | 256       | 8     | [pythia-31m](https://huggingface.co/EleutherAI/pythia-31m) |
+| 70M  | 70M        | 6      | 512       | 8     | [pythia-70m](https://huggingface.co/EleutherAI/pythia-70m) |
+| 160M | 160M       | 12     | 768       | 12    | [pythia-160m](https://huggingface.co/EleutherAI/pythia-160m) |
+| 410M | 410M       | 24     | 1024      | 16    | [pythia-410m](https://huggingface.co/EleutherAI/pythia-410m) |
+
+All models were trained on 300B tokens from [The Pile](https://pile.eleuther.ai/).
+
+## Naming Convention
+
+- **`pythia-{size}m`** - Original Pythia model (seed 1234)
+- **`pythia-{size}m-seed{1-9}`** - PolyPythias variants with different random seeds
+- **`pythia-160m-data-seed{1-3}`** - 160M models with only data ordering varied (weight init fixed)
+- **`pythia-160m-weight-seed{1-3}`** - 160M models with only weight initialization varied (data order fixed)
+
+The decoupled seed variants (data-seed and weight-seed) allow researchers to separately study the effects of data ordering vs. weight initialization.
+
+## Quick Start
+
+```python
+from transformers import GPTNeoXForCausalLM, AutoTokenizer
+
+# Load the final checkpoint
+model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/pythia-70m-seed3")
+tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-70m-seed3")
+
+# Generate text
+inputs = tokenizer("The quick brown fox", return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=20)
+print(tokenizer.decode(outputs[0]))
+```
+
+## Available Checkpoints
+
+Each model provides **154 intermediate checkpoints** saved as Git branches:
+
+| Checkpoint | Training Tokens | Description |
+|------------|-----------------|-------------|
+| `step0` | 0 | Initialization (before training) |
+| `step1`, `step2`, `step4`, ..., `step512` | 2M - 1B | 10 log-spaced early checkpoints |
+| `step1000`, `step2000`, ..., `step143000` | 2B - 300B | 143 evenly-spaced checkpoints |
+
+To load a specific checkpoint:
+
+```python
+model = GPTNeoXForCausalLM.from_pretrained(
+    "EleutherAI/pythia-70m-seed3",
+    revision="step50000",  # Any checkpoint step
+)
+```
+
+## Training Data
+
+All models were trained on The Pile using pre-shuffled data orderings. The shuffled index files for each seed are available at:
+
+**[EleutherAI/pile-preshuffled-seeds](https://huggingface.co/datasets/EleutherAI/pile-preshuffled-seeds)**
+
+This dataset contains `.idx` files for seeds 0-9 used with `MMapIndexedDataset` to load the memory-mapped Pile data in the correct order for each seed.
+
+### Reproducing Training Data Order
+
+To reproduce the exact data ordering used for a specific seed:
+
+1. Download the Pile dataset and tokenize it using the Pythia tokenizer
+2. Download the corresponding seed folder from `pile-preshuffled-seeds`:
+   ```bash
+   # Using huggingface_hub
+   from huggingface_hub import snapshot_download
+   snapshot_download(
+       repo_id="EleutherAI/pile-preshuffled-seeds",
+       repo_type="dataset",
+       allow_patterns="seed3/*",  # Download only seed3
+       local_dir="./pile-seeds"
+   )
+   ```
+3. Use the idx files with GPT-NeoX's `MMapIndexedDataset`:
+   ```python
+   from dataset import MMapIndexedDataset
+   dataset = MMapIndexedDataset(path_prefix, skip_warmup=True)
+   ```
+
+For complete training reproduction instructions, see the [Pythia GitHub repository](https://github.com/EleutherAI/pythia).
+
+## All PolyPythias Models
+
+The complete collection is available at: [EleutherAI/polypythias](https://huggingface.co/collections/EleutherAI/polypythias)
+
+### 14M Parameter Models
+- [pythia-14m-seed1](https://huggingface.co/EleutherAI/pythia-14m-seed1) through [pythia-14m-seed9](https://huggingface.co/EleutherAI/pythia-14m-seed9)
+
+### 31M Parameter Models
+- [pythia-31m-seed1](https://huggingface.co/EleutherAI/pythia-31m-seed1) through [pythia-31m-seed9](https://huggingface.co/EleutherAI/pythia-31m-seed9)
+
+### 70M Parameter Models
+- [pythia-70m-seed1](https://huggingface.co/EleutherAI/pythia-70m-seed1) through [pythia-70m-seed9](https://huggingface.co/EleutherAI/pythia-70m-seed9)
+
+### 160M Parameter Models
+- [pythia-160m-seed1](https://huggingface.co/EleutherAI/pythia-160m-seed1) through [pythia-160m-seed9](https://huggingface.co/EleutherAI/pythia-160m-seed9)
+- [pythia-160m-data-seed1](https://huggingface.co/EleutherAI/pythia-160m-data-seed1) through [pythia-160m-data-seed3](https://huggingface.co/EleutherAI/pythia-160m-data-seed3)
+- [pythia-160m-weight-seed1](https://huggingface.co/EleutherAI/pythia-160m-weight-seed1) through [pythia-160m-weight-seed3](https://huggingface.co/EleutherAI/pythia-160m-weight-seed3)
+
+### 410M Parameter Models
+- [pythia-410m-seed1](https://huggingface.co/EleutherAI/pythia-410m-seed1) through [pythia-410m-seed9](https://huggingface.co/EleutherAI/pythia-410m-seed9)
+
+## Evaluation Results
+
+Evaluation results for all models are available in the [polypythias-evals](https://huggingface.co/datasets/EleutherAI/polypythias-evals) dataset.
+
+## Limitations
+
+These models are released for research purposes only. They are **not** intended for deployment in production systems.
+
+- **Not instruction-tuned**: These are base language models that predict the next token; they will not follow instructions like ChatGPT
+- **May generate harmful content**: The Pile contains diverse internet text that includes biased, offensive, and factually incorrect content
+- **English only**: Models were trained primarily on English text
+- **No safety filtering**: Outputs are not filtered for safety or accuracy
+
+## License
+
+Apache 2.0
+
+## Contact
+
+For questions about these models, please use:
+- [EleutherAI Discord](https://discord.gg/eleutherai) - #release-discussion channel
+- [GitHub Issues](https://github.com/EleutherAI/pythia/issues)
+
+## Citation
+
+If you use these models, please cite:
+
+```bibtex
+@inproceedings{vanderwal2025polypythias,
+    title={PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs},
+    author={van der Wal, Oskar and Lesci, Pietro and Muller-Eberstein, Max and Saphra, Naomi and Schoelkopf, Hailey and Zuidema, Willem and Biderman, Stella},
+    booktitle={International Conference on Learning Representations},
+    year={2025},
+    url={https://arxiv.org/abs/2503.09543}
+}
+```

config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "GPTNeoXForCausalLM"
+  ],
+  "bos_token_id": 0,
+  "classifier_dropout": 0.1,
+  "eos_token_id": 0,
+  "hidden_act": "gelu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 2048,
+  "model_type": "gpt_neox",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "rotary_emb_base": 10000,
+  "rotary_pct": 0.25,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.30.2",
+  "use_cache": true,
+  "use_parallel_residual": true,
+  "vocab_size": 50304
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.30.2"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f5cf48b0d83b05ee1e06f67421088b42dbdc2c17854b394184788eedae41db5
+size 375036845

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "add_prefix_space": false,
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}