初始化项目，由ModelHub XC社区提供模型

Model: finnianx/michel-nano
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,70 @@
+---
+library_name: transformers
+tags:
+- mergekit
+- merge
+datasets:
+- HuggingFaceFW/fineweb-edu
+- epfml/FineWeb2-HQ
+- HuggingFaceTB/cosmopedia
+language:
+- en
+---
+# michel-nano
+
+michel-nano is an ultra-tiny ~6 million parameter base language model trained on 1.14 billion tokens. It was created by merging two intermediate training checkpoints using [mergekit](https://github.com/cg123/mergekit) to combine their strengths into a single, superior model.
+
+## Merge Details
+
+This model is a 50/50 SLERP (Spherical Linear Interpolation) merge of two checkpoints from the same training run:
+* **Checkpoint A (Step 60,000):** Excelled at language modeling and reasoning, achieving the best WikiText perplexity and ARC-Easy scores.
+* **Checkpoint B (Step 70,000):** Showed stronger grammatical understanding, achieving the best BLiMP score.
+
+By merging them, this model inherits the best traits of both—achieving lower WikiText perplexity and higher BLiMP/ARC-E scores than either parent checkpoint individually.
+
+## Training Details
+
+* **Parameters:** ~6,000,000
+* **Training Tokens:** 1.14 Billion
+* **Context Length:** 512 tokens
+* **Post-training:** None (This is a base pre-trained model)
+
+### Dataset Mixture
+
+| Dataset | Weight |
+| :--- | :--- |
+| `HuggingFaceFW/fineweb-edu` | 50% |
+| `epfml/FineWeb-HQ` | 30% |
+| `HuggingFaceTB/cosmopedia` (stories split) | 20% |
+
+### Tokenizer
+
+The tokenizer is a basic Byte-Pair Encoding (BPE) tokenizer trained from scratch on a subset of 100,000 samples drawn from the same data mixture. It features a compact vocabulary size of 6,000 plus chatml special tokens for future finetuning.
+
+## Evaluation
+
+Evaluated using the `lm-evaluation-harness` (0-shot).
+
+| Task | Metric | Value |
+| :--- | :--- | :--- |
+| **BLiMP (Avg)** | acc | 0.6523 |
+| ARC-Challenge | acc | 0.1732 |
+| ARC-Challenge | acc_norm | 0.2278 |
+| ARC-Easy | acc | 0.3443 |
+| ARC-Easy | acc_norm | 0.3338 |
+| BoolQ | acc | 0.3801 |
+| HellaSwag | acc | 0.2667 |
+| HellaSwag | acc_norm | 0.2708 |
+| PIQA | acc | 0.5647 |
+| PIQA | acc_norm | 0.5457 |
+| WikiText | bits_per_byte | 1.6987 |
+| WikiText | byte_perplexity | 3.2461 |
+| WikiText | word_perplexity | 542.6415 |
+| Winogrande | acc | 0.5012 |
+
+## Intended Use
+
+As a base model that has not undergone any instruction tuning or alignment, michel-nano is very limited in its raw conversational abilities. It is best suited as a lightweight foundation for fine-tuning on specific downstream tasks. Its small footprint and strong grammatical foundation make it highly adaptable for applications such as:
+* Text classification (e.g., toxic comment detection, sentiment analysis)
+* Lightweight named entity recognition (NER)
+* Random toy projects

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 3,
+  "dtype": "float32",
+  "eos_token_id": 1,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 704,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 6,
+  "num_key_value_heads": 2,
+  "pad_token_id": 2,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.10.2",
+  "use_cache": false,
+  "vocab_size": 6000
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 3,
+  "eos_token_id": [
+    1
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 2,
+  "transformers_version": "5.10.2",
+  "use_cache": true
+}

mergekit_config.yml

@@ -0,0 +1,9 @@
+models:
+  - model: /home/nvidia/pretrain/michel-nano/checkpoint-60000/
+  - model: /home/nvidia/pretrain/michel-nano/checkpoint-70000/
+merge_method: slerp
+base_model: /home/nvidia/pretrain/michel-nano/checkpoint-60000/
+parameters:
+  t:
+    - value: 0.5  # This means a 50/50 blend of the two checkpoints
+dtype: float32   # Keep float32 for the merge to preserve precision; you can quantize later

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34d3dc7d5b2dc6da1adf7f35b3e432fb5697cacb1569f384db368f87f62ea546
+size 23858096

tokenizer_config.json

@@ -0,0 +1,10 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "eos_token": "<|im_end|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 4096,
+  "pad_token": "<pad>",
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}