初始化项目，由ModelHub XC社区提供模型

Model: HuggingFaceTB/smollm-360M-instruct-add-basics Source: Original Platform
2026-05-16 19:13:49 +08:00
commit 2858e0f108
27 changed files with 148861 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,44 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text
+onnx/model.onnx filter=lfs diff=lfs merge=lfs -text
+onnx/model_bnb4.onnx filter=lfs diff=lfs merge=lfs -text
+onnx/model_fp16.onnx filter=lfs diff=lfs merge=lfs -text
+onnx/model_int8.onnx filter=lfs diff=lfs merge=lfs -text
+onnx/model_q4.onnx filter=lfs diff=lfs merge=lfs -text
+onnx/model_q4f16.onnx filter=lfs diff=lfs merge=lfs -text
+onnx/model_quantized.onnx filter=lfs diff=lfs merge=lfs -text
+onnx/model_uint8.onnx filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,100 @@
+---
+license: apache-2.0
+base_model: HuggingFaceTB/SmolLM-360M
+tags:
+- alignment-handbook
+- trl
+- sft
+datasets:
+- Magpie-Align/Magpie-Pro-300K-Filtered
+- bigcode/self-oss-instruct-sc2-exec-filter-50k
+- teknium/OpenHermes-2.5
+- HuggingFaceTB/everyday-conversations-llama3.1-2k
+library_name: transformers
+language:
+- en
+---
+
+
+# SmolLM-360M-Instruct
+
+<center>
+    <img src="https://huggingface.co/datasets/HuggingFaceTB/images/resolve/main/banner_smol.png" alt="SmolLM" width="1100" height="600">
+</center>
+
+
+## Model Summary
+Chat with the model at: https://huggingface.co/spaces/HuggingFaceTB/instant-smol
+
+SmolLM is a series of language models available in three sizes: 135M, 360M, and 1.7B parameters. 
+
+These models are trained on [SmolLM-Corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus), a curated collection of high-quality educational and synthetic data designed for training LLMs. For further details, we refer to our [blogpost](https://huggingface.co/blog/smollm).
+
+To build SmolLM-Instruct, we finetune the base models on publicly available datasets.
+
+## Changelog
+
+|Release|Description|
+|-|-|
+|v0.1| Initial release of SmolLM-Instruct. We finetune on the permissive subset of the WebInstructSub dataset, combined with StarCoder2-Self-OSS-Instruct. Then, we perform DPO (Direct Preference Optimization) for one epoch on HelpSteer for the 135M and 1.7B models, and argilla/dpo-mix-7k for the 360M model.|
+|v0.2| We changed the finetuning mix to datasets more suitable for smol models. We train on a new dataset of 2k simple everyday conversations we generated by llama3.1-70B [everyday-conversations-llama3.1-2k](https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k/), [Magpie-Pro-300K-Filtere](https://huggingface.co/datasets/Magpie-Align/Magpie-Pro-300K-Filtered), [self-oss-instruct-sc2-exec-filter-50k](https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k), and a small subset of [OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5)|
+
+We've noticed that the v0.2 models are better at staying on topic and responding appropriately to standard prompts, such as greetings and questions about their role as AI assistants. Additionally, SmolLM-360M-Instruct (v0.2) has a 63.3% win rate over SmolLM-360M-Instruct (v0.1) on AlpacaEval. You can find the details [here](https://huggingface.co/datasets/HuggingFaceTB/alpaca_eval_details/).
+
+## Usage
+
+### Local Applications
+⚡ For local applications, you can find optimized implementations of the model in MLC, GGUF and Transformers.js formats, in addition to fast in-browser demos in this collection: https://huggingface.co/collections/HuggingFaceTB/local-smollms-66c0f3b2a15b4eed7fb198d0
+
+We noticed that 4bit quantization degrades the quality of the 135M and 360M, so we use `q016` for MLC and ONNX/Transformers.js checkpoints for the WebGPU demos. We also suggest using temperature 0.2 and top-p 0.9.
+
+### Transformers
+```bash
+pip install transformers
+```
+
+```python
+# pip install transformers
+from transformers import AutoModelForCausalLM, AutoTokenizer
+checkpoint = "HuggingFaceTB/SmolLM-360M-Instruct"
+
+device = "cuda" # for GPU usage or "cpu" for CPU usage
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
+model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
+
+messages = [{"role": "user", "content": "What is the capital of France."}]
+input_text=tokenizer.apply_chat_template(messages, tokenize=False)
+print(input_text)
+inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
+outputs = model.generate(inputs, max_new_tokens=50, temperature=0.2, top_p=0.9, do_sample=True)
+print(tokenizer.decode(outputs[0]))
+```
+
+### Chat in TRL
+You can also use the TRL CLI to chat with the model from the terminal:
+```bash
+pip install trl
+trl chat --model_name_or_path HuggingFaceTB/SmolLM-360M-Instruct --device cpu
+```
+
+## Limitations
+
+Additionally, the generated content may not always be factually accurate, logically consistent, or free from biases present in the training data, we invite users to leverage them as assistive tools rather than definitive sources of information. We find that they can handle general knowledge questions, creative writing and basic Python programming. But they are English only and may have difficulty with arithmetics, editing tasks and complex reasoning. For more details about the models' capabilities, please refer to our [blog post](https://huggingface.co/blog/smollm).
+
+## Training parameters
+We train the models using the [alignement-handbook](https://github.com/huggingface/alignment-handbook) with the datasets mentioned in the changelog, using these parameters for v0.2:
+- 1 epoch
+- lr 1e-3
+- cosine schedule
+- warmup ratio 0.1
+- global batch size 262k tokens
+
+# Citation
+```bash
+@misc{allal2024SmolLM,
+      title={SmolLM - blazingly fast and remarkably powerful}, 
+      author={Loubna Ben Allal and Anton Lozhkov and Elie Bakouch and Leandro von Werra and Thomas Wolf},
+      year={2024},
+}
+```
--- a/all_results.json
+++ b/all_results.json
@@ -0,0 +1,14 @@
+{
+    "epoch": 0.9990828492815653,
+    "eval_loss": 1.2038620710372925,
+    "eval_runtime": 110.4149,
+    "eval_samples": 82731,
+    "eval_samples_per_second": 189.748,
+    "eval_steps_per_second": 5.932,
+    "total_flos": 80161328332800.0,
+    "train_loss": 0.9149785230034276,
+    "train_runtime": 1861.2659,
+    "train_samples": 321338,
+    "train_samples_per_second": 56.226,
+    "train_steps_per_second": 0.439
+}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "HuggingFaceTB/SmolLM-360M",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 960,
+  "initializer_range": 0.02,
+  "intermediate_size": 2560,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 15,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 5,
+  "pad_token_id": 2,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.42.3",
+  "use_cache": true,
+  "vocab_size": 49152
+}
--- a/configuration.json
+++ b/configuration.json
@@ -0,0 +1 @@
+{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
--- a/eval_results.json
+++ b/eval_results.json
@@ -0,0 +1,8 @@
+{
+    "epoch": 0.9990828492815653,
+    "eval_loss": 1.2038620710372925,
+    "eval_runtime": 110.4149,
+    "eval_samples": 82731,
+    "eval_samples_per_second": 189.748,
+    "eval_steps_per_second": 5.932
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 2,
+  "transformers_version": "4.42.3"
+}
--- a/merges.txt
+++ b/merges.txt
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f841de9bf656ea5af3c8d89e5647ee0f39121b28adf4202d1eec02f69619c5e
+size 723674912
--- a/onnx/model.onnx
+++ b/onnx/model.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:454394e1f92c1479bf71926b2cc845a3e29040c0844ba0d97ce693a390bca40c
+size 1448299818
--- a/onnx/model_bnb4.onnx
+++ b/onnx/model_bnb4.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:359ed0877d941aa2adf227aed8c209017dc0a7de6416ef322df6492356ba50e9
+size 366990971
--- a/onnx/model_fp16.onnx
+++ b/onnx/model_fp16.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e617d2597f06572ed87667f62a8a29e18bd5763fcb2606f7ad956b3f42d1cc6
+size 724678342
--- a/onnx/model_int8.onnx
+++ b/onnx/model_int8.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c42d79415353fe2d1470c2b3e27ad1dd84595cd2d31c93773d93e2df92ab5bcb
+size 363269280
--- a/onnx/model_q4.onnx
+++ b/onnx/model_q4.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8001535cd703e7c9afe6f6cf8f48e41ec6e162eb677ac39632787ad380ac2d7
+size 386650043
--- a/onnx/model_q4f16.onnx
+++ b/onnx/model_q4f16.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1a788453e1393e8642f43ca729b7f2301ba61cc1f8ac1f1904c809869fc1ffb
+size 272513495
--- a/onnx/model_quantized.onnx
+++ b/onnx/model_quantized.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c42d79415353fe2d1470c2b3e27ad1dd84595cd2d31c93773d93e2df92ab5bcb
+size 363269280
--- a/onnx/model_uint8.onnx
+++ b/onnx/model_uint8.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f3309c58be313a8dce00de2f9d009e52c273d814ddc230e0a83e7cee8075c63
+size 363269392
--- a/runs/Aug13_00-09-25_ip-26-0-165-24/events.out.tfevents.1723508228.ip-26-0-165-24.567957.0
+++ b/runs/Aug13_00-09-25_ip-26-0-165-24/events.out.tfevents.1723508228.ip-26-0-165-24.567957.0
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cbf81f9daa2cd0057156e183839b638d8fdfe7a7ac0b45ffb909c9c9e08473ff
+size 40277
--- a/runs/Aug13_00-09-25_ip-26-0-165-24/events.out.tfevents.1723510219.ip-26-0-165-24.567957.1
+++ b/runs/Aug13_00-09-25_ip-26-0-165-24/events.out.tfevents.1723510219.ip-26-0-165-24.567957.1
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43eb4fa1b53e614063d5a5a67b87a90b75e1b11a489ba487528378af12ad167d
+size 359
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,28 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": "<|im_start|>",
+  "eos_token": "<|im_end|>",
+  "pad_token": "<|im_end|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/test_prompts.py
+++ b/test_prompts.py
@@ -0,0 +1,81 @@
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+BASE_PATH = "/fsx/loubna/projects/alignment-handbook/recipes/cosmo2/sft/data"
+TEMPERATURE = 0.2
+TOP_P = 0.9
+
+CHECKPOINT = "HuggingFaceTB/smollm-350M-instruct-add-basics"
+
+print(f"💾 Loading the model and tokenizer: {CHECKPOINT}...")
+device = "cuda"
+tokenizer = AutoTokenizer.from_pretrained(CHECKPOINT)
+model_s = AutoModelForCausalLM.from_pretrained(CHECKPOINT).to(device)
+
+print("🧪 Testing single-turn conversations...")
+L = [
+    # Witing and general knowledge prompts
+    "Discuss the ethical implications of using AI in hiring processes.",
+    "Give me some tips to improve my time management skills?",
+    "Write a short dialogue between a customer and a waiter at a restaurant.",
+    "wassup?",
+    "Tell me a joke",
+    "Hi, what are some popular dishes from Japan?",
+    "What is the capital of Switzerland?",
+    "What is the capital of France?",
+    "What's the capital of Portugal?",
+    "What is the capital of Morocco?",
+    "How do I make pancakes?",
+    "Write a poem about Helium",
+    "Do you think it's important for a company to have a strong company culture? Why or why not?",
+    "What is your favorite book?",
+    "What is the most interesting fact you know?",
+    "What is your favorite movie?",
+
+    # Science prompts
+    "Can you tell me what is gravity?",
+    "Who discovered gravity?",
+    "How does a rainbow form?",
+    "What are the three states of matter?",
+    "Why is the sky blue?",
+    "What is the water cycle?",
+    "How do magnets work?",
+    "What is buoyancy?",
+    "What is the speed of light?",
+    "What's 2+2?",
+    "what's the sum of 2 and 2?",
+    "what's the sum of 2 and 3?",
+    "What is the term for the process by which plants make their own food?",
+    "If you have 8 apples and you give away 3, how many apples do you have left?",
+
+    # Python prompts
+    "How do I define a function in Python?",
+    "Can you explain what a dictionary is in Python?",
+    "Write a sort alrogithm in Python",
+    "Write a fibonacci sequence in Python",
+    "How do I read a file in Python?",
+    "How do I make everything uppercase in Python?",
+    "implement bubble sort in Python",
+
+    # Creative prompts
+    "Write a short story about a time traveler",
+    "Describe a futuristic city in three sentences",
+    "Describe a new color that doesn't exist",
+    "Create a slogan for a time machine company",
+    "Describe a world where plants can speak",
+]
+
+for i in range(len(L)):
+    print(f"🔮 {L[i]}")
+    messages = [{"role": "user", "content": L[i]}]
+    input_text = tokenizer.apply_chat_template(messages, tokenize=False)
+    inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
+    outputs = model_s.generate(
+        inputs, max_new_tokens=200, top_p=TOP_P, do_sample=True, temperature=TEMPERATURE
+    )
+    with open(
+        f"{BASE_PATH}/{CHECKPOINT.split('/')[-1]}_temp_{TEMPERATURE}_topp{TOP_P}.txt",
+        "a",
+    ) as f:
+        f.write("=" * 50 + "\n")
+        f.write(tokenizer.decode(outputs[0]))
+        f.write("\n")
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,154 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": "<|im_start|>",
+  "chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "model_max_length": 2048,
+  "pad_token": "<|im_end|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}
--- a/train_results.json
+++ b/train_results.json
@@ -0,0 +1,9 @@
+{
+    "epoch": 0.9990828492815653,
+    "total_flos": 80161328332800.0,
+    "train_loss": 0.9149785230034276,
+    "train_runtime": 1861.2659,
+    "train_samples": 321338,
+    "train_samples_per_second": 56.226,
+    "train_steps_per_second": 0.439
+}
--- a/trainer_state.json
+++ b/trainer_state.json
--- a/training_args.bin
+++ b/training_args.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2967192bec0ec77ce987842471623f5363ad8202fd752d12b15f31dc803004c7
+size 6520
--- a/vocab.json
+++ b/vocab.json
				`@@ -0,0 +1 @@`
				`{"framework": "pytorch", "task": "text-generation", "allow_remote": true}`