Ma7ee7/SmolLM2-135M-Reasoning-5K
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: Ma7ee7/SmolLM2-135M-Reasoning-5K
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-20 14:50:11 +08:00
commit f93173c899
12 changed files with 294302 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

114
README.md Normal file
View File

@@ -0,0 +1,114 @@
---
base_model: HuggingFaceTB/SmolLM2-135M-Instruct
datasets:
- SupraLabs/reasoning-corpus-4K-5M-v1
library_name: transformers
pipeline_tag: text-generation
license: apache-2.0
language:
- en
tags:
- smollm2
- reasoning
- supervised-fine-tuning
- chat
- transformers
- safetensors
---
# SmolLM2-135M Reasoning-5K
A full-parameter reasoning fine-tune of
[`HuggingFaceTB/SmolLM2-135M-Instruct`](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct) using 5,000 examples
sampled from [`SupraLabs/reasoning-corpus-4K-5M-v1`](https://huggingface.co/datasets/SupraLabs/reasoning-corpus-4K-5M-v1).
The model was trained to place its reasoning trace inside `<think>` and
`</think>` tags, followed by a separate final answer.
## Training summary
| Setting | Value |
|---|---:|
| Training examples | 5,000 |
| Evaluation examples | 128 |
| Epochs | 2 |
| Maximum sequence length | 4,096 tokens |
| Learning rate | 3e-05 |
| Training objective | Assistant-only causal cross-entropy |
| Parameter training | Full model |
| Precision | bfloat16/float16 depending on training GPU |
The system and user portions were masked from the loss. Samples exceeding the
maximum context length were rejected instead of being cut through the middle of
a reasoning trace.
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "YOUR_USERNAME/SmolLM2-135M-Reasoning-5K"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
messages = [
{
"role": "system",
"content": 'You are a helpful AI assistant. For difficult problems, reason carefully inside <think> and </think> tags, then provide a clear final answer.',
},
{
"role": "user",
"content": "A farmer has 17 sheep. All but 9 run away. How many remain?",
},
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
).to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=256,
do_sample=False,
repetition_penalty=1.05,
)
new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=False))
```
## Reasoning format
The expected assistant format is:
```text
<think>
Internal reasoning trace
</think>
Final answer
```
This small model is experimental. It should not be assumed to produce correct
reasoning merely because it emits a structured reasoning trace.
## Files
`training_info.json` records the training configuration, any metrics found in
the local output directory, and SHA-256 hashes of the uploaded weight files.
## License
The model follows the Apache 2.0 license used by the base SmolLM2 model. Review
the base model repository and source dataset for their complete terms.

6
chat_template.jinja Normal file
View File

@@ -0,0 +1,6 @@
{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
' }}{% endif %}{{'<|im_start|>' + message['role'] + '
' + message['content'] + '<|im_end|>' + '
'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
' }}{% endif %}

38
config.json Normal file
View File

@@ -0,0 +1,38 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 1,
"dtype": "bfloat16",
"eos_token_id": 2,
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 576,
"initializer_range": 0.041666666666666664,
"intermediate_size": 1536,
"is_llama_config": true,
"max_position_embeddings": 8192,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 9,
"num_hidden_layers": 30,
"num_key_value_heads": 3,
"pad_token_id": 2,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_interleaved": false,
"rope_scaling": null,
"rope_theta": 100000,
"tie_word_embeddings": true,
"transformers.js_config": {
"kv_cache_dtype": {
"fp16": "float16",
"q4f16": "float16"
}
},
"transformers_version": "4.57.6",
"use_cache": true,
"vocab_size": 49152
}

7
generation_config.json Normal file
View File

@@ -0,0 +1,7 @@
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"pad_token_id": 2,
"transformers_version": "4.57.6"
}

48901
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5d4e7b05c25b575e1519ea785a69792f0cff31843fbaf3e1f05e213cfb3315ce
size 269060552

34
special_tokens_map.json Normal file
View File

@@ -0,0 +1,34 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"bos_token": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

244949
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

154
tokenizer_config.json Normal file
View File

@@ -0,0 +1,154 @@
{
"add_prefix_space": false,
"added_tokens_decoder": {
"0": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"3": {
"content": "<repo_name>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"4": {
"content": "<reponame>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"5": {
"content": "<file_sep>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"6": {
"content": "<filename>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"7": {
"content": "<gh_stars>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"8": {
"content": "<issue_start>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"9": {
"content": "<issue_comment>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"10": {
"content": "<issue_closed>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"11": {
"content": "<jupyter_start>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"12": {
"content": "<jupyter_text>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"13": {
"content": "<jupyter_code>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"14": {
"content": "<jupyter_output>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"15": {
"content": "<jupyter_script>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"16": {
"content": "<empty_output>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"bos_token": "<|im_start|>",
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"extra_special_tokens": {},
"model_max_length": 8192,
"pad_token": "<|im_end|>",
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>",
"vocab_size": 49152
}

60
training_info.json Normal file
View File

@@ -0,0 +1,60 @@
{
"model_name": "SmolLM2-135M-Reasoning-5K",
"base_model": "HuggingFaceTB/SmolLM2-135M-Instruct",
"dataset": "SupraLabs/reasoning-corpus-4K-5M-v1",
"training_method": "full-parameter supervised fine-tuning",
"training_rows": 5000,
"evaluation_rows": 128,
"epochs": 2,
"learning_rate": 3e-05,
"maximum_sequence_length": 4096,
"loss_masking": "assistant tokens only",
"reasoning_format": "<think>reasoning</think> followed by the final answer",
"system_prompt": "You are a helpful AI assistant. For difficult problems, reason carefully inside <think> and </think> tags, then provide a clear final answer.",
"architecture": [
"LlamaForCausalLM"
],
"model_type": "llama",
"hidden_size": 576,
"num_hidden_layers": 30,
"vocab_size": 49152,
"weight_size_bytes": 269060552,
"weight_sha256": {
"model.safetensors": "5d4e7b05c25b575e1519ea785a69792f0cff31843fbaf3e1f05e213cfb3315ce"
},
"recorded_training_metrics": {
"train_results.json": {
"epoch": 1.0,
"total_flos": 1.2703418486728704e+16,
"train_loss": 1.257179560470581,
"train_runtime": 2169.844,
"train_samples_per_second": 4.609,
"train_steps_per_second": 0.288
},
"eval_results.json": {
"epoch": 1.0,
"eval_loss": 1.272692322731018,
"eval_runtime": 7.2216,
"eval_samples_per_second": 17.725,
"eval_steps_per_second": 8.862
},
"all_results.json": {
"epoch": 1.0,
"eval_loss": 1.272692322731018,
"eval_runtime": 7.2216,
"eval_samples_per_second": 17.725,
"eval_steps_per_second": 8.862,
"total_flos": 1.2703418486728704e+16,
"train_loss": 1.257179560470581,
"train_runtime": 2169.844,
"train_samples_per_second": 4.609,
"train_steps_per_second": 0.288
},
"trainer_state_summary": {
"best_metric": 1.2715508937835693,
"best_model_checkpoint": "/content/SmolLM2-135M-Reasoning-5K/checkpoint-624",
"epoch": 1.0,
"global_step": 625
}
}
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long