初始化项目，由ModelHub XC社区提供模型
Model: OliverSundaram/Llama-3.2-1B-MathCodeInstruct-5k Source: Original Platform
This commit is contained in:
ModelHub XC
3
assets/mmlu_5k.png
Normal file
3
assets/mmlu_5k.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:03893630d52428741a52a22d98c5c1e4ce84337b506aff0e3f3760b813b72158
|
||||
size 441935
|
||||
138
assets/model_card.md
Normal file
138
assets/model_card.md
Normal file
@@ -0,0 +1,138 @@
|
||||
---
|
||||
license: other
|
||||
license_name: llama3.2
|
||||
license_link: https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE
|
||||
base_model: unsloth/Llama-3.2-1B
|
||||
tags:
|
||||
- math
|
||||
- fine-tuned
|
||||
- lora
|
||||
- unsloth
|
||||
- llama
|
||||
datasets:
|
||||
- MathLLMs/MathCodeInstruct
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
model-index:
|
||||
- name: Llama-3.2-1B-MathCodeInstruct-{{SIZE}}
|
||||
results:
|
||||
- task:
|
||||
type: text-generation
|
||||
name: GSM8K
|
||||
dataset:
|
||||
type: gsm8k
|
||||
name: GSM8K
|
||||
metrics:
|
||||
- type: exact_match
|
||||
value: {{GSM8K_ACC}}
|
||||
name: exact match (flexible-extract, 5-shot)
|
||||
- task:
|
||||
type: text-generation
|
||||
name: ARC-Challenge
|
||||
dataset:
|
||||
type: ai2_arc
|
||||
name: ARC-Challenge
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: {{ARC_ACC}}
|
||||
name: acc_norm (25-shot)
|
||||
- task:
|
||||
type: text-generation
|
||||
name: HellaSwag
|
||||
dataset:
|
||||
type: hellaswag
|
||||
name: HellaSwag
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: {{HELLASWAG_ACC}}
|
||||
name: acc_norm (10-shot)
|
||||
- task:
|
||||
type: text-generation
|
||||
name: WinoGrande
|
||||
dataset:
|
||||
type: winogrande
|
||||
name: WinoGrande
|
||||
metrics:
|
||||
- type: acc
|
||||
value: {{WINOGRANDE_ACC}}
|
||||
name: acc (5-shot)
|
||||
- task:
|
||||
type: text-generation
|
||||
name: MMLU
|
||||
dataset:
|
||||
type: mmlu
|
||||
name: MMLU
|
||||
metrics:
|
||||
- type: acc
|
||||
value: {{MMLU_ACC}}
|
||||
name: acc (5-shot)
|
||||
---
|
||||
|
||||
# Llama-3.2-1B-MathCodeInstruct-5k
|
||||
|
||||
A [Llama-3.2-1B](https://huggingface.co/unsloth/Llama-3.2-1B) fine-tune on **5k examples** from
|
||||
[MathLLMs/MathCodeInstruct](https://huggingface.co/datasets/MathLLMs/MathCodeInstruct), trained to solve math
|
||||
word problems with step-by-step natural-language reasoning interleaved with executable Python.
|
||||
|
||||
This is one of three sibling models trained on {5k, 10k, 20k}-example subsets of the same dataset, to study how
|
||||
fine-tuning data volume trades off against both math performance and general capability. See the
|
||||
[training write-up](https://github.com/OliverSundaram/finetuning-Llama3.2-1B) for the full comparison across all three.
|
||||
|
||||
## Training details
|
||||
|
||||
| | |
|
||||
|---|-----------------------------------------------------------------------------------------|
|
||||
| Base model | `unsloth/Llama-3.2-1B` |
|
||||
| Method | LoRA (r=16, α=16, dropout=0) on all attention + MLP projections, merged to full weights |
|
||||
| Dataset | MathLLMs/MathCodeInstruct, 5k training examples |
|
||||
| Epochs | 1 |
|
||||
| Effective batch size | 16 (batch 1 × grad. accum. 16) |
|
||||
| Learning rate | 2e-4, cosine schedule, warmup ratio 0.03 |
|
||||
| Hardware | 1× RTX 4060 (8GB) |
|
||||
| Framework | Unsloth + TRL `SFTTrainer` |
|
||||
|
||||
## Benchmark results
|
||||
|
||||
All benchmarks run with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), each at
|
||||
its standard published shot count, compared against the un-tuned base model.
|
||||
|
||||
| Benchmark | Llama-3.2-1B (base) | This model | Change |
|
||||
|---|---|---|---|
|
||||
| GSM8K | 5.8% | 7.4% | 🟢 +1.5% |
|
||||
| ARC-Challenge | 36.9% | 36.8% | ⚪ -0.1% |
|
||||
| HellaSwag | 64.2% | 63.8% | 🔴 -0.3% |
|
||||
| WinoGrande | 60.8% | 62.4% | 🟢 +1.7% |
|
||||
|
||||
**Speed** (single-request generation, greedy, RTX 4060): **38.36 tokens/sec**
|
||||
(base model: 12.74 tokens/sec)
|
||||
|
||||
### MMLU by category
|
||||
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "OliverSundaram/Llama-3.2-1B-MathCodeInstruct-5k}"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "Below is a math problem. Please solve it step by step."},
|
||||
{"role": "user", "content": "If a train travels 60 miles in 45 minutes, what is its speed in miles per hour?"},
|
||||
]
|
||||
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
|
||||
output = model.generate(inputs, max_new_tokens=512, do_sample=False)
|
||||
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- Trained on a single epoch of a 5k-example subset — not intended to be a general-purpose assistant.
|
||||
- MMLU/ARC/HellaSwag/WinoGrande scores reflect a small 1B-parameter base model and should be read relative to
|
||||
the base model's own scores, not against much larger models.
|
||||
- No safety alignment or RLHF was applied beyond what the base Llama-3.2-1B already has.
|
||||
92
assets/speed_results.json
Normal file
92
assets/speed_results.json
Normal file
@@ -0,0 +1,92 @@
|
||||
{
|
||||
"config": {
|
||||
"prompt": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Solve step by step.",
|
||||
"tokens_per_run": 256,
|
||||
"rounds": 3,
|
||||
"interleaved": true,
|
||||
"dtype": "bfloat16",
|
||||
"greedy": true,
|
||||
"device": "NVIDIA GeForce RTX 4060",
|
||||
"torch": "2.11.0+cu128"
|
||||
},
|
||||
"models": {
|
||||
"base": {
|
||||
"model_path": "unsloth/Llama-3.2-1B",
|
||||
"decode_tok_s": 40.59,
|
||||
"decode_tok_s_min": 39.28,
|
||||
"decode_tok_s_max": 40.62,
|
||||
"end_to_end_tok_s": 40.56,
|
||||
"prefill_ms": 29.3,
|
||||
"prompt_tokens": 44,
|
||||
"generated_tokens": 256,
|
||||
"rounds": [
|
||||
{
|
||||
"prompt_tokens": 44,
|
||||
"generated_tokens": 256,
|
||||
"prefill_s": 0.0321,
|
||||
"total_s": 6.5236,
|
||||
"decode_tok_s": 39.28,
|
||||
"end_to_end_tok_s": 39.24,
|
||||
"gpu": "51C / 2775MHz"
|
||||
},
|
||||
{
|
||||
"prompt_tokens": 44,
|
||||
"generated_tokens": 256,
|
||||
"prefill_s": 0.0293,
|
||||
"total_s": 6.307,
|
||||
"decode_tok_s": 40.62,
|
||||
"end_to_end_tok_s": 40.59,
|
||||
"gpu": "58C / 2775MHz"
|
||||
},
|
||||
{
|
||||
"prompt_tokens": 44,
|
||||
"generated_tokens": 256,
|
||||
"prefill_s": 0.0293,
|
||||
"total_s": 6.3119,
|
||||
"decode_tok_s": 40.59,
|
||||
"end_to_end_tok_s": 40.56,
|
||||
"gpu": "51C / 2775MHz"
|
||||
}
|
||||
]
|
||||
},
|
||||
"5k": {
|
||||
"model_path": "Llama-3.2-1B-MathCodeInstruct-5k/outputs/llama-3.2-1b-5k",
|
||||
"decode_tok_s": 40.67,
|
||||
"decode_tok_s_min": 40.47,
|
||||
"decode_tok_s_max": 40.85,
|
||||
"end_to_end_tok_s": 40.63,
|
||||
"prefill_ms": 30.0,
|
||||
"prompt_tokens": 44,
|
||||
"generated_tokens": 256,
|
||||
"rounds": [
|
||||
{
|
||||
"prompt_tokens": 44,
|
||||
"generated_tokens": 256,
|
||||
"prefill_s": 0.0302,
|
||||
"total_s": 6.3309,
|
||||
"decode_tok_s": 40.47,
|
||||
"end_to_end_tok_s": 40.44,
|
||||
"gpu": "53C / 2775MHz"
|
||||
},
|
||||
{
|
||||
"prompt_tokens": 44,
|
||||
"generated_tokens": 256,
|
||||
"prefill_s": 0.0293,
|
||||
"total_s": 6.2714,
|
||||
"decode_tok_s": 40.85,
|
||||
"end_to_end_tok_s": 40.82,
|
||||
"gpu": "53C / 2775MHz"
|
||||
},
|
||||
{
|
||||
"prompt_tokens": 44,
|
||||
"generated_tokens": 256,
|
||||
"prefill_s": 0.03,
|
||||
"total_s": 6.3007,
|
||||
"decode_tok_s": 40.67,
|
||||
"end_to_end_tok_s": 40.63,
|
||||
"gpu": "49C / 2775MHz"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
6
assets/table_5k.md
Normal file
6
assets/table_5k.md
Normal file
@@ -0,0 +1,6 @@
|
||||
| Benchmark | Llama-3.2-1B (base) | MathCodeInstruct-5k | Change |
|
||||
|---|---|---|---|
|
||||
| GSM8K | 5.8% | 7.4% | 🟢 +1.5% |
|
||||
| ARC-Challenge | 36.9% | 36.8% | ⚪ -0.1% |
|
||||
| HellaSwag | 64.2% | 63.8% | 🔴 -0.3% |
|
||||
| WinoGrande | 60.8% | 62.4% | 🟢 +1.7% |
|
||||
Reference in New Issue
Block a user