114 lines
3.9 KiB
Markdown
114 lines
3.9 KiB
Markdown
---
|
||
base_model: meta-llama/Llama-3.2-3B-Instruct
|
||
library_name: transformers
|
||
license: llama3.2
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- sft
|
||
- scientific-reasoning
|
||
- instruction-tuning
|
||
- open-instruct
|
||
---
|
||
|
||
# Divij/Llama-3.2-3B-Instruct-sft-with-thoughts
|
||
|
||
Supervised fine-tune of [`meta-llama/Llama-3.2-3B-Instruct`](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) on a scientific-methodology
|
||
instruction dataset, where each assistant response interleaves `<Thought_i>` reasoning with `<Step_i>` actions.
|
||
|
||
The project goal is to compare whether including explicit `<Thought_i>` reasoning
|
||
traces alongside each `<Step_i>` action during SFT produces stronger scientific-methodology
|
||
generators than training on step-only plans.
|
||
|
||
## Variant
|
||
|
||
This checkpoint is the **with-thoughts** variant:
|
||
The assistant target alternates `<Thought_i>` / `<Step_i>` pairs, so the model learns to produce explicit reasoning before each action. Trained with `max_seq_length=6144` to fit the longer sequences.
|
||
|
||
## Training data
|
||
|
||
- Source: `sft_with_thoughts.jsonl` from the `verl_scientific_discovery`
|
||
repeated-sampling pipeline.
|
||
- 4,990 `messages`-format examples (`system` + `user` + `assistant`).
|
||
- Each assistant response is a step-by-step research methodology for a given
|
||
`Research Goal` + `Constraints` prompt.
|
||
|
||
## Training setup
|
||
|
||
- **Framework:** [open-instruct](https://github.com/allenai/open-instruct) `finetune.py` (accelerate + FSDP2).
|
||
- **Hardware:** 2× NVIDIA H100 NVL (96 GB).
|
||
- **Precision:** bf16 mixed precision.
|
||
- **Attention:** FlashAttention-2.
|
||
- **Memory:** gradient checkpointing enabled.
|
||
|
||
### Hyperparameters
|
||
|
||
| | |
|
||
|---|---|
|
||
| `max_seq_length` | **6144** |
|
||
| `num_train_epochs` | 3 |
|
||
| `per_device_train_batch_size` | 1 |
|
||
| `gradient_accumulation_steps` | 8 |
|
||
| Effective batch size | 16 (1 × 2 GPU × 8 accum) |
|
||
| `learning_rate` | 2e-5 |
|
||
| `lr_scheduler_type` | linear |
|
||
| `warmup_ratio` | 0.03 |
|
||
| `weight_decay` | 0.0 |
|
||
| `seed` | 42 |
|
||
| Optimizer | fused AdamW |
|
||
| Total optimization steps | 936 |
|
||
| **Final training loss** | **0.839** |
|
||
|
||
The chat template is inherited from the base model
|
||
(`meta-llama/Llama-3.2-3B-Instruct`). Labels are masked on the `system` and
|
||
`user` turns so only the assistant response contributes to the loss
|
||
(open-instruct's `sft_tulu_tokenize_and_truncate_v1` transform).
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
repo = "Divij/Llama-3.2-3B-Instruct-sft-with-thoughts"
|
||
tokenizer = AutoTokenizer.from_pretrained(repo)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
repo,
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
)
|
||
|
||
messages = [
|
||
{"role": "system", "content": "You are an expert research scientist. Produce reasoning/action pairs: <Thought_i> followed by <Step_i>."},
|
||
{"role": "user", "content": (
|
||
"You are given a scientific research problem.\n\n"
|
||
"Research Goal:\n<your research goal here>\n\n"
|
||
"Constraints:\n1) <constraint 1>\n2) <constraint 2>"
|
||
)},
|
||
]
|
||
|
||
inputs = tokenizer.apply_chat_template(
|
||
messages,
|
||
add_generation_prompt=True,
|
||
return_tensors="pt",
|
||
).to(model.device)
|
||
|
||
output = model.generate(
|
||
inputs,
|
||
max_new_tokens=1024,
|
||
do_sample=True,
|
||
temperature=0.7,
|
||
top_p=0.9,
|
||
)
|
||
print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
## Notes
|
||
|
||
- **Context length.** Use `max_seq_length` ≥ **6144** at inference time to match
|
||
the training regime; generations longer than this were not seen during training.
|
||
- **Intended use.** Research artifact for generating structured scientific research
|
||
plans. Not aligned for general-purpose chat or safety-critical use.
|
||
- **Compared to sibling.** A matching **without-thoughts** checkpoint at
|
||
[`Divij/Llama-3.2-3B-Instruct-sft-without-thoughts`](https://huggingface.co/Divij/Llama-3.2-3B-Instruct-sft-without-thoughts) is trained on
|
||
the same data but with the opposite treatment of reasoning traces.
|