107 lines
2.8 KiB
Markdown
107 lines
2.8 KiB
Markdown
|
---
|
||
|
|
library_name: transformers
|
||
|
|
tags:
|
||
|
|
- sft
|
||
|
|
- grpo
|
||
|
|
- unsloth
|
||
|
|
- physics
|
||
|
|
license: apache-2.0
|
||
|
|
datasets:
|
||
|
|
- jilp00/YouToks-Instruct-Quantum-Physics-II
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
base_model:
|
||
|
|
- unsloth/Qwen2.5-0.5B-Instruct
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
# Model Card for HeisenbergQ-0.5B
|
||
|
|
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
HeisenbergQ-0.5B is a fine-tuned version of Qwen2.5-0.5B-Instruct, optimized for quantum physics reasoning using GRPO reinforcement learning with custom reward functions.
|
||
|
|
This model is trained to produce structured answers in XML format with <reasoning> and <answer> tags. It excels at step-by-step logical reasoning in physics-related problems.
|
||
|
|
|
||
|
|
### Model Description
|
||
|
|
|
||
|
|
|
||
|
|
- **Language(s) (NLP):** English
|
||
|
|
- **License:** MIT
|
||
|
|
- **Finetuned from model:** Qwen/Qwen2.5-0.5B-Instruct
|
||
|
|
- **Fine-Tuning Method:** GRPO with LoRA
|
||
|
|
- **Domain**: Quantum Physics
|
||
|
|
- **Dataset**: jilp00/YouToks-Instruct-Quantum-Physics-II
|
||
|
|
|
||
|
|
|
||
|
|
## Uses
|
||
|
|
|
||
|
|
### Direct Use
|
||
|
|
|
||
|
|
- Primary: Solving and reasoning through quantum physics problems
|
||
|
|
- Secondary: General scientific reasoning in math & physics
|
||
|
|
- Not for: General-purpose conversation (model is specialized)
|
||
|
|
|
||
|
|
|
||
|
|
## Bias, Risks, and Limitations
|
||
|
|
|
||
|
|
- Trained only on ~1K samples (domain-specific)
|
||
|
|
- May hallucinate outside physics domain
|
||
|
|
- Small 0.5B parameter size = lightweight, but reasoning depth is limited compared to larger models
|
||
|
|
|
||
|
|
## How to Get Started with the Model
|
||
|
|
|
||
|
|
Use the code below to get started with the model.
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
|
|
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("khazarai/HeisenbergQ-0.5B-RL")
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
||
|
|
"khazarai/HeisenbergQ-0.5B-RL",
|
||
|
|
device_map={"": 0}
|
||
|
|
)
|
||
|
|
|
||
|
|
system = """
|
||
|
|
Respond in the following format:
|
||
|
|
<reasoning>
|
||
|
|
...
|
||
|
|
</reasoning>
|
||
|
|
<answer>
|
||
|
|
...
|
||
|
|
</answer>
|
||
|
|
"""
|
||
|
|
|
||
|
|
question = """
|
||
|
|
What is the significance of setting mass equal to 1 in a quantum dynamical system, and how does it impact the formulation of the Hamiltonian and the operators?
|
||
|
|
"""
|
||
|
|
|
||
|
|
messages = [
|
||
|
|
{"role": "system", "content": system},
|
||
|
|
{"role": "user", "content": question}
|
||
|
|
]
|
||
|
|
text = tokenizer.apply_chat_template(
|
||
|
|
messages,
|
||
|
|
tokenize = False,
|
||
|
|
add_generation_prompt = True,
|
||
|
|
)
|
||
|
|
|
||
|
|
from transformers import TextStreamer
|
||
|
|
_ = model.generate(
|
||
|
|
**tokenizer(text, return_tensors = "pt").to("cuda"),
|
||
|
|
max_new_tokens = 1800,
|
||
|
|
streamer = TextStreamer(tokenizer, skip_prompt = True),
|
||
|
|
)
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
## Training Details
|
||
|
|
|
||
|
|
### Training Procedure
|
||
|
|
|
||
|
|
- Training Method: GRPO (Grouped Relative Policy Optimization)
|
||
|
|
- Reward Models: Reasoning Quality Reward: Encourages logical markers & coherent chains of thought
|
||
|
|
- Token Count Reward: Prevents under- or over-explaining
|
||
|
|
- XML Reward: Enforces <reasoning> / <answer> format
|
||
|
|
- Soft Format Reward: Ensures graceful handling of edge cases
|
||
|
|
- Steps: ~390 steps, 3 epochs
|
||
|
|
- Batch Size: 16 (with 2 generations per prompt)
|