79 lines
1.7 KiB
Markdown
79 lines
1.7 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- gpt2
|
|
- causal-lm
|
|
- ff-llm
|
|
---
|
|
|
|
# FF_3 — FF-LLM 2.02B
|
|
|
|
FF_3 is a 2.02B parameter language model trained from scratch.
|
|
|
|
## Model Details
|
|
- **Architecture**: GPT-2 decoder-only (custom)
|
|
- **Parameters**: 2,022,739,072
|
|
- **Vocabulary**: 50,257 (GPT-2 BPE tokenizer)
|
|
- **Context length**: 2,048 tokens
|
|
- **Training**: From scratch on 90B tokens
|
|
|
|
## Training Pipeline
|
|
1. **Pretraining**: 90B tokens (web + STEM data)
|
|
2. **SFT**: 760K examples + 100K high-quality examples
|
|
3. **DPO**: 38,863 preference pairs
|
|
4. **Distillation**: 20K examples from Qwen2.5-32B teacher
|
|
|
|
## Prompt Format
|
|
```
|
|
### System:
|
|
You are FF-LLM, a helpful assistant.
|
|
|
|
### Instruction:
|
|
{your question here}
|
|
|
|
### Response:
|
|
```
|
|
|
|
## Usage with Transformers
|
|
|
|
```python
|
|
from transformers import GPT2LMHeadModel, GPT2Tokenizer
|
|
|
|
model = GPT2LMHeadModel.from_pretrained("ff-llm/FF_3")
|
|
tokenizer = GPT2Tokenizer.from_pretrained("ff-llm/FF_3")
|
|
|
|
prompt = (
|
|
"### System:\nYou are FF-LLM, a helpful assistant.\n\n"
|
|
"### Instruction:\nWhat is the capital of France?\n\n### Response:\n"
|
|
)
|
|
input_ids = tokenizer.encode(prompt, return_tensors="pt")
|
|
output = model.generate(
|
|
input_ids, max_new_tokens=256, do_sample=True,
|
|
temperature=0.7, top_p=0.9,
|
|
eos_token_id=tokenizer.eos_token_id,
|
|
pad_token_id=tokenizer.eos_token_id,
|
|
)
|
|
print(tokenizer.decode(output[0][input_ids.shape[1]:], skip_special_tokens=True))
|
|
```
|
|
|
|
## Usage with Ollama
|
|
```bash
|
|
ollama run ff-llm/FF_3
|
|
```
|
|
|
|
## Limitations
|
|
- Weak mathematical reasoning
|
|
- May hallucinate on factual questions
|
|
- English only
|
|
|
|
## Training Cost
|
|
~\,000 total compute cost
|
|
Trained by a single researcher
|
|
|
|
## License
|
|
Apache 2.0
|