101 lines
2.8 KiB
Markdown
101 lines
2.8 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
base_model:
|
||
|
|
- Qwen/Qwen3-8B-Base
|
||
|
|
datasets:
|
||
|
|
- IIGroup/X-Coder-SFT-376k
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
tags:
|
||
|
|
- code
|
||
|
|
- sft
|
||
|
|
- competitive-programming
|
||
|
|
---
|
||
|
|
|
||
|
|
# X-Coder-SFT-Qwen3-8B
|
||
|
|
|
||
|
|
X-Coder-SFT-Qwen3-8B is a code generation model fine-tuned on fully synthetic instruction data, designed for competitive programming tasks. It serves as the foundation for subsequent RLVR training.
|
||
|
|
|
||
|
|
## Model Description
|
||
|
|
|
||
|
|
- **Base Model**: [Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base)
|
||
|
|
- **Training Method**: Supervised Fine-Tuning (SFT)
|
||
|
|
- **Training Data**: [IIGroup/X-Coder-SFT-376k](https://huggingface.co/datasets/IIGroup/X-Coder-SFT-376k)
|
||
|
|
- **Parameters**: 8B
|
||
|
|
|
||
|
|
## Training
|
||
|
|
|
||
|
|
This model was trained using [ms-swift](https://github.com/modelscope/ms-swift). For training details and code, please refer to the [X-Coder GitHub repository](https://github.com/JieWu02/X-Coder).
|
||
|
|
|
||
|
|
### Training Hyperparameters
|
||
|
|
|
||
|
|
| Parameter | Value |
|
||
|
|
|-----------|-------|
|
||
|
|
| Base Model | Qwen/Qwen3-8B-Base |
|
||
|
|
| Training Type | Full Parameter |
|
||
|
|
| Epochs | 8 |
|
||
|
|
| Global Batch Size | 128 |
|
||
|
|
| Learning Rate | 5e-5 |
|
||
|
|
| Max Grad Norm | 1.0 |
|
||
|
|
| Max Length | 32768 |
|
||
|
|
| Torch Dtype | bfloat16 |
|
||
|
|
| DeepSpeed | Zero3 Offload (80GB VRAM) / Zero2 (142GB VRAM) |
|
||
|
|
| Packing | True (2x faster training, slightly worse performance) |
|
||
|
|
|
||
|
|
## Performance on LiveCodeBench v5.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## Recommended Inference Parameters
|
||
|
|
|
||
|
|
| Parameter | Value |
|
||
|
|
|-----------|-------|
|
||
|
|
| temperature | 0.6 |
|
||
|
|
| top_p | 0.95 |
|
||
|
|
| top_k | 20 (or -1 to disable) |
|
||
|
|
| max_new_tokens | 32768 |
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
model_name = "IIGroup/X-Coder-SFT-Qwen3-8B"
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
|
||
|
|
|
||
|
|
prompt = "Write a Python function to solve the two sum problem."
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||
|
|
outputs = model.generate(
|
||
|
|
**inputs,
|
||
|
|
max_new_tokens=32768,
|
||
|
|
temperature=0.6,
|
||
|
|
top_p=0.95,
|
||
|
|
top_k=20,
|
||
|
|
do_sample=True
|
||
|
|
)
|
||
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
|
|
```
|
||
|
|
|
||
|
|
## Related Models
|
||
|
|
|
||
|
|
- **RL Model**: [IIGroup/X-Coder-RL-Qwen3-8B](https://huggingface.co/IIGroup/X-Coder-RL-Qwen3-8B) - RLVR trained version achieving 64.0 on LiveCodeBench
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
|
||
|
|
```bibtex
|
||
|
|
@misc{wu2026xcoderadvancingcompetitiveprogramming,
|
||
|
|
title={X-Coder: Advancing Competitive Programming with Fully Synthetic Tasks, Solutions, and Tests},
|
||
|
|
author={Jie Wu and Haoling Li and Xin Zhang and Jiani Guo and Jane Luo and Steven Liu and Yangyu Huang and Ruihang Chu and Scarlett Li and Yujiu Yang},
|
||
|
|
year={2026},
|
||
|
|
eprint={2601.06953},
|
||
|
|
archivePrefix={arXiv},
|
||
|
|
primaryClass={cs.CL},
|
||
|
|
url={https://arxiv.org/abs/2601.06953},
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
## License
|
||
|
|
|
||
|
|
This project is licensed under the Apache License 2.0.
|