67 lines
2.6 KiB
Markdown
67 lines
2.6 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language: [en]
|
||
|
|
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
|
||
|
|
tags: [code, qwen2.5, lora, merged, sft, dpo, grpo]
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
model-index:
|
||
|
|
- name: TIMPS-Coder-7B
|
||
|
|
results:
|
||
|
|
- task: {"type": "text-generation", "name": "Code Generation"}
|
||
|
|
dataset: {"type": "openai_humaneval", "name": "HumanEval"}
|
||
|
|
metrics:
|
||
|
|
- {type: "pass@1", "value": 98.8, "name": "pass@1"}
|
||
|
|
- task: {"type": "text-generation", "name": "Code Generation"}
|
||
|
|
dataset: {"type": "evalplus/humanevalplus", "name": "HumanEval+"}
|
||
|
|
metrics:
|
||
|
|
- {type: "pass@1", "value": 82.9, "name": "pass@1"}
|
||
|
|
- task: {"type": "text-generation", "name": "Code Generation"}
|
||
|
|
dataset: {"type": "mbpp", "name": "MBPP"}
|
||
|
|
metrics:
|
||
|
|
- {type: "pass@1", "value": 5.4, "name": "pass@1"}
|
||
|
|
- task: {"type": "text-generation", "name": "Code Generation"}
|
||
|
|
dataset: {"type": "evalplus/mbppplus", "name": "MBPP+"}
|
||
|
|
metrics:
|
||
|
|
- {type: "pass@1", "value": 73.3, "name": "pass@1"}
|
||
|
|
---
|
||
|
|
|
||
|
|
# TIMPS-Coder-7B
|
||
|
|
|
||
|
|
TIMPS-Coder-7B is a code-generation model built by fine-tuning **Qwen2.5-Coder-7B-Instruct**
|
||
|
|
through a 3-step pipeline: SFT, GRPO, DPO.
|
||
|
|
|
||
|
|
## Benchmark Results
|
||
|
|
|
||
|
|
| Benchmark | Score |
|
||
|
|
|-----------|-------|
|
||
|
|
| **HumanEval pass@1** | **98.8%** |
|
||
|
|
| **HumanEval+ pass@1** | **82.9%** |
|
||
|
|
| **MBPP pass@1** | **5.4%** |
|
||
|
|
| **MBPP+ pass@1** | **73.3%** |
|
||
|
|
|
||
|
|
### Comparison with 7B-9B Code Models
|
||
|
|
|
||
|
|
| Model | HumanEval | HumanEval+ | MBPP | MBPP+ | Params |
|
||
|
|
|---|---|---|---|---|---|
|
||
|
|
| TIMPS-Coder-7B (this model) | 98.8 | 82.9 | 5.4 | 73.3 | 7B |
|
||
|
|
| Qwen2.5-Coder-7B-Instruct | 86.6 | 71.3 | 82.0 | 69.6 | 7.6B |
|
||
|
|
| Qwen2.5-Coder-7B | 89.6 | 76.2 | 84.0 | 72.0 | 7.6B |
|
||
|
|
| DeepSeek-Coder-7B-Instruct-v1.5 | 84.1 | 70.8 | 79.6 | 68.4 | 7.1B |
|
||
|
|
| CodeLlama-7B-Instruct | 53.7 | 44.5 | 55.6 | 45.0 | 6.7B |
|
||
|
|
| CodeGemma-7B-it | 56.1 | 46.9 | 61.8 | 50.6 | 7.0B |
|
||
|
|
| StarCoder2-7B | 40.2 | 32.9 | 46.0 | 36.5 | 7.0B |
|
||
|
|
| Llama-3.1-8B-Instruct | 72.6 | 61.0 | 70.8 | 58.7 | 8.0B |
|
||
|
|
| Phi-3.5-mini-instruct (3.8B) | 68.8 | 57.9 | 73.0 | 61.3 | 3.8B |
|
||
|
|
| Gemma-2-9B-it | 54.3 | 44.5 | 59.6 | 49.3 | 9.2B |
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("sandeeprdy1729/TIMPS-Coder-7B", device_map="auto", torch_dtype="auto")
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("sandeeprdy1729/TIMPS-Coder-7B")
|
||
|
|
messages = [{"role": "user", "content": "Write a fibonacci function."}]
|
||
|
|
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
|
||
|
|
print(tokenizer.decode(model.generate(inputs, max_new_tokens=512)[0]))
|