TIMPS-Coder-7B/README.md

license, language, base_model, tags, library_name, pipeline_tag, model-index

license	language	base_model	tags	library_name	pipeline_tag	model-index
apache-2.0	en	Qwen/Qwen2.5-Coder-7B-Instruct	code qwen2.5 lora merged sft dpo grpo	transformers	text-generation	name results TIMPS-Coder-7B task dataset metrics type name text-generation Code Generation type name openai_humaneval HumanEval type value name pass@1 98.8 pass@1 task dataset metrics type name text-generation Code Generation type name evalplus/humanevalplus HumanEval+ type value name pass@1 82.9 pass@1 task dataset metrics type name text-generation Code Generation type name mbpp MBPP type value name pass@1 5.4 pass@1 task dataset metrics type name text-generation Code Generation type name evalplus/mbppplus MBPP+ type value name pass@1 73.3 pass@1

TIMPS-Coder-7B

TIMPS-Coder-7B is a code-generation model built by fine-tuning Qwen2.5-Coder-7B-Instruct through a 3-step pipeline: SFT, GRPO, DPO.

Benchmark Results

Benchmark	Score
HumanEval pass@1	98.8%
HumanEval+ pass@1	82.9%
MBPP pass@1	5.4%
MBPP+ pass@1	73.3%

Comparison with 7B-9B Code Models

Model	HumanEval	HumanEval+	MBPP	MBPP+	Params
TIMPS-Coder-7B (this model)	98.8	82.9	5.4	73.3	7B
Qwen2.5-Coder-7B-Instruct	86.6	71.3	82.0	69.6	7.6B
Qwen2.5-Coder-7B	89.6	76.2	84.0	72.0	7.6B
DeepSeek-Coder-7B-Instruct-v1.5	84.1	70.8	79.6	68.4	7.1B
CodeLlama-7B-Instruct	53.7	44.5	55.6	45.0	6.7B
CodeGemma-7B-it	56.1	46.9	61.8	50.6	7.0B
StarCoder2-7B	40.2	32.9	46.0	36.5	7.0B
Llama-3.1-8B-Instruct	72.6	61.0	70.8	58.7	8.0B
Phi-3.5-mini-instruct (3.8B)	68.8	57.9	73.0	61.3	3.8B
Gemma-2-9B-it	54.3	44.5	59.6	49.3	9.2B

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("sandeeprdy1729/TIMPS-Coder-7B", device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("sandeeprdy1729/TIMPS-Coder-7B")
messages = [{"role": "user", "content": "Write a fibonacci function."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(inputs, max_new_tokens=512)[0]))