ef53f4cc43
Model: micymike/CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5 Source: Original Platform
142 lines
2.9 KiB
Markdown
142 lines
2.9 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: micymike/codemate-qwen-1.5B-8k
|
|
tags:
|
|
- code
|
|
- coding
|
|
- qwen
|
|
- qwen2
|
|
- transformers
|
|
- text-generation
|
|
- distillation
|
|
- 32k-context
|
|
- software-engineering
|
|
- chat
|
|
pipeline_tag: text-generation
|
|
language:
|
|
- en
|
|
---
|
|
|
|
# CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5
|
|
|
|
CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5 is a fine-tuned coding assistant built on top of **CodeMate-Qwen-1.5B-8K**.
|
|
|
|
The model was further trained on converted Claude Fable 5 coding traces to improve:
|
|
|
|
- Code generation
|
|
- Code explanation
|
|
- Debugging
|
|
- Multi-turn coding conversations
|
|
- Software engineering reasoning
|
|
|
|
## Model Details
|
|
|
|
- **Base Model:** `micymike/codemate-qwen-1.5B-8k`
|
|
- **Architecture:** Qwen2 Causal LM
|
|
- **Training Method:** LoRA fine-tuning merged into full weights
|
|
- **Precision:** BF16
|
|
- **Configured Context Length:** 32,768 tokens
|
|
|
|
## Context Configuration
|
|
|
|
This model has been configured for a 32K context window using YaRN RoPE scaling.
|
|
|
|
```python
|
|
from transformers import AutoConfig
|
|
|
|
config = AutoConfig.from_pretrained(
|
|
"micymike/CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5"
|
|
)
|
|
|
|
print(config.max_position_embeddings)
|
|
print(config.rope_scaling)
|
|
```
|
|
|
|
Current configuration:
|
|
|
|
```python
|
|
{
|
|
"rope_type": "yarn",
|
|
"factor": 4.0,
|
|
"original_max_position_embeddings": 8192,
|
|
"rope_theta": 1000000.0
|
|
}
|
|
```
|
|
|
|
Note: Long-context performance beyond the original context length should be evaluated carefully for specific workloads.
|
|
|
|
## Dataset
|
|
|
|
The model was trained on converted Claude Fable 5 coding traces formatted into OpenAI-style conversations.
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
import torch
|
|
|
|
model_id = "micymike/CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_id,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto"
|
|
)
|
|
|
|
messages = [
|
|
{
|
|
"role": "system",
|
|
"content": "You are CodeMate, an expert programming assistant."
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "Write a Python function to compute edit distance."
|
|
}
|
|
]
|
|
|
|
prompt = tokenizer.apply_chat_template(
|
|
messages,
|
|
tokenize=False,
|
|
add_generation_prompt=True,
|
|
)
|
|
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=512,
|
|
temperature=0.7,
|
|
top_p=0.9,
|
|
do_sample=True
|
|
)
|
|
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
## Limitations
|
|
|
|
* Experimental research model.
|
|
* Long-context capabilities require further evaluation.
|
|
* May generate incorrect or insecure code.
|
|
|
|
## Acknowledgements
|
|
|
|
Built upon:
|
|
|
|
* Qwen2
|
|
* Transformers
|
|
* PEFT
|
|
* Hugging Face
|
|
* llama.cpp
|
|
* Claude Fable traces
|
|
|
|
## Disclaimer
|
|
|
|
This project is an independent research effort and is not affiliated with or endorsed by Anthropic, Claude, Alibaba, or Qwen.
|
|
|
|
## Author
|
|
|
|
Built by **micymike** 🇰🇪
|