---
license: apache-2.0
base_model: micymike/codemate-qwen-1.5B-8k
tags:
- code
- coding
- qwen
- qwen2
- transformers
- text-generation
- distillation
- 32k-context
- software-engineering
- chat
pipeline_tag: text-generation
language:
- en
---

# CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5

CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5 is a fine-tuned coding assistant built on top of **CodeMate-Qwen-1.5B-8K**.

The model was further trained on converted Claude Fable 5 coding traces to improve:

- Code generation
- Code explanation
- Debugging
- Multi-turn coding conversations
- Software engineering reasoning

## Model Details

- **Base Model:** `micymike/codemate-qwen-1.5B-8k`
- **Architecture:** Qwen2 Causal LM
- **Training Method:** LoRA fine-tuning merged into full weights
- **Precision:** BF16
- **Configured Context Length:** 32,768 tokens

## Context Configuration

This model has been configured for a 32K context window using YaRN RoPE scaling.

```python
from transformers import AutoConfig

config = AutoConfig.from_pretrained(
    "micymike/CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5"
)

print(config.max_position_embeddings)
print(config.rope_scaling)
```

Current configuration:

```python
{
    "rope_type": "yarn",
    "factor": 4.0,
    "original_max_position_embeddings": 8192,
    "rope_theta": 1000000.0
}
```

Note: Long-context performance beyond the original context length should be evaluated carefully for specific workloads.

## Dataset

The model was trained on converted Claude Fable 5 coding traces formatted into OpenAI-style conversations.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "micymike/CodeMate-Qwen-1.5B-32K-Distilled-on-Claude-Fable-5"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {
        "role": "system",
        "content": "You are CodeMate, an expert programming assistant."
    },
    {
        "role": "user",
        "content": "Write a Python function to compute edit distance."
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Limitations

* Experimental research model.
* Long-context capabilities require further evaluation.
* May generate incorrect or insecure code.

## Acknowledgements

Built upon:

* Qwen2
* Transformers
* PEFT
* Hugging Face
* llama.cpp
* Claude Fable traces

## Disclaimer

This project is an independent research effort and is not affiliated with or endorsed by Anthropic, Claude, Alibaba, or Qwen.

## Author

Built by **micymike** 🇰🇪