117 lines
4.7 KiB
Markdown
117 lines
4.7 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
license: mit
|
||
base_model: Qwen/Qwen2.5-Coder-1.5B
|
||
library_name: transformers
|
||
model_type: qwen2
|
||
tags:
|
||
- code
|
||
- hallucination-reduction
|
||
- heartly
|
||
- decide-verify-stop
|
||
- boundary-head
|
||
- pytorch
|
||
- text-generation
|
||
---
|
||
|
||
# Heartly Qwen-Code v3
|
||
|
||
A 1.5B coding LLM with the **Heartly** hallucination-reduction architecture,
|
||
fine-tuned from **Qwen2.5-Coder-1.5B** with the conversational Stage-5 SFT recipe
|
||
(Fix1–4: natural phrasing, single refusal, persona — 5,200 samples in
|
||
`heartly-qwen-code/sft_dataset_code_v3.jsonl`).
|
||
|
||
v3 builds on the same v1/v2 Stage 1–4 numbers (grammar adoption 100%, boundary-head
|
||
AUROC 1.000, critic AUROC 1.000) — same Qwen2.5-Coder-1.5B base, now trained for
|
||
multi-turn conversational code chat. See [`HF_MODEL_CARD.md`](HF_MODEL_CARD.md) for
|
||
the Stage 1–2 probe/critic results carried over from the identical architecture.
|
||
|
||
## Output grammar
|
||
|
||
```
|
||
thinking [reasoning] response<decide>speak|stop</decide><verify>known|unknown</verify> [answer] <stop>
|
||
```
|
||
|
||
Only `[answer]` should reach the user.
|
||
|
||
## Usage
|
||
|
||
### 0. Recommended — chat via the GitHub server (strips the grammar for you)
|
||
|
||
This model emits the Heartly grammar as ordinary multi-token text (the tags are
|
||
**not** tokenizer special tokens), so some front-ends (e.g. LM Studio) may decode
|
||
them mangled. The **`server.py`** FastAPI loader on GitHub loads this model and
|
||
runs every reply through **`reply_formatter.py`**, which canonicalises the tags
|
||
and returns only the clean answer.
|
||
|
||
```bash
|
||
pip install -r requirements.txt # fastapi + uvicorn + transformers + torch
|
||
python server.py --model eivintobias/heartly-qwen-code --port 8000
|
||
|
||
curl -X POST http://127.0.0.1:8000/chat \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"prompt":"Write a function that reverses a string"}'
|
||
```
|
||
|
||
Response: `{"model":"eivintobias/heartly-qwen-code","raw":"...<decide>...","reply":"<clean answer>"}`.
|
||
|
||
Quick browser test (no curl): open `http://127.0.0.1:8000/` — `server.py` serves an
|
||
HTML chat UI at `GET /`. The first message lazy-loads the model; code answers render
|
||
with real line breaks, and the Heartly grammar is stripped by the reply formatter.
|
||
|
||
Quick offline test (no server): `python chat_smoke.py "Write a function that sorts a list"`.
|
||
|
||
📦 **Model card source:** this file (`HF_MODEL_CARD_v3.md`). When uploaded to
|
||
HuggingFace, copy it to `README.md` on the hub repo.
|
||
|
||
### 1. Transformers
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
tok = AutoTokenizer.from_pretrained("eivintobias/heartly-qwen-code")
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
"eivintobias/heartly-qwen-code", torch_dtype=torch.float32, device_map="cpu"
|
||
)
|
||
model.eval()
|
||
ids = tok.encode("User: Write a function that reverses a string\nAssistant: ", return_tensors="pt")
|
||
out = model.generate(**ids, max_new_tokens=256, pad_token_id=tok.eos_token_id, do_sample=False)
|
||
raw = tok.decode(out[0][ids.shape[1]:], skip_special_tokens=False)
|
||
# Strip the grammar -> clean answer:
|
||
from reply_formatter import format_reply
|
||
print(format_reply(raw))
|
||
```
|
||
|
||
> `reply_formatter.py` (grammar strip) and `server.py` are bundled in this repo (HF clone = flat layout; GitHub = `heartly-qwen-code/`). Clone it so `from reply_formatter import format_reply` resolves before the offline example.
|
||
|
||
## Files in this repo
|
||
|
||
| File | Description |
|
||
|------|-------------|
|
||
| `config.json` | Qwen2ForCausalLM (28 layers, d=1536) + `heartly_stop_token_id=9495` |
|
||
| `generation_config.json` | default generate params |
|
||
| `chat_template.jinja` | standard Qwen chat template |
|
||
| `tokenizer.json` / `tokenizer_config.json` | Qwen BPE tokenizer |
|
||
| `model.safetensors` | v3 fine-tuned weights (**full fine-tune**, not a LoRA adapter) |
|
||
| `server.py` | FastAPI server: lazy-loads the model; serves `/`, `/health`, `/chat`; strips Heartly grammar via reply_formatter |
|
||
| `reply_formatter.py` | strips `thinking` / `<decide>` / `<verify>` / `<stop>` -> clean answer; unescapes code newlines |
|
||
| `chat_smoke.py` | offline load + chat smoke test (no server) |
|
||
| `requirements.txt` | torch, transformers, fastapi, uvicorn, sentencepiece, datasets, scikit-learn, numpy, accelerate, huggingface_hub |
|
||
|
||
## Training
|
||
|
||
- **Base:** Qwen/Qwen2.5-Coder-1.5B
|
||
- **Method:** full fine-tune (fp16), max-length 512, 2 epochs, freeze bottom 12 layers
|
||
- **Dataset:** `sft_dataset_code_v3.jsonl` (5,200 conversational Heartly samples)
|
||
- **GPU:** 1× RTX 3090 (24GB)
|
||
|
||
## License
|
||
|
||
MIT — built on Qwen2.5-Coder (Apache 2.0).
|
||
|
||
## Links
|
||
|
||
- [GitHub (server + tools)](https://github.com/eivintobias/heartly/tree/master/heartly-qwen-code)
|
||
- [v1/v2 model card (LoRA)](HF_MODEL_CARD.md)
|
||
- [Heartly RWKV7 model](https://huggingface.co/eivintobias/heartly-rwkv7-1.5b)
|