113 lines
5.4 KiB
Markdown
113 lines
5.4 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: Qwen/Qwen3-4B-Instruct-2507
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
language:
|
||
- en
|
||
tags:
|
||
- code
|
||
- c
|
||
- qlora
|
||
- unsloth
|
||
- text-generation-inference
|
||
---
|
||
|
||
# Qwen3-4B-C-Coder-SFT-v1
|
||
|
||
A C-language coding specialist built on **Qwen3-4B-Instruct-2507**, fine-tuned with QLoRA
|
||
(rank 32, all linear layers) on ~23k curated C instruction pairs. This is **SFT v1** — the
|
||
first stage of a larger pipeline (synthetic execution-filtered data and GRPO with a
|
||
compiler/sanitizer reward are planned follow-ups).
|
||
|
||
The headline improvement is **instruction compliance for C**: the model follows exact
|
||
function signatures and emits compilable code far more reliably than its base.
|
||
|
||
## Results
|
||
|
||
All benchmarks are execution-based: generated C is compiled with gcc and run against hidden
|
||
tests in a sandboxed container. `safe-pass@1` additionally requires zero ASan/UBSan reports.
|
||
|
||
|
||
|
||
| Benchmark | Metric | Base Qwen3-4B | **This model** | Δ |
|
||
|---|---|---|---|---|
|
||
| CEval-priv (161 tasks) | pass@1 | 36.6% | **59.0%** | **+22.4** |
|
||
| CEval-priv | compile rate | 41% | **80%** | +39 |
|
||
| McEval-C (50 tasks) | pass@1 | 52.0% | 48.0% | −4 (within ±14 pt CI) |
|
||
| McEval-C | compile rate | 72% | **90%** | +18 |
|
||
|
||
- **CEval-priv** is a private, contamination-proof eval set: 161 tasks machine-translated from
|
||
HumanEval+/MBPP+ to C with deterministic type mapping, kept only if the reference solution
|
||
compiles, passes its own tests, and runs sanitizer-clean. It was never trained on and all
|
||
training data was decontaminated against it (10-gram overlap).
|
||
- **McEval-C** is the public McEval C-generation split. The pass@1 delta is within noise at
|
||
n=50, single-sample; the compile-rate gain is the real signal.
|
||
- Sanitizer-clean pass rates equalled pass@1 for both models on both benchmarks.
|
||
|
||
**Interpretation:** one epoch of public C data (StackOverflow Q&A, curated instruction sets)
|
||
teaches *behavior* — signature compliance, compilable output — not new algorithmic ability.
|
||
Exactly what you'd expect, and what the later pipeline stages are for.
|
||
|
||
## Training
|
||
|
||
| | |
|
||
|---|---|
|
||
| Base | Qwen/Qwen3-4B-Instruct-2507 (non-thinking) |
|
||
| Method | QLoRA via Unsloth: r=32, α=64, dropout 0.05, all linear layers (66M trainable, 1.62%) |
|
||
| Data | 22,913 train / 467 valid; max seq 2048; loss on assistant tokens only |
|
||
| Schedule | 2,400 steps (≈0.84 epoch), effective batch 8, cosine LR 1e-4, warmup 100 |
|
||
| Hardware | Single Kaggle T4 (fp16), 8h33m, total compute cost $0 |
|
||
| Dynamics | train loss 1.78 → 0.97; eval loss 1.094 → 1.067, improving monotonically (no overfit) |
|
||
|
||
### Data
|
||
|
||
~34.7k cleaned pairs, sampled to 22.9k train after mixing:
|
||
|
||
| Slice | Count | Source / license |
|
||
|---|---|---|
|
||
| StackOverflow C Q&A | ~15k (capped) | [Mxode/StackOverflow-QA-C-Language-40k](https://huggingface.co/datasets/Mxode/StackOverflow-QA-C-Language-40k) — **CC BY-SA**; content © original Stack Overflow contributors, attribution per SO terms |
|
||
| Glaive code assistant v3 (C, syntax-verified) | ~3.9k | [glaiveai/glaive-code-assistant-v3](https://huggingface.co/datasets/glaiveai/glaive-code-assistant-v3) (Apache 2.0) — kept only samples whose code parses as C |
|
||
| McEval-Instruct (C) | ~1k | [Multilingual-Multimodal-NLP/McEval-Instruct](https://huggingface.co/datasets/Multilingual-Multimodal-NLP/McEval-Instruct) |
|
||
| General replay (anti-forgetting) | 15% | [allenai/tulu-3-sft-mixture](https://huggingface.co/datasets/allenai/tulu-3-sft-mixture) (ODC-BY) |
|
||
|
||
All slices were exact-deduplicated and decontaminated (word-level 10-gram overlap) against
|
||
McEval, MdEval, HumanEval(+), MBPP(+), and the private eval set.
|
||
|
||
## Usage
|
||
|
||
Non-thinking model — use Qwen's recommended sampling: `temperature=0.7, top_p=0.8, top_k=20`.
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model_id = "harshpreet931/Qwen3-4B-C-Coder-SFT-v1"
|
||
tok = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
|
||
|
||
messages = [{"role": "user", "content":
|
||
"Write a C function `int popcount32(unsigned int x)` that counts set bits."}]
|
||
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
|
||
out = model.generate(inputs, max_new_tokens=512, temperature=0.7, top_p=0.8, top_k=20)
|
||
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
Apple Silicon (4-bit MLX): see
|
||
[harshpreet931/Qwen3-4B-C-Coder-SFT-v1-mlx-4bit](https://huggingface.co/harshpreet931/Qwen3-4B-C-Coder-SFT-v1-mlx-4bit).
|
||
LoRA adapters only: [harshpreet931/Qwen3-4B-C-Coder-SFT-v1-LoRA](https://huggingface.co/harshpreet931/Qwen3-4B-C-Coder-SFT-v1-LoRA).
|
||
|
||
## Limitations
|
||
|
||
- **Algorithmic ability is unchanged from the base model.** This stage improved format/signature
|
||
compliance and compile rates, not problem-solving. Don't expect gains on hard competitive tasks.
|
||
- English-only instruction data; C17/glibc-flavored; not tuned for embedded/kernel dialects.
|
||
- Inherits base-model limitations and possible biases; generated code should be reviewed and
|
||
tested — compile-and-run verification (ideally with `-fsanitize=address,undefined`) is cheap, use it.
|
||
|
||
## Provenance
|
||
|
||
Built as part of an open, $0-compute project (MacBook M4 Pro + Kaggle free T4s): six-agent
|
||
research sweep → sandboxed compile/run/sanitizer harness → data pipeline → this SFT run.
|
||
Fun fact surfaced by the harness: one of McEval-C's own canonical solutions fails
|
||
LeakSanitizer.
|