57 lines
1.7 KiB
Markdown
57 lines
1.7 KiB
Markdown
|
---
|
||
|
|
base_model: Qwen/Qwen3-4B-Instruct-2507
|
||
|
|
license: apache-2.0
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
tags:
|
||
|
|
- qwen3
|
||
|
|
- instruct
|
||
|
|
---
|
||
|
|
# kosa-4B-it-v1
|
||
|
|
|
||
|
|
An instruction-tuned model built on Qwen/Qwen3-4B-Instruct-2507 using the Kosa Method. Kosa Labs is a UK-based independent lab.
|
||
|
|
|
||
|
|
## Benchmarks
|
||
|
|
|
||
|
|
| Benchmark | Qwen3-4B-Instruct-2507 | kosa-4B-it-v1 |
|
||
|
|
|---|---:|---:|
|
||
|
|
| GSM8K (strict) | 73.24% | 84.23% |
|
||
|
|
| GSM8K (flexible) | 79.15% | 85.60% |
|
||
|
|
| IFEval (prompt strict) | 83.36% | 85.77% |
|
||
|
|
| IFEval (instruction strict) | 88.61% | 90.29% |
|
||
|
|
| ARC-Challenge (acc_norm) | 43.09% | 52.13% |
|
||
|
|
| MMLU | 61.89% | 65.76% |
|
||
|
|
| **Average** | **71.56%** | **77.30%** |
|
||
|
|
|
||
|
|
Both models evaluated in the same session with identical settings: lm-evaluation-harness 0.4.12, vLLM, bfloat16, temperature 0, chat template applied. Raw result files in /benchmarks. Training data verified clean against all four benchmark test sets (13-gram and 8-gram overlap checks with positive-control validation).
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
import torch
|
||
|
|
|
||
|
|
model_id = "kosa-labs/kosa-4B-it-v1"
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
||
|
|
model_id,
|
||
|
|
torch_dtype=torch.bfloat16,
|
||
|
|
device_map="auto",
|
||
|
|
)
|
||
|
|
|
||
|
|
messages = [{"role": "user", "content": "Write a concise project update."}]
|
||
|
|
inputs = tokenizer.apply_chat_template(
|
||
|
|
messages,
|
||
|
|
add_generation_prompt=True,
|
||
|
|
return_tensors="pt",
|
||
|
|
).to(model.device)
|
||
|
|
outputs = model.generate(inputs, max_new_tokens=256)
|
||
|
|
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
|
||
|
|
```
|
||
|
|
|
||
|
|
GGUF quantizations are available in /gguf:
|
||
|
|
- gguf/kosa-4B-it-v1-Q4_K_M.gguf
|
||
|
|
- gguf/kosa-4B-it-v1-Q5_K_M.gguf
|
||
|
|
- gguf/kosa-4B-it-v1-Q8_0.gguf
|