Quintus/docs/huggingface_model_card.md

# Quintus-1.7B

Quintus-1.7B is a compact instruction-following assistant derived from `Qwen/Qwen3-1.7B-Base`. It was trained with online full-vocabulary knowledge distillation from a larger Qwen3-8B teacher, followed by targeted SFT for assistant behavior and generation stability.

## Model Details

- Base architecture: Qwen3-1.7B
- Base checkpoint: `Qwen/Qwen3-1.7B-Base`
- Distillation teacher: Qwen3-8B class teacher
- Training method: Online full-vocabulary KD + targeted SFT
- Context length used in training: 4096 tokens
- Primary language focus: English
- Release repository: `iamrahulreddy/Quintus`
- Attention path: FlashAttention-2 when available
- Training kernels: Liger kernels for compatible Qwen-family operators
- Optimizer: fused AdamW

## Intended Use

Quintus is intended for:

- General assistant use.
- Reasoning and math prompts.
- Lightweight coding assistance.
- Local experimentation with compact LLMs.
- Research into online KD and small-model alignment.

It is not intended as a safety-critical decision system. Like other compact language models, it can hallucinate and should be verified on high-stakes tasks.

## Training Summary

The training pipeline has two main stages:

1. Online KD: The student learns from the teacher's dense full-vocabulary probability distribution. This avoids the sparse top-k ceiling encountered in earlier offline KD experiments.
2. SFT: The distilled checkpoint is tuned on curated instruction/persona data to improve assistant-style behavior and reduce repetition or formatting drift.

The KD loss combines assistant-token cross entropy and teacher-student KL divergence:

$$
\mathcal{L}_{\text{total}}
= \alpha \mathcal{L}_{\text{CE}}
+ (1 - \alpha)\mathcal{L}_{\text{KD}}
$$

For the release run, $\alpha = 0.3$ and $T = 2.0$.

`torch.compile` was kept disabled for the final KD path because this workload showed high Inductor memory overhead, dynamic-shape graph breaks, recompile overhead, and checkpoint portability risk from `_orig_mod.` state-dict prefixes when compiled modules are not unwrapped before saving.

## Evaluation

| Benchmark | Qwen3-1.7B-Base | Qwen3-1.7B-Instruct | Quintus-1.7B |
| :--- | :---: | :---: | :---: |
| HumanEval pass@1 | 67.1% | 70.7% | 67.7% |
| MBPP pass@1 | 67.2% | 58.2% | 64.8% |
| GSM8K, 10-shot flexible | 69.98% | 69.75% | 74.30% |
| ARC-Challenge acc_norm | 55.72% | 52.99% | 58.36% |
| WinoGrande, 5-shot | 65.67% | 61.01% | 66.38% |
| PIQA acc_norm | 75.63% | 72.09% | 75.57% |

## Strengths

- Strong math and reasoning transfer for the 1.7B parameter scale.
- Good commonsense and ARC-style benchmark performance.
- Compact enough for lower-resource deployment compared with larger teachers.
- Public weight audit indicates healthy structural divergence from the base checkpoint without collapse.

## Limitations

- The model can still produce confident factual errors.
- Code generation can contradict stated complexity constraints.
- It is smaller than the teacher and inherits capacity limits of the 1.7B scale.
- Evaluation results depend on prompt format; raw and chat-template modes are not interchangeable.
- Additional preference tuning would likely improve calibration and refusal behavior.

## Example Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

PUBLIC_REPO_ID = "iamrahulreddy/Quintus"

print(f"Loading Quintus from {PUBLIC_REPO_ID}...")
tokenizer = AutoTokenizer.from_pretrained(PUBLIC_REPO_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    PUBLIC_REPO_ID,
    device_map="auto",
    dtype=torch.float16,
    trust_remote_code=True,
)

stop_tokens = ["<|endoftext|>", "<|im_end|>"]
eos_token_ids = [tokenizer.eos_token_id] if tokenizer.eos_token_id is not None else []
for token in stop_tokens:
    token_id = tokenizer.convert_tokens_to_ids(token)
    if token_id is not None and token_id not in eos_token_ids:
        eos_token_ids.append(token_id)

streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

conversation_history = [
    {
        "role": "system",
        "content": (
            "You are Quintus, a highly capable AI assistant created by "
            "Muskula Rahul. You are helpful, precise, and logically sound."
        ),
    }
]

print()
print("Quintus Chat (type 'quit' to exit)")
print()

while True:
    try:
        user_input = input("You: ").strip()
        if user_input.lower() in ["quit", "exit"]:
            print("\nGoodbye!")
            break
        if not user_input:
            continue

        conversation_history.append({"role": "user", "content": user_input})

        prompt = tokenizer.apply_chat_template(
            conversation_history,
            tokenize=False,
            add_generation_prompt=True,
        )

        inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

        print("Quintus: ", end="", flush=True)

        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=512,
                temperature=0.7,
                top_p=0.9,
                do_sample=True,
                streamer=streamer,
                pad_token_id=tokenizer.eos_token_id,
                eos_token_id=eos_token_ids,
            )

        generated_ids = outputs[0][inputs.input_ids.shape[-1]:]
        assistant_response = tokenizer.decode(
            generated_ids,
            skip_special_tokens=True,
        ).strip()
        conversation_history.append({"role": "assistant", "content": assistant_response})
        print()

    except KeyboardInterrupt:
        print("\n\nGoodbye!")
        break
```

## Credits

- [Qwen Team](https://qwenlm.github.io/) and the [Qwen Hugging Face organization](https://huggingface.co/Qwen) for the Qwen3 model family.
- [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B), used as the distillation teacher.
- [`Qwen/Qwen3-1.7B-Base`](https://huggingface.co/Qwen/Qwen3-1.7B-Base), used as the base student checkpoint.
- [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B), used for the tokenizer and chat-template contract.
- [Alibaba PAI](https://huggingface.co/alibaba-pai) for [`DistilQwen_100k`](https://huggingface.co/datasets/alibaba-pai/DistilQwen_100k), the primary instruction source after filtering.
- [Hugging Face Transformers](https://github.com/huggingface/transformers), [vLLM](https://github.com/vllm-project/vllm), [EvalPlus](https://github.com/evalplus/evalplus), [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), [FlashAttention](https://github.com/Dao-AILab/flash-attention), and [Liger Kernel](https://github.com/linkedin/Liger-Kernel) for training and evaluation infrastructure.

## License And Author

This software is distributed under the MIT License. Refer to the repository [LICENSE](../LICENSE) file for full text.

Author: Muskula Rahul - [@iamrahulreddy](https://github.com/iamrahulreddy)

## Citation

If you use this model or code, cite the repository and the upstream Qwen3 models.