Quintus/docs/huggingface_model_card.md

Quintus-1.7B

Quintus-1.7B is a compact instruction-following assistant derived from Qwen/Qwen3-1.7B-Base. It was trained with online full-vocabulary knowledge distillation from a larger Qwen3-8B teacher, followed by targeted SFT for assistant behavior and generation stability.

Model Details

• Base architecture: Qwen3-1.7B
• Base checkpoint: Qwen/Qwen3-1.7B-Base
• Distillation teacher: Qwen3-8B class teacher
• Training method: Online full-vocabulary KD + targeted SFT
• Context length used in training: 4096 tokens
• Primary language focus: English
• Release repository: iamrahulreddy/Quintus
• Attention path: FlashAttention-2 when available
• Training kernels: Liger kernels for compatible Qwen-family operators
• Optimizer: fused AdamW

Intended Use

Quintus is intended for:

• General assistant use.
• Reasoning and math prompts.
• Lightweight coding assistance.
• Local experimentation with compact LLMs.
• Research into online KD and small-model alignment.

It is not intended as a safety-critical decision system. Like other compact language models, it can hallucinate and should be verified on high-stakes tasks.

Training Summary

The training pipeline has two main stages:

1. Online KD: The student learns from the teacher's dense full-vocabulary probability distribution. This avoids the sparse top-k ceiling encountered in earlier offline KD experiments.
2. SFT: The distilled checkpoint is tuned on curated instruction/persona data to improve assistant-style behavior and reduce repetition or formatting drift.

The KD loss combines assistant-token cross entropy and teacher-student KL divergence:

\mathcal{L}_{\text{total}}
= \alpha \mathcal{L}_{\text{CE}}
+ (1 - \alpha)\mathcal{L}_{\text{KD}}

For the release run, \alpha = 0.3 and T = 2.0.

torch.compile was kept disabled for the final KD path because this workload showed high Inductor memory overhead, dynamic-shape graph breaks, recompile overhead, and checkpoint portability risk from _orig_mod. state-dict prefixes when compiled modules are not unwrapped before saving.

Evaluation

Benchmark	Qwen3-1.7B-Base	Qwen3-1.7B-Instruct	Quintus-1.7B
HumanEval pass@1	67.1%	70.7%	67.7%
MBPP pass@1	67.2%	58.2%	64.8%
GSM8K, 10-shot flexible	69.98%	69.75%	74.30%
ARC-Challenge acc_norm	55.72%	52.99%	58.36%
WinoGrande, 5-shot	65.67%	61.01%	66.38%
PIQA acc_norm	75.63%	72.09%	75.57%

Strengths

• Strong math and reasoning transfer for the 1.7B parameter scale.
• Good commonsense and ARC-style benchmark performance.
• Compact enough for lower-resource deployment compared with larger teachers.
• Public weight audit indicates healthy structural divergence from the base checkpoint without collapse.

Limitations

• The model can still produce confident factual errors.
• Code generation can contradict stated complexity constraints.
• It is smaller than the teacher and inherits capacity limits of the 1.7B scale.
• Evaluation results depend on prompt format; raw and chat-template modes are not interchangeable.
• Additional preference tuning would likely improve calibration and refusal behavior.

Example Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

PUBLIC_REPO_ID = "iamrahulreddy/Quintus"

print(f"Loading Quintus from {PUBLIC_REPO_ID}...")
tokenizer = AutoTokenizer.from_pretrained(PUBLIC_REPO_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    PUBLIC_REPO_ID,
    device_map="auto",
    dtype=torch.float16,
    trust_remote_code=True,
)

stop_tokens = ["<|endoftext|>", "<|im_end|>"]
eos_token_ids = [tokenizer.eos_token_id] if tokenizer.eos_token_id is not None else []
for token in stop_tokens:
    token_id = tokenizer.convert_tokens_to_ids(token)
    if token_id is not None and token_id not in eos_token_ids:
        eos_token_ids.append(token_id)

streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

conversation_history = [
    {
        "role": "system",
        "content": (
            "You are Quintus, a highly capable AI assistant created by "
            "Muskula Rahul. You are helpful, precise, and logically sound."
        ),
    }
]

print()
print("Quintus Chat (type 'quit' to exit)")
print()

while True:
    try:
        user_input = input("You: ").strip()
        if user_input.lower() in ["quit", "exit"]:
            print("\nGoodbye!")
            break
        if not user_input:
            continue

        conversation_history.append({"role": "user", "content": user_input})

        prompt = tokenizer.apply_chat_template(
            conversation_history,
            tokenize=False,
            add_generation_prompt=True,
        )

        inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

        print("Quintus: ", end="", flush=True)

        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=512,
                temperature=0.7,
                top_p=0.9,
                do_sample=True,
                streamer=streamer,
                pad_token_id=tokenizer.eos_token_id,
                eos_token_id=eos_token_ids,
            )

        generated_ids = outputs[0][inputs.input_ids.shape[-1]:]
        assistant_response = tokenizer.decode(
            generated_ids,
            skip_special_tokens=True,
        ).strip()
        conversation_history.append({"role": "assistant", "content": assistant_response})
        print()

    except KeyboardInterrupt:
        print("\n\nGoodbye!")
        break

Credits

• Qwen Team and the Qwen Hugging Face organization for the Qwen3 model family.
• Qwen/Qwen3-8B, used as the distillation teacher.
• Qwen/Qwen3-1.7B-Base, used as the base student checkpoint.
• Qwen/Qwen3-1.7B, used for the tokenizer and chat-template contract.
• Alibaba PAI for DistilQwen_100k, the primary instruction source after filtering.
• Hugging Face Transformers, vLLM, EvalPlus, lm-evaluation-harness, FlashAttention, and Liger Kernel for training and evaluation infrastructure.

License And Author

This software is distributed under the MIT License. Refer to the repository LICENSE file for full text.

Author: Muskula Rahul - @iamrahulreddy

Citation

If you use this model or code, cite the repository and the upstream Qwen3 models.