初始化项目，由ModelHub XC社区提供模型
Model: iamrahulreddy/Quintus Source: Original Platform
This commit is contained in:
ModelHub XC
178
docs/huggingface_model_card.md
Normal file
178
docs/huggingface_model_card.md
Normal file
@@ -0,0 +1,178 @@
|
||||
# Quintus-1.7B
|
||||
|
||||
Quintus-1.7B is a compact instruction-following assistant derived from `Qwen/Qwen3-1.7B-Base`. It was trained with online full-vocabulary knowledge distillation from a larger Qwen3-8B teacher, followed by targeted SFT for assistant behavior and generation stability.
|
||||
|
||||
## Model Details
|
||||
|
||||
- Base architecture: Qwen3-1.7B
|
||||
- Base checkpoint: `Qwen/Qwen3-1.7B-Base`
|
||||
- Distillation teacher: Qwen3-8B class teacher
|
||||
- Training method: Online full-vocabulary KD + targeted SFT
|
||||
- Context length used in training: 4096 tokens
|
||||
- Primary language focus: English
|
||||
- Release repository: `iamrahulreddy/Quintus`
|
||||
- Attention path: FlashAttention-2 when available
|
||||
- Training kernels: Liger kernels for compatible Qwen-family operators
|
||||
- Optimizer: fused AdamW
|
||||
|
||||
## Intended Use
|
||||
|
||||
Quintus is intended for:
|
||||
|
||||
- General assistant use.
|
||||
- Reasoning and math prompts.
|
||||
- Lightweight coding assistance.
|
||||
- Local experimentation with compact LLMs.
|
||||
- Research into online KD and small-model alignment.
|
||||
|
||||
It is not intended as a safety-critical decision system. Like other compact language models, it can hallucinate and should be verified on high-stakes tasks.
|
||||
|
||||
## Training Summary
|
||||
|
||||
The training pipeline has two main stages:
|
||||
|
||||
1. Online KD: The student learns from the teacher's dense full-vocabulary probability distribution. This avoids the sparse top-k ceiling encountered in earlier offline KD experiments.
|
||||
2. SFT: The distilled checkpoint is tuned on curated instruction/persona data to improve assistant-style behavior and reduce repetition or formatting drift.
|
||||
|
||||
The KD loss combines assistant-token cross entropy and teacher-student KL divergence:
|
||||
|
||||
$$
|
||||
\mathcal{L}_{\text{total}}
|
||||
= \alpha \mathcal{L}_{\text{CE}}
|
||||
+ (1 - \alpha)\mathcal{L}_{\text{KD}}
|
||||
$$
|
||||
|
||||
For the release run, $\alpha = 0.3$ and $T = 2.0$.
|
||||
|
||||
`torch.compile` was kept disabled for the final KD path because this workload showed high Inductor memory overhead, dynamic-shape graph breaks, recompile overhead, and checkpoint portability risk from `_orig_mod.` state-dict prefixes when compiled modules are not unwrapped before saving.
|
||||
|
||||
## Evaluation
|
||||
|
||||
| Benchmark | Qwen3-1.7B-Base | Qwen3-1.7B-Instruct | Quintus-1.7B |
|
||||
| :--- | :---: | :---: | :---: |
|
||||
| HumanEval pass@1 | 67.1% | 70.7% | 67.7% |
|
||||
| MBPP pass@1 | 67.2% | 58.2% | 64.8% |
|
||||
| GSM8K, 10-shot flexible | 69.98% | 69.75% | 74.30% |
|
||||
| ARC-Challenge acc_norm | 55.72% | 52.99% | 58.36% |
|
||||
| WinoGrande, 5-shot | 65.67% | 61.01% | 66.38% |
|
||||
| PIQA acc_norm | 75.63% | 72.09% | 75.57% |
|
||||
|
||||
## Strengths
|
||||
|
||||
- Strong math and reasoning transfer for the 1.7B parameter scale.
|
||||
- Good commonsense and ARC-style benchmark performance.
|
||||
- Compact enough for lower-resource deployment compared with larger teachers.
|
||||
- Public weight audit indicates healthy structural divergence from the base checkpoint without collapse.
|
||||
|
||||
## Limitations
|
||||
|
||||
- The model can still produce confident factual errors.
|
||||
- Code generation can contradict stated complexity constraints.
|
||||
- It is smaller than the teacher and inherits capacity limits of the 1.7B scale.
|
||||
- Evaluation results depend on prompt format; raw and chat-template modes are not interchangeable.
|
||||
- Additional preference tuning would likely improve calibration and refusal behavior.
|
||||
|
||||
## Example Usage
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
|
||||
|
||||
PUBLIC_REPO_ID = "iamrahulreddy/Quintus"
|
||||
|
||||
print(f"Loading Quintus from {PUBLIC_REPO_ID}...")
|
||||
tokenizer = AutoTokenizer.from_pretrained(PUBLIC_REPO_ID, trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
PUBLIC_REPO_ID,
|
||||
device_map="auto",
|
||||
dtype=torch.float16,
|
||||
trust_remote_code=True,
|
||||
)
|
||||
|
||||
stop_tokens = ["<|endoftext|>", "<|im_end|>"]
|
||||
eos_token_ids = [tokenizer.eos_token_id] if tokenizer.eos_token_id is not None else []
|
||||
for token in stop_tokens:
|
||||
token_id = tokenizer.convert_tokens_to_ids(token)
|
||||
if token_id is not None and token_id not in eos_token_ids:
|
||||
eos_token_ids.append(token_id)
|
||||
|
||||
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
|
||||
|
||||
conversation_history = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": (
|
||||
"You are Quintus, a highly capable AI assistant created by "
|
||||
"Muskula Rahul. You are helpful, precise, and logically sound."
|
||||
),
|
||||
}
|
||||
]
|
||||
|
||||
print()
|
||||
print("Quintus Chat (type 'quit' to exit)")
|
||||
print()
|
||||
|
||||
while True:
|
||||
try:
|
||||
user_input = input("You: ").strip()
|
||||
if user_input.lower() in ["quit", "exit"]:
|
||||
print("\nGoodbye!")
|
||||
break
|
||||
if not user_input:
|
||||
continue
|
||||
|
||||
conversation_history.append({"role": "user", "content": user_input})
|
||||
|
||||
prompt = tokenizer.apply_chat_template(
|
||||
conversation_history,
|
||||
tokenize=False,
|
||||
add_generation_prompt=True,
|
||||
)
|
||||
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
|
||||
print("Quintus: ", end="", flush=True)
|
||||
|
||||
with torch.no_grad():
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=512,
|
||||
temperature=0.7,
|
||||
top_p=0.9,
|
||||
do_sample=True,
|
||||
streamer=streamer,
|
||||
pad_token_id=tokenizer.eos_token_id,
|
||||
eos_token_id=eos_token_ids,
|
||||
)
|
||||
|
||||
generated_ids = outputs[0][inputs.input_ids.shape[-1]:]
|
||||
assistant_response = tokenizer.decode(
|
||||
generated_ids,
|
||||
skip_special_tokens=True,
|
||||
).strip()
|
||||
conversation_history.append({"role": "assistant", "content": assistant_response})
|
||||
print()
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n\nGoodbye!")
|
||||
break
|
||||
```
|
||||
|
||||
## Credits
|
||||
|
||||
- [Qwen Team](https://qwenlm.github.io/) and the [Qwen Hugging Face organization](https://huggingface.co/Qwen) for the Qwen3 model family.
|
||||
- [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B), used as the distillation teacher.
|
||||
- [`Qwen/Qwen3-1.7B-Base`](https://huggingface.co/Qwen/Qwen3-1.7B-Base), used as the base student checkpoint.
|
||||
- [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B), used for the tokenizer and chat-template contract.
|
||||
- [Alibaba PAI](https://huggingface.co/alibaba-pai) for [`DistilQwen_100k`](https://huggingface.co/datasets/alibaba-pai/DistilQwen_100k), the primary instruction source after filtering.
|
||||
- [Hugging Face Transformers](https://github.com/huggingface/transformers), [vLLM](https://github.com/vllm-project/vllm), [EvalPlus](https://github.com/evalplus/evalplus), [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), [FlashAttention](https://github.com/Dao-AILab/flash-attention), and [Liger Kernel](https://github.com/linkedin/Liger-Kernel) for training and evaluation infrastructure.
|
||||
|
||||
## License And Author
|
||||
|
||||
This software is distributed under the MIT License. Refer to the repository [LICENSE](../LICENSE) file for full text.
|
||||
|
||||
Author: Muskula Rahul - [@iamrahulreddy](https://github.com/iamrahulreddy)
|
||||
|
||||
## Citation
|
||||
|
||||
If you use this model or code, cite the repository and the upstream Qwen3 models.
|
||||
Reference in New Issue
Block a user