初始化项目，由ModelHub XC社区提供模型
Model: iamrahulreddy/Quintus Source: Original Platform
This commit is contained in:
ModelHub XC
438
README.md
Normal file
438
README.md
Normal file
@@ -0,0 +1,438 @@
|
||||
---
|
||||
license: mit
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
base_model: Qwen/Qwen3-1.7B-Base
|
||||
base_model_relation: finetune
|
||||
datasets:
|
||||
- alibaba-pai/DistilQwen_100k
|
||||
metrics:
|
||||
- accuracy
|
||||
- exact_match
|
||||
- code_eval
|
||||
tags:
|
||||
- qwen3
|
||||
- qwen
|
||||
- qwen3-1.7b
|
||||
- qwen3-8b
|
||||
- quintus
|
||||
- quintus-1.7b
|
||||
- causal-lm
|
||||
- text-generation
|
||||
- language-model
|
||||
- chat
|
||||
- assistant
|
||||
- compact-llm
|
||||
- small-language-model
|
||||
- knowledge-distillation
|
||||
- online-kd
|
||||
- full-vocabulary-kd
|
||||
- supervised-fine-tuning
|
||||
- sft
|
||||
- reasoning
|
||||
- code-generation
|
||||
- english
|
||||
- pytorch
|
||||
- transformers
|
||||
- vllm
|
||||
widget:
|
||||
- text: "Explain knowledge distillation in simple terms."
|
||||
- text: "Solve this step by step: If a train travels 180 km in 3 hours, what is its average speed?"
|
||||
---
|
||||
|
||||
# Quintus
|
||||
|
||||
[](https://colab.research.google.com/drive/1TdMSN5HzD1mToCFVf_qQoj10NGZLy2V0?usp=sharing)
|
||||
[](https://huggingface.co/iamrahulreddy/Quintus)
|
||||
[](docs/index.md)
|
||||
[](docs/benchmarks.md)
|
||||
[](LICENSE)
|
||||
[](https://huggingface.co/Qwen/Qwen3-1.7B-Base)
|
||||
[](https://huggingface.co/Qwen/Qwen3-8B)
|
||||
|
||||
**Quintus-1.7B** is a compact English-focused assistant built from
|
||||
`Qwen/Qwen3-1.7B-Base`. The project uses **online full-vocabulary knowledge
|
||||
distillation** from a `Qwen/Qwen3-8B` teacher, followed by a targeted SFT stage
|
||||
for assistant behavior, identity grounding, and generation stability.
|
||||
|
||||
Final model weights:
|
||||
[iamrahulreddy/Quintus](https://huggingface.co/iamrahulreddy/Quintus)
|
||||
|
||||
## Core Technical Points
|
||||
|
||||
- **Dense KD signal:** the final training path streams the teacher's full
|
||||
vocabulary distribution live instead of relying on sparse cached top-k logits.
|
||||
- **Base-student strategy:** the student starts from `Qwen/Qwen3-1.7B-Base`,
|
||||
leaving more room for distillation before assistant-format tuning.
|
||||
- **Assistant-only supervision:** prompt text, chat headers, separators, and
|
||||
padding are masked out of the supervised target region.
|
||||
- **Sequence packing:** deterministic first-fit decreasing packing improves
|
||||
useful-token throughput at 4096-token context length.
|
||||
- **Public benchmark controls:** raw/chat prompt format, metric extraction,
|
||||
generation budget, and artifact hygiene are documented explicitly.
|
||||
|
||||
## Training Summary
|
||||
|
||||
The release training path is a two-stage pipeline:
|
||||
|
||||
1. **Online KD:** train the 1.7B base student against live teacher logits from a
|
||||
Qwen3-8B teacher.
|
||||
2. **Targeted SFT:** tune the distilled checkpoint for assistant-style
|
||||
interaction, persona consistency, and repetition control.
|
||||
|
||||
## Reuse As A KD Framework
|
||||
|
||||
Quintus is released as a trained 1.7B assistant, but the repository is also a
|
||||
reusable reference pipeline for compact-model distillation. The same structure
|
||||
can be adapted to other teacher/student pairs with changes to the model IDs,
|
||||
tokenizer, dataset source, local paths, sequence length, batch schedule, and
|
||||
hardware-specific memory settings in [configs/config.yaml](configs/config.yaml).
|
||||
|
||||
The reusable pieces are split across the codebase: assistant-only masking,
|
||||
sequence packing, online full-vocabulary KD loss, checkpoint/resume metadata,
|
||||
validation, provenance checks, SFT, and evaluation. The final pattern is:
|
||||
|
||||
1. Distill a smaller base student from a stronger teacher with online KD.
|
||||
2. Apply targeted SFT to recover assistant behavior, formatting, identity, and
|
||||
generation stability.
|
||||
|
||||
|
||||
|
||||
Core KD objective:
|
||||
|
||||
$$
|
||||
\mathcal{L}_{\text{total}}
|
||||
= \alpha \mathcal{L}_{\text{CE}}
|
||||
+ (1 - \alpha)\mathcal{L}_{\text{KD}}
|
||||
$$
|
||||
|
||||
For the final run,
|
||||
|
||||
$$
|
||||
\alpha = 0.3,\quad T = 2.0
|
||||
$$
|
||||
|
||||
Configuration snapshot:
|
||||
|
||||
| Setting | Value |
|
||||
| :--- | :--- |
|
||||
| Teacher | `Qwen/Qwen3-8B` |
|
||||
| Student | `Qwen/Qwen3-1.7B-Base` |
|
||||
| Tokenizer | `Qwen/Qwen3-1.7B` |
|
||||
| Data | ~90K English-only samples from DistilQwen_100k |
|
||||
| Max sequence length | 4096 |
|
||||
| Epochs | 1 |
|
||||
| Learning rate | `5.0e-6` |
|
||||
| Weight decay | `0.1` |
|
||||
| Warmup ratio | `0.05` |
|
||||
| Online KD token chunk | 2048 |
|
||||
| Micro batch | 4 |
|
||||
| Gradient accumulation | 2 |
|
||||
| Sequence packing | enabled, `pack_length = 4096` |
|
||||
| Attention | FlashAttention-2 when available |
|
||||
| Liger kernels | enabled for compatible Qwen-family ops |
|
||||
| Optimizer | fused AdamW |
|
||||
| `torch.compile` | disabled |
|
||||
| Gradient checkpointing | disabled |
|
||||
| Seed | 25 |
|
||||
|
||||
> [!NOTE]
|
||||
> FlashAttention-2, Liger kernels, and fused AdamW are acceleration paths. Keep
|
||||
> the baseline load path compatible with standard Transformers and vLLM APIs
|
||||
> before publishing checkpoints. `torch.compile` stayed disabled because this
|
||||
> KD shape showed high Inductor memory overhead, dynamic-shape graph breaks,
|
||||
> recompile overhead, and checkpoint portability risk from `_orig_mod.` state
|
||||
> dict prefixes when compiled modules are not unwrapped before saving.
|
||||
|
||||
> [!TIP]
|
||||
> The B200-oriented defaults are conservative for the 8B teacher to 1.7B
|
||||
> student workload. Smaller teacher/student pairs may tolerate larger
|
||||
> micro-batches, but full-vocabulary KD scales sharply with vocabulary width.
|
||||
|
||||
The editable run configuration lives in [configs/config.yaml](configs/config.yaml).
|
||||
Paths and Hub destinations are left as placeholders so each runner can set local
|
||||
directories and repository names directly.
|
||||
|
||||
## Why Online KD Replaced Offline Top-K KD
|
||||
|
||||
Earlier experiments cached only the teacher's top-k logits. That made storage
|
||||
smaller, but with a Qwen vocabulary around 151K tokens, $k = 8$ exposes only:
|
||||
|
||||
$$
|
||||
\frac{k}{|V|}
|
||||
= \frac{8}{151{,}665}
|
||||
\approx 5.3 \times 10^{-5}
|
||||
= 0.0053\%
|
||||
$$
|
||||
|
||||
of the vocabulary support at each position. The sparse signal could perturb the
|
||||
student, but it did not consistently transfer deeper reasoning behavior.
|
||||
|
||||
The final online path keeps the teacher and student in memory together and
|
||||
computes KL divergence against the teacher's full-vocabulary distribution. Token
|
||||
chunking keeps that dense objective feasible without materializing a single
|
||||
large KL workspace.
|
||||
|
||||
## Benchmark Scoreboard
|
||||
|
||||
The final public scoreboard compares `Qwen/Qwen3-1.7B-Base`,
|
||||
`Qwen/Qwen3-1.7B-Instruct`, and Quintus-1.7B.
|
||||
|
||||
|
||||
|
||||
The strongest signal is the reasoning crossover: Quintus beats both the base
|
||||
and official 1.7B instruct model on GSM8K, ARC-Challenge, and WinoGrande while
|
||||
remaining at the same parameter scale.
|
||||
|
||||
See [docs/benchmarks.md](docs/benchmarks.md) for the numeric table and
|
||||
interpretation. See
|
||||
[docs/evaluation_methodology.md](docs/evaluation_methodology.md) for benchmark
|
||||
controls.
|
||||
|
||||
## Evaluation Notes
|
||||
|
||||
Evaluation uses a mixture of EvalPlus and `lm-evaluation-harness`/vLLM style
|
||||
benchmarks. The repository keeps evaluation methodology separate because prompt
|
||||
format can change the result:
|
||||
|
||||
- Raw completion comparisons are used for base capability.
|
||||
- Chat-template comparisons are used for assistant-format behavior.
|
||||
- Log-likelihood tasks such as ARC-Challenge and PIQA should usually stay raw.
|
||||
- GSM8K can differ between strict `####` parsing and flexible number
|
||||
extraction.
|
||||
- Metric extraction must ignore `stderr`, aliases, and wrong filter keys.
|
||||
- Runtime versions, checkpoint identity, generation budget, and stale output
|
||||
cleanup are part of the evaluation contract.
|
||||
|
||||
The active benchmark runner is [sft/evaluate.py](sft/evaluate.py). It covers
|
||||
EvalPlus code tasks and `lm-evaluation-harness`/vLLM tasks, including GSM8K
|
||||
10-shot evaluation with an extended generation budget.
|
||||
|
||||
## Repository Map
|
||||
|
||||
```text
|
||||
configs/ Public run profile and DeepSpeed Zero-2 template.
|
||||
src/ Data prep, online KD, losses, packing, checkpoints, provenance.
|
||||
sft/ Post-KD SFT, local chat, and consolidated evaluation runner.
|
||||
docs/ Public architecture, training, evaluation, and release notes.
|
||||
weight_audit/ Checkpoint structure and weight-divergence audit material.
|
||||
```
|
||||
|
||||
Key files:
|
||||
|
||||
- [src/train.py](src/train.py): SFT, offline KD compatibility, and final
|
||||
`online_kd` training entry point.
|
||||
- [src/download.py](src/download.py): model setup, dataset loading, schema
|
||||
normalization, tokenization, and assistant-only loss masks.
|
||||
- [src/losses.py](src/losses.py): CE/KD objective, including online full-vocab
|
||||
KD token chunking.
|
||||
- [src/sequence_packing.py](src/sequence_packing.py): deterministic first-fit
|
||||
decreasing sequence packing.
|
||||
- [src/checkpoints.py](src/checkpoints.py): checkpoint save/resume metadata and
|
||||
packing compatibility checks.
|
||||
- [src/provenance.py](src/provenance.py): tokenizer/model/data contract checks.
|
||||
- [sft/train_sft.py](sft/train_sft.py): post-KD supervised fine-tuning.
|
||||
- [sft/evaluate.py](sft/evaluate.py): EvalPlus and
|
||||
`lm-evaluation-harness`/vLLM benchmark runner.
|
||||
- [sft/chat.py](sft/chat.py): local interactive chat wrapper.
|
||||
|
||||
## Commands
|
||||
|
||||
Install the base dependencies:
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
For training and benchmark runs, install the matching extras:
|
||||
|
||||
```bash
|
||||
pip install -r requirements-train.txt
|
||||
pip install -r requirements-eval.txt
|
||||
```
|
||||
|
||||
Inspect or prepare data/model assets:
|
||||
|
||||
```bash
|
||||
python -m src.download --help
|
||||
```
|
||||
|
||||
Run the final KD path after editing [configs/config.yaml](configs/config.yaml)
|
||||
for local paths and hardware:
|
||||
|
||||
```bash
|
||||
python -m src.train --phase online_kd
|
||||
```
|
||||
|
||||
Hub checkpoint uploads are off by default for local runs. Pass
|
||||
`--upload_last_checkpoint` or the step/epoch upload flags only after setting the
|
||||
target repository and `HF_TOKEN`.
|
||||
|
||||
Run the consolidated benchmark suite:
|
||||
|
||||
```bash
|
||||
python sft/evaluate.py
|
||||
```
|
||||
|
||||
Start local chat with a downloaded or local checkpoint:
|
||||
|
||||
```bash
|
||||
python sft/chat.py --model_path path/to/quintus/checkpoint
|
||||
```
|
||||
|
||||
## Interactive Chat
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
|
||||
|
||||
PUBLIC_REPO_ID = "iamrahulreddy/Quintus"
|
||||
|
||||
print(f"Loading Quintus from {PUBLIC_REPO_ID}...")
|
||||
tokenizer = AutoTokenizer.from_pretrained(PUBLIC_REPO_ID, trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
PUBLIC_REPO_ID,
|
||||
device_map="auto",
|
||||
dtype=torch.float16,
|
||||
trust_remote_code=True,
|
||||
)
|
||||
|
||||
stop_tokens = ["<|endoftext|>", "<|im_end|>"]
|
||||
eos_token_ids = [tokenizer.eos_token_id] if tokenizer.eos_token_id is not None else []
|
||||
for token in stop_tokens:
|
||||
token_id = tokenizer.convert_tokens_to_ids(token)
|
||||
if token_id is not None and token_id not in eos_token_ids:
|
||||
eos_token_ids.append(token_id)
|
||||
|
||||
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
|
||||
|
||||
conversation_history = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": (
|
||||
"You are Quintus, a highly capable AI assistant created by "
|
||||
"Muskula Rahul. You are helpful, precise, and logically sound."
|
||||
),
|
||||
}
|
||||
]
|
||||
|
||||
print()
|
||||
print("Quintus Chat (type 'quit' to exit)")
|
||||
print()
|
||||
|
||||
while True:
|
||||
try:
|
||||
user_input = input("You: ").strip()
|
||||
if user_input.lower() in ["quit", "exit"]:
|
||||
print("\nGoodbye!")
|
||||
break
|
||||
if not user_input:
|
||||
continue
|
||||
|
||||
conversation_history.append({"role": "user", "content": user_input})
|
||||
|
||||
prompt = tokenizer.apply_chat_template(
|
||||
conversation_history,
|
||||
tokenize=False,
|
||||
add_generation_prompt=True,
|
||||
)
|
||||
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
|
||||
print("Quintus: ", end="", flush=True)
|
||||
|
||||
with torch.no_grad():
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=512,
|
||||
temperature=0.7,
|
||||
top_p=0.9,
|
||||
do_sample=True,
|
||||
streamer=streamer,
|
||||
pad_token_id=tokenizer.eos_token_id,
|
||||
eos_token_id=eos_token_ids,
|
||||
)
|
||||
|
||||
generated_ids = outputs[0][inputs.input_ids.shape[-1]:]
|
||||
assistant_response = tokenizer.decode(
|
||||
generated_ids,
|
||||
skip_special_tokens=True,
|
||||
).strip()
|
||||
conversation_history.append({"role": "assistant", "content": assistant_response})
|
||||
print()
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n\nGoodbye!")
|
||||
break
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Documentation Index](docs/index.md): recommended public reading order.
|
||||
- [Architecture](docs/architecture.md): end-to-end data flow, modules, and
|
||||
training phases.
|
||||
- [Experiment Timeline](docs/experiment_timeline.md): why the project moved
|
||||
from offline top-k KD to online full-vocabulary KD.
|
||||
- [Training Playbook](docs/training_playbook.md): memory rules, packing,
|
||||
kernels, checkpointing, and B200-oriented guidance.
|
||||
- [Pipeline Hardening](docs/pipeline_hardening.md): silent-failure classes,
|
||||
artifact contracts, and safety checks.
|
||||
- [Evaluation Methodology](docs/evaluation_methodology.md): raw/chat controls,
|
||||
parser traps, metric extraction, and qualitative evaluation rules.
|
||||
- [Engineering Insights](docs/engineering_insights.md): condensed lessons and
|
||||
design decisions.
|
||||
- [Benchmarks](docs/benchmarks.md): verified scoreboard and interpretation.
|
||||
- [Weight Audit](docs/weight_audit.md): structural checkpoint sanity checks and
|
||||
weight-divergence summary.
|
||||
- [Hugging Face Model Card](docs/huggingface_model_card.md): release-page
|
||||
copy for the public model card.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Quintus is still a 1.7B model and inherits compact-model capacity limits.
|
||||
- Factual answers can be confidently wrong and should be verified.
|
||||
- Code generation may still contradict stated complexity or edge-case
|
||||
requirements.
|
||||
- Raw and chat-template results are not interchangeable.
|
||||
- Additional preference tuning or DPO would likely improve calibration, refusal
|
||||
behavior, and open-ended assistant polish.
|
||||
|
||||
## Credits
|
||||
|
||||
Quintus builds on open model, dataset, and tooling work from the broader LLM
|
||||
community:
|
||||
|
||||
- [Qwen Team](https://qwenlm.github.io/) and the
|
||||
[Qwen Hugging Face organization](https://huggingface.co/Qwen) for the Qwen3
|
||||
model family.
|
||||
- [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B), used as the
|
||||
distillation teacher.
|
||||
- [`Qwen/Qwen3-1.7B-Base`](https://huggingface.co/Qwen/Qwen3-1.7B-Base), used
|
||||
as the base student checkpoint.
|
||||
- [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B), used for the
|
||||
tokenizer and chat-template contract.
|
||||
- [Alibaba PAI](https://huggingface.co/alibaba-pai) for the
|
||||
[`DistilQwen_100k`](https://huggingface.co/datasets/alibaba-pai/DistilQwen_100k)
|
||||
dataset used as the primary instruction source after filtering.
|
||||
- [Hugging Face Transformers](https://github.com/huggingface/transformers) for
|
||||
model loading, tokenization, and generation APIs.
|
||||
- [vLLM](https://github.com/vllm-project/vllm),
|
||||
[EvalPlus](https://github.com/evalplus/evalplus), and
|
||||
[lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
|
||||
for evaluation infrastructure.
|
||||
- [FlashAttention](https://github.com/Dao-AILab/flash-attention) and
|
||||
[Liger Kernel](https://github.com/linkedin/Liger-Kernel) for performance
|
||||
kernels used or validated during training.
|
||||
|
||||
## License And Author
|
||||
|
||||
This software is distributed under the MIT License. Refer to the
|
||||
[LICENSE](LICENSE) file for full text.
|
||||
|
||||
Author: Muskula Rahul - [@iamrahulreddy](https://github.com/iamrahulreddy)
|
||||
|
||||
## Citation
|
||||
|
||||
If this model, codebase, or training pipeline is useful in your work, please cite this repository and acknowledge the upstream Qwen3 models.
|
||||
Reference in New Issue
Block a user