119 lines
4.0 KiB
Markdown
119 lines
4.0 KiB
Markdown
---
|
|
license: apache-2.0
|
|
datasets:
|
|
- open-r1/Mixture-of-Thoughts
|
|
language:
|
|
- en
|
|
base_model:
|
|
- Qwen/Qwen3-0.6B
|
|
pipeline_tag: text-generation
|
|
library_name: transformers
|
|
tags:
|
|
- text-generation-inference
|
|
- code
|
|
- moe
|
|
- math
|
|
- stem
|
|
- trl
|
|
---
|
|
|
|
|
|
|
|
# **Nenque-MoT-0.6B-Elite14**
|
|
|
|
> **Nenque-MoT-0.6B-Elite14** is a compact, high-efficiency model tailored for **mathematical reasoning**, **code generation**, and **structured technical inference**. Fine-tuned from **Qwen3-0.6B** using the **MoT (Mixture of Thoughts) dataset**—with a focus on **math expert clusters**—this model delivers strong symbolic performance in low-resource environments. Despite its **0.6B parameter** size, it offers elite-level precision across STEM and multilingual technical domains.
|
|
|
|
> [!note]
|
|
GGUF: [https://huggingface.co/prithivMLmods/Nenque-MoT-0.6B-Elite14-GGUF](https://huggingface.co/prithivMLmods/Nenque-MoT-0.6B-Elite14-GGUF)
|
|
|
|
---
|
|
|
|
## **Key Features**
|
|
|
|
1. **MoT Fine-Tuning on Math Expert Clusters**
|
|
Trained on a curated **Mixture of Thoughts (MoT)** dataset emphasizing symbolic mathematics, code reasoning, and problem-solving, enhancing precision in structured tasks.
|
|
|
|
2. **Elite Mathematical Reasoning**
|
|
Excels in solving algebraic equations, calculus, and symbolic logic step-by-step—suitable for education, competitions, and STEM support tools.
|
|
|
|
3. **Compact Code Assistant**
|
|
Generates concise, explainable code in Python, JavaScript, and others—ideal for code tutoring, bug diagnosis, and fast prototyping.
|
|
|
|
4. **Structured Output Generation**
|
|
Supports generation in **Markdown**, **JSON**, **LaTeX**, and **tabular formats**, making it a valuable tool for documentation and technical data generation.
|
|
|
|
5. **Multilingual Technical Mastery**
|
|
Delivers consistent results across 20+ languages for math and code—serving global academic and development use cases.
|
|
|
|
6. **Lightweight Inference-Ready Design**
|
|
Optimized for **edge devices**, **GPUs with limited VRAM**, and **offline deployments**, enabling high-quality results on constrained systems.
|
|
|
|
---
|
|
|
|
## **Quickstart with Transformers**
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_name = "prithivMLmods/Nenque-MoT-0.6B-Elite14"
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_name,
|
|
torch_dtype="auto",
|
|
device_map="auto"
|
|
)
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
|
|
prompt = "Solve the equation: 2(x - 4) + 3 = 11. Show all steps."
|
|
|
|
messages = [
|
|
{"role": "system", "content": "You are a step-by-step math tutor."},
|
|
{"role": "user", "content": prompt}
|
|
]
|
|
|
|
text = tokenizer.apply_chat_template(
|
|
messages,
|
|
tokenize=False,
|
|
add_generation_prompt=True
|
|
)
|
|
|
|
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
|
|
|
generated_ids = model.generate(
|
|
**model_inputs,
|
|
max_new_tokens=512
|
|
)
|
|
generated_ids = [
|
|
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
|
|
]
|
|
|
|
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
|
|
print(response)
|
|
```
|
|
|
|
---
|
|
|
|
## **Intended Use**
|
|
|
|
* Step-by-step mathematical reasoning and symbolic computation
|
|
* Lightweight multilingual code generation and debugging
|
|
* Structured content generation (e.g., LaTeX, JSON, Markdown)
|
|
* Academic tutoring and technical assistant roles
|
|
* Deployment in resource-constrained or edge scenarios
|
|
|
|
---
|
|
|
|
## **Limitations**
|
|
|
|
* Not suitable for extended creative generation or conversational fluency
|
|
* Limited context length impacts performance on long multi-step tasks
|
|
* Fine-tuned on technical domains—general chat or abstract logic tasks may underperform
|
|
* Specialized for structured outputs—free-form generation is not its focus
|
|
|
|
---
|
|
|
|
## **References**
|
|
|
|
1. [Qwen2.5 Technical Report (2024)](https://arxiv.org/pdf/2412.15115)
|
|
2. [YaRN: Efficient Context Window Extension of Large Language Models](https://arxiv.org/pdf/2309.00071)
|
|
3. [open-r1/Mixture-of-Thoughts](https://huggingface.co/datasets/open-r1/Mixture-of-Thoughts) |