---
language:
- en
license: apache-2.0
tags:
- moe
- mixture-of-experts
- qwen3
- function-calling
- tool-use
pipeline_tag: text-generation
library_name: transformers
---

# loggenix-moe-0.4B-0.2A-sft-s3.1

## Model Description

This is a Mixture of Experts (MoE) language model fine-tuned for various tasks including:
- Tool/Function Calling
- Code Generation
- Reasoning & Math
- Safety & Content Moderation

## Evaluation Results

**Evaluation Date:** 20260128_105522

### Standard Benchmarks (lm-evaluation-harness)

| Benchmark | Score |
|-----------|-------|
| arc_challenge | 10.00% |
| arc_easy | 10.00% |
| boolq | 40.00% |
| gsm8k | 0.00% |
| hellaswag | 40.00% |
| mmlu | 25.26% |
| mmlu_humanities | 26.15% |
| mmlu_formal_logic | 20.00% |
| mmlu_high_school_european_history | 20.00% |
| mmlu_high_school_us_history | 30.00% |
| mmlu_high_school_world_history | 30.00% |
| mmlu_international_law | 30.00% |
| mmlu_jurisprudence | 40.00% |
| mmlu_logical_fallacies | 40.00% |
| mmlu_moral_disputes | 10.00% |
| mmlu_moral_scenarios | 20.00% |
| mmlu_philosophy | 10.00% |
| mmlu_prehistory | 30.00% |
| mmlu_professional_law | 20.00% |
| mmlu_world_religions | 40.00% |
| mmlu_other | 26.15% |
| mmlu_business_ethics | 10.00% |
| mmlu_clinical_knowledge | 40.00% |
| mmlu_college_medicine | 40.00% |
| mmlu_global_facts | 20.00% |
| mmlu_human_aging | 40.00% |
| mmlu_management | 0.00% |
| mmlu_marketing | 50.00% |
| mmlu_medical_genetics | 30.00% |
| mmlu_miscellaneous | 20.00% |
| mmlu_nutrition | 20.00% |
| mmlu_professional_accounting | 30.00% |
| mmlu_professional_medicine | 20.00% |
| mmlu_virology | 20.00% |
| mmlu_social_sciences | 27.50% |
| mmlu_econometrics | 40.00% |
| mmlu_high_school_geography | 20.00% |
| mmlu_high_school_government_and_politics | 30.00% |
| mmlu_high_school_macroeconomics | 0.00% |
| mmlu_high_school_microeconomics | 10.00% |
| mmlu_high_school_psychology | 50.00% |
| mmlu_human_sexuality | 20.00% |
| mmlu_professional_psychology | 40.00% |
| mmlu_public_relations | 20.00% |
| mmlu_security_studies | 30.00% |
| mmlu_sociology | 20.00% |
| mmlu_us_foreign_policy | 50.00% |
| mmlu_stem | 22.63% |
| mmlu_abstract_algebra | 20.00% |
| mmlu_anatomy | 20.00% |
| mmlu_astronomy | 20.00% |
| mmlu_college_biology | 30.00% |
| mmlu_college_chemistry | 10.00% |
| mmlu_college_computer_science | 20.00% |
| mmlu_college_mathematics | 20.00% |
| mmlu_college_physics | 30.00% |
| mmlu_computer_security | 50.00% |
| mmlu_conceptual_physics | 30.00% |
| mmlu_electrical_engineering | 40.00% |
| mmlu_elementary_mathematics | 0.00% |
| mmlu_high_school_biology | 30.00% |
| mmlu_high_school_chemistry | 30.00% |
| mmlu_high_school_computer_science | 20.00% |
| mmlu_high_school_mathematics | 20.00% |
| mmlu_high_school_physics | 10.00% |
| mmlu_high_school_statistics | 0.00% |
| mmlu_machine_learning | 30.00% |
| openbookqa | 40.00% |
| piqa | 70.00% |
| winogrande | 60.00% |

### Synthetic Task Categories

| Category | Score | Tasks Evaluated |
|----------|-------|-----------------|
| Tool Calling | 50.00% | 1 |
| SRE DevOps | 16.67% | 12 |
| Programming | 25.88% | 17 |
| Reasoning | 11.25% | 4 |
| LLM Evaluation | 22.00% | 5 |
| Safety Ethics | 20.00% | 8 |
| Financial | 23.75% | 8 |
| Customer Support | 16.88% | 8 |
| Observability | 26.25% | 8 |
| Content Generation | 13.33% | 3 |
| Core AI | 35.00% | 3 |

### Tool-Calling Performance

| Metric | Score |
|--------|-------|
| Format Accuracy | 20.00% |
| Function Name Accuracy | 0.00% |
| Parameter Accuracy | 0.00% |
| Overall Accuracy | 6.67% |

### Code Generation Performance

| Metric | Score |
|--------|-------|
| Syntax Accuracy | 8.33% |
| Keyword Coverage | 29.38% |
| Completion Rate | 75.00% |

### Overall Summary

- **Synthetic Tasks Mean Score:** 21.68%
- **Total Tasks Evaluated:** 170
- **Task Coverage:** 180.9%

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("kshitijthakkar/loggenix-moe-0.4B-0.2A-sft-s3.1")
model = AutoModelForCausalLM.from_pretrained("kshitijthakkar/loggenix-moe-0.4B-0.2A-sft-s3.1", trust_remote_code=True)

# For tool calling
messages = [
    {"role": "system", "content": "You are a helpful assistant with access to tools."},
    {"role": "user", "content": "What's the weather in San Francisco?"}
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))
```

## Training Data

The model was fine-tuned on a diverse dataset including:
- Tool-calling datasets (Toucan, ToolACE, SmoLAgents)
- Safety datasets (HelpSteer3, Safety-Guard, Content-Safety-Reasoning)
- Math datasets (GSM8K, MetaMath, Big-Math-RL)
- Code datasets (Magicoder)
- Reasoning datasets (Reasoning-Gemini, Textbook-Reasoning)

## License

Apache 2.0