初始化项目，由ModelHub XC社区提供模型
Model: kshitijthakkar/loggenix-moe-0.4B-0.2A-sft-s3.1 Source: Original Platform
This commit is contained in:
ModelHub XC
172
README.md
Normal file
172
README.md
Normal file
@@ -0,0 +1,172 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- moe
|
||||
- mixture-of-experts
|
||||
- qwen3
|
||||
- function-calling
|
||||
- tool-use
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
---
|
||||
|
||||
# loggenix-moe-0.4B-0.2A-sft-s3.1
|
||||
|
||||
## Model Description
|
||||
|
||||
This is a Mixture of Experts (MoE) language model fine-tuned for various tasks including:
|
||||
- Tool/Function Calling
|
||||
- Code Generation
|
||||
- Reasoning & Math
|
||||
- Safety & Content Moderation
|
||||
|
||||
## Evaluation Results
|
||||
|
||||
**Evaluation Date:** 20260128_105522
|
||||
|
||||
### Standard Benchmarks (lm-evaluation-harness)
|
||||
|
||||
| Benchmark | Score |
|
||||
|-----------|-------|
|
||||
| arc_challenge | 10.00% |
|
||||
| arc_easy | 10.00% |
|
||||
| boolq | 40.00% |
|
||||
| gsm8k | 0.00% |
|
||||
| hellaswag | 40.00% |
|
||||
| mmlu | 25.26% |
|
||||
| mmlu_humanities | 26.15% |
|
||||
| mmlu_formal_logic | 20.00% |
|
||||
| mmlu_high_school_european_history | 20.00% |
|
||||
| mmlu_high_school_us_history | 30.00% |
|
||||
| mmlu_high_school_world_history | 30.00% |
|
||||
| mmlu_international_law | 30.00% |
|
||||
| mmlu_jurisprudence | 40.00% |
|
||||
| mmlu_logical_fallacies | 40.00% |
|
||||
| mmlu_moral_disputes | 10.00% |
|
||||
| mmlu_moral_scenarios | 20.00% |
|
||||
| mmlu_philosophy | 10.00% |
|
||||
| mmlu_prehistory | 30.00% |
|
||||
| mmlu_professional_law | 20.00% |
|
||||
| mmlu_world_religions | 40.00% |
|
||||
| mmlu_other | 26.15% |
|
||||
| mmlu_business_ethics | 10.00% |
|
||||
| mmlu_clinical_knowledge | 40.00% |
|
||||
| mmlu_college_medicine | 40.00% |
|
||||
| mmlu_global_facts | 20.00% |
|
||||
| mmlu_human_aging | 40.00% |
|
||||
| mmlu_management | 0.00% |
|
||||
| mmlu_marketing | 50.00% |
|
||||
| mmlu_medical_genetics | 30.00% |
|
||||
| mmlu_miscellaneous | 20.00% |
|
||||
| mmlu_nutrition | 20.00% |
|
||||
| mmlu_professional_accounting | 30.00% |
|
||||
| mmlu_professional_medicine | 20.00% |
|
||||
| mmlu_virology | 20.00% |
|
||||
| mmlu_social_sciences | 27.50% |
|
||||
| mmlu_econometrics | 40.00% |
|
||||
| mmlu_high_school_geography | 20.00% |
|
||||
| mmlu_high_school_government_and_politics | 30.00% |
|
||||
| mmlu_high_school_macroeconomics | 0.00% |
|
||||
| mmlu_high_school_microeconomics | 10.00% |
|
||||
| mmlu_high_school_psychology | 50.00% |
|
||||
| mmlu_human_sexuality | 20.00% |
|
||||
| mmlu_professional_psychology | 40.00% |
|
||||
| mmlu_public_relations | 20.00% |
|
||||
| mmlu_security_studies | 30.00% |
|
||||
| mmlu_sociology | 20.00% |
|
||||
| mmlu_us_foreign_policy | 50.00% |
|
||||
| mmlu_stem | 22.63% |
|
||||
| mmlu_abstract_algebra | 20.00% |
|
||||
| mmlu_anatomy | 20.00% |
|
||||
| mmlu_astronomy | 20.00% |
|
||||
| mmlu_college_biology | 30.00% |
|
||||
| mmlu_college_chemistry | 10.00% |
|
||||
| mmlu_college_computer_science | 20.00% |
|
||||
| mmlu_college_mathematics | 20.00% |
|
||||
| mmlu_college_physics | 30.00% |
|
||||
| mmlu_computer_security | 50.00% |
|
||||
| mmlu_conceptual_physics | 30.00% |
|
||||
| mmlu_electrical_engineering | 40.00% |
|
||||
| mmlu_elementary_mathematics | 0.00% |
|
||||
| mmlu_high_school_biology | 30.00% |
|
||||
| mmlu_high_school_chemistry | 30.00% |
|
||||
| mmlu_high_school_computer_science | 20.00% |
|
||||
| mmlu_high_school_mathematics | 20.00% |
|
||||
| mmlu_high_school_physics | 10.00% |
|
||||
| mmlu_high_school_statistics | 0.00% |
|
||||
| mmlu_machine_learning | 30.00% |
|
||||
| openbookqa | 40.00% |
|
||||
| piqa | 70.00% |
|
||||
| winogrande | 60.00% |
|
||||
|
||||
### Synthetic Task Categories
|
||||
|
||||
| Category | Score | Tasks Evaluated |
|
||||
|----------|-------|-----------------|
|
||||
| Tool Calling | 50.00% | 1 |
|
||||
| SRE DevOps | 16.67% | 12 |
|
||||
| Programming | 25.88% | 17 |
|
||||
| Reasoning | 11.25% | 4 |
|
||||
| LLM Evaluation | 22.00% | 5 |
|
||||
| Safety Ethics | 20.00% | 8 |
|
||||
| Financial | 23.75% | 8 |
|
||||
| Customer Support | 16.88% | 8 |
|
||||
| Observability | 26.25% | 8 |
|
||||
| Content Generation | 13.33% | 3 |
|
||||
| Core AI | 35.00% | 3 |
|
||||
|
||||
### Tool-Calling Performance
|
||||
|
||||
| Metric | Score |
|
||||
|--------|-------|
|
||||
| Format Accuracy | 20.00% |
|
||||
| Function Name Accuracy | 0.00% |
|
||||
| Parameter Accuracy | 0.00% |
|
||||
| Overall Accuracy | 6.67% |
|
||||
|
||||
### Code Generation Performance
|
||||
|
||||
| Metric | Score |
|
||||
|--------|-------|
|
||||
| Syntax Accuracy | 8.33% |
|
||||
| Keyword Coverage | 29.38% |
|
||||
| Completion Rate | 75.00% |
|
||||
|
||||
### Overall Summary
|
||||
|
||||
- **Synthetic Tasks Mean Score:** 21.68%
|
||||
- **Total Tasks Evaluated:** 170
|
||||
- **Task Coverage:** 180.9%
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("kshitijthakkar/loggenix-moe-0.4B-0.2A-sft-s3.1")
|
||||
model = AutoModelForCausalLM.from_pretrained("kshitijthakkar/loggenix-moe-0.4B-0.2A-sft-s3.1", trust_remote_code=True)
|
||||
|
||||
# For tool calling
|
||||
messages = [
|
||||
{"role": "system", "content": "You are a helpful assistant with access to tools."},
|
||||
{"role": "user", "content": "What's the weather in San Francisco?"}
|
||||
]
|
||||
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
|
||||
outputs = model.generate(inputs, max_new_tokens=256)
|
||||
print(tokenizer.decode(outputs[0]))
|
||||
```
|
||||
|
||||
## Training Data
|
||||
|
||||
The model was fine-tuned on a diverse dataset including:
|
||||
- Tool-calling datasets (Toucan, ToolACE, SmoLAgents)
|
||||
- Safety datasets (HelpSteer3, Safety-Guard, Content-Safety-Reasoning)
|
||||
- Math datasets (GSM8K, MetaMath, Big-Math-RL)
|
||||
- Code datasets (Magicoder)
|
||||
- Reasoning datasets (Reasoning-Gemini, Textbook-Reasoning)
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0
|
||||
Reference in New Issue
Block a user