163 lines
5.2 KiB
Markdown
163 lines
5.2 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- qwen2
|
||
- sakthai
|
||
- tool-calling
|
||
- instruct
|
||
- lora
|
||
datasets:
|
||
- Nanthasit/sakthai-combined-v4
|
||
base_model: Qwen/Qwen2.5-1.5B-Instruct
|
||
model-index:
|
||
- name: sakthai-context-1.5b-merged
|
||
results:
|
||
- task:
|
||
type: text-generation
|
||
name: Tool-Calling & Instruction Following
|
||
dataset:
|
||
name: SakThai Eval Suite
|
||
type: Nanthasit/sakthai-combined-v4
|
||
metrics:
|
||
- type: pass_rate
|
||
value: 100
|
||
name: Overall Pass Rate (45/45)
|
||
- type: pass_rate
|
||
value: 100
|
||
name: Basic (6/6)
|
||
- type: pass_rate
|
||
value: 100
|
||
name: Multi-Turn (9/9)
|
||
- type: pass_rate
|
||
value: 100
|
||
name: Instruction Following (6/6)
|
||
- type: pass_rate
|
||
value: 100
|
||
name: Tool Calling (6/6)
|
||
- type: pass_rate
|
||
value: 100
|
||
name: Reasoning (6/6)
|
||
- type: pass_rate
|
||
value: 100
|
||
name: Format Adherence (12/12)
|
||
---
|
||
|
||
# SakThai Context 1.5B
|
||
|
||
Fine-tuned from **Qwen2.5-1.5B-Instruct** on the SakThai combined dataset for **tool-calling, multi-turn context, and instruction-following** capabilities. Designed as the reasoning backbone for the SakThai agent (running on Hermes Agent framework).
|
||
|
||
## Model Details
|
||
|
||
| Property | Value |
|
||
|----------|-------|
|
||
| **Base Model** | [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) |
|
||
| **Architecture** | Qwen2 (decoder-only transformer) |
|
||
| **Hidden Size** | 1536 |
|
||
| **Layers** | 28 |
|
||
| **Attention Heads** | 12 |
|
||
| **Intermediate Size** | 8960 |
|
||
| **Vocab Size** | 151936 |
|
||
| **Fine-tuning Method** | LoRA (r=16, α=32, dropout=0.1) |
|
||
| **Target Modules** | q_proj, k_proj, v_proj, o_proj |
|
||
| **Training Steps** | 220 |
|
||
| **Training Duration** | ~39 minutes (4 epochs on 974 examples) |
|
||
| **License** | Apache 2.0 |
|
||
|
||
## Training
|
||
|
||
- **Base model:** Qwen/Qwen2.5-1.5B-Instruct
|
||
- **Dataset:** [Nanthasit/sakthai-combined-v4](https://huggingface.co/datasets/Nanthasit/sakthai-combined-v4)
|
||
— 974 training + 51 test examples covering 25 canonical tool schemas
|
||
- **Method:** LoRA via PEFT (rank=16, alpha=32, dropout=0.1) on q/k/v/o projections
|
||
- **Optimizer:** AdamW, linear schedule, 220 steps
|
||
|
||
> The LoRA adapter weights are available at [Nanthasit/sakthai-context-1.5b-tools](https://huggingface.co/Nanthasit/sakthai-context-1.5b-tools).
|
||
|
||
## Evaluation
|
||
|
||
**45/45 tests passed (100%)** across 3 runs (15 tests/run).
|
||
|
||
| Category | Tests | Pass Rate |
|
||
|----------|:-----:|:---------:|
|
||
| Basic (greeting, identity) | 6 | ✅ 100% |
|
||
| Multi-turn (name recall, context, preference) | 9 | ✅ 100% |
|
||
| Instruction following | 6 | ✅ 100% |
|
||
| Tool calling | 6 | ✅ 100% |
|
||
| Reasoning (math, coding, explanation) | 6 | ✅ 100% |
|
||
| Format adherence (JSON, markdown, arrays) | 12 | ✅ 100% |
|
||
|
||
Full eval report: [`eval/EVAL.md`](https://huggingface.co/Nanthasit/sakthai-context-1.5b-merged/blob/main/eval/EVAL.md)
|
||
|
||
## Usage
|
||
|
||
### Via Transformers
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model = AutoModelForCausalLM.from_pretrained("Nanthasit/sakthai-context-1.5b-merged")
|
||
tokenizer = AutoTokenizer.from_pretrained("Nanthasit/sakthai-context-1.5b-merged")
|
||
|
||
messages = [{"role": "user", "content": "What's the capital of Japan?"}]
|
||
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
inputs = tokenizer(text, return_tensors="pt")
|
||
outputs = model.generate(**inputs, max_new_tokens=256)
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
```
|
||
|
||
### Via Inference Client
|
||
|
||
```python
|
||
from huggingface_hub import InferenceClient
|
||
|
||
client = InferenceClient("Nanthasit/sakthai-context-1.5b-merged")
|
||
response = client.chat_completion(
|
||
messages=[{"role": "user", "content": "Hello!"}],
|
||
max_tokens=256
|
||
)
|
||
print(response.choices[0].message.content)
|
||
```
|
||
|
||
### Merging the Adapter
|
||
|
||
```python
|
||
from peft import PeftModel
|
||
from transformers import AutoModelForCausalLM
|
||
|
||
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
|
||
model = PeftModel.from_pretrained(base, "Nanthasit/sakthai-context-1.5b-tools")
|
||
merged = model.merge_and_unload()
|
||
merged.save_pretrained("./sakthai-context-1.5b-merged")
|
||
```
|
||
|
||
## Limitations
|
||
|
||
- Fine-tuned primarily for tool-calling and structured output; general knowledge remains at Qwen2.5-1.5B-Instruct baseline level.
|
||
- Tested on CPU — performance on GPU inference may produce slightly different output distributions.
|
||
- Best suited for agentic workflows with well-defined tool schemas. Complex multi-hop reasoning may require a larger base model.
|
||
|
||
## Bias, Risks & Safety
|
||
|
||
This model is fine-tuned from Qwen2.5-1.5B-Instruct and inherits its base strengths and limitations. As a small language model (1.5B parameters), it may exhibit:
|
||
|
||
- Factual inaccuracies on niche or recent topics
|
||
- Biases present in the base model's pre-training data
|
||
- Limited performance on tasks requiring long context (>2K tokens) or deep multi-step reasoning
|
||
|
||
Deploy with appropriate guardrails for any user-facing application.
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{sakthai-context-1.5b,
|
||
author = {Nanthasit},
|
||
title = {SakThai Context 1.5B — Tool-Calling Fine-Tune},
|
||
year = {2026},
|
||
publisher = {Hugging Face},
|
||
howpublished = {\url{https://huggingface.co/Nanthasit/sakthai-context-1.5b-merged}}
|
||
}
|
||
``` |