121 lines
3.7 KiB
Markdown
121 lines
3.7 KiB
Markdown
# SakThai 1.5B Merged Model — Evaluation Report
|
|
|
|
**Model:** `Nanthasit/sakthai-context-1.5b-merged`
|
|
**Base:** Qwen/Qwen2.5-1.5B-Instruct
|
|
**Adapter:** Nanthasit/sakthai-context-1.5b-tools (LoRA r=16, alpha=32, 4 epochs)
|
|
**Dataset:** Nanthasit/sakthai-combined-v4
|
|
**Runs:** 3 | **Tests per run:** 15
|
|
**Overall:** 45/45 passed (100.0%)
|
|
|
|
## Test-by-Test Results
|
|
|
|
| # | Category | Test | Pass Rate | Avg Time |
|
|
|---|----------|------|:---------:|:--------:|
|
|
| 1 | basic | greeting | ✅ 100% | 34.2s |
|
|
| 2 | basic | self-identity | ✅ 100% | 72.4s |
|
|
| 3 | multi-turn | name-recall | ✅ 100% | 13.3s |
|
|
| 4 | multi-turn | context-follow | ✅ 100% | 9.5s |
|
|
| 5 | multi-turn | preference-remember | ✅ 100% | 33.7s |
|
|
| 6 | instruction | json-array-output | ✅ 100% | 10.2s |
|
|
| 7 | instruction | numbered-list | ✅ 100% | 38.9s |
|
|
| 8 | instruction | concise-output | ✅ 100% | 15.0s |
|
|
| 9 | tool-calling | weather-query | ✅ 100% | 133.3s |
|
|
| 10 | tool-calling | limitation-awareness | ✅ 100% | 8.6s |
|
|
| 11 | reasoning | simple-math | ✅ 100% | 72.8s |
|
|
| 12 | reasoning | coding-query | ✅ 100% | 51.1s |
|
|
| 13 | reasoning | explain-concept | ✅ 100% | 39.6s |
|
|
| 14 | format | json-object | ✅ 100% | 20.3s |
|
|
| 15 | format | markdown-table | ✅ 100% | 49.1s |
|
|
|
|
|
|
## Comparison: 0.5B vs 1.5B
|
|
|
|
Pass rates from 0.5B eval (single run): See `eval/EVAL.md` in 0.5b-merged repo.
|
|
|
|
## Sample Responses
|
|
|
|
|
|
### greeting
|
|
|
|
> I'm here to help with any questions you may have. Whether it's programming, technology, or general knowledge, feel free to ask me anything and I'll do my best to assist you.
|
|
|
|
### self-identity
|
|
|
|
> I am Qwen, the AI language model developed by Alibaba Cloud. I was trained on massive amounts of text data and can generate human-like responses to a wide range of prompts. My capabilities include gen
|
|
|
|
### name-recall
|
|
|
|
> Your name is Beer. It's nice to know that you're here with me today. How can I assist you?
|
|
|
|
### context-follow
|
|
|
|
> You like cats and programming.
|
|
|
|
### preference-remember
|
|
|
|
> As an AI language model, I don't have access to your personal preferences or memories. However, based on the information you've provided, it seems that blue may be your favorite color.
|
|
|
|
### json-array-output
|
|
|
|
> ["red","blue","yellow"]
|
|
|
|
### numbered-list
|
|
|
|
> 1. Boil water in a kettle or on the stove.
|
|
2. Place loose tea leaves or tea bags into a teapot or cup.
|
|
3. Pour hot water over the tea and let it steep for several minutes.
|
|
|
|
### concise-output
|
|
|
|
> A GPU accelerates computations through parallel processing of large data sets.
|
|
|
|
### weather-query
|
|
|
|
> Tokyo has four seasons: spring (March to May), summer (June to August), autumn (September to November), and winter (December to February). The temperature ranges from 10°C to 35°C throughout the year.
|
|
|
|
### limitation-awareness
|
|
|
|
> Subject: Hello John!
|
|
|
|
### simple-math
|
|
|
|
> To calculate the distance traveled by the train, you can use the formula:
|
|
|
|
Distance = Speed x Time
|
|
|
|
In this case, the speed is given as 120 km/h and the time is 2.5 hours.
|
|
|
|
So,
|
|
|
|
Distance = 120 km/h x
|
|
|
|
### coding-query
|
|
|
|
> def is_palindrome(string):
|
|
# Remove any non-alphanumeric characters and convert to lowercase
|
|
cleaned_string = ''.join(char.lower() for char in string if char.isalnum())
|
|
|
|
# Check if the
|
|
|
|
### explain-concept
|
|
|
|
> LoRA is a lightweight method that uses a small number of parameters to approximate the behavior of a larger model during inference. Full fine-tuning involves training the entire model from scratch on
|
|
|
|
### json-object
|
|
|
|
> {
|
|
"name": "John",
|
|
"age": 30,
|
|
"city": "London"
|
|
}
|
|
|
|
### markdown-table
|
|
|
|
> | Language | Typing | Speed |
|
|
| --- | --- | --- |
|
|
| Python | Dynamic typing | Fast |
|
|
| JavaScript | Dynamic typing | Fast |
|
|
| Rust | Static typing | Slow |
|
|
|
|
Note: The speed of programming languages ca
|