初始化项目,由ModelHub XC社区提供模型
Model: Nanthasit/sakthai-context-1.5b-merged Source: Original Platform
This commit is contained in:
120
eval/EVAL.md
Normal file
120
eval/EVAL.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# SakThai 1.5B Merged Model — Evaluation Report
|
||||
|
||||
**Model:** `Nanthasit/sakthai-context-1.5b-merged`
|
||||
**Base:** Qwen/Qwen2.5-1.5B-Instruct
|
||||
**Adapter:** Nanthasit/sakthai-context-1.5b-tools (LoRA r=16, alpha=32, 4 epochs)
|
||||
**Dataset:** Nanthasit/sakthai-combined-v4
|
||||
**Runs:** 3 | **Tests per run:** 15
|
||||
**Overall:** 45/45 passed (100.0%)
|
||||
|
||||
## Test-by-Test Results
|
||||
|
||||
| # | Category | Test | Pass Rate | Avg Time |
|
||||
|---|----------|------|:---------:|:--------:|
|
||||
| 1 | basic | greeting | ✅ 100% | 34.2s |
|
||||
| 2 | basic | self-identity | ✅ 100% | 72.4s |
|
||||
| 3 | multi-turn | name-recall | ✅ 100% | 13.3s |
|
||||
| 4 | multi-turn | context-follow | ✅ 100% | 9.5s |
|
||||
| 5 | multi-turn | preference-remember | ✅ 100% | 33.7s |
|
||||
| 6 | instruction | json-array-output | ✅ 100% | 10.2s |
|
||||
| 7 | instruction | numbered-list | ✅ 100% | 38.9s |
|
||||
| 8 | instruction | concise-output | ✅ 100% | 15.0s |
|
||||
| 9 | tool-calling | weather-query | ✅ 100% | 133.3s |
|
||||
| 10 | tool-calling | limitation-awareness | ✅ 100% | 8.6s |
|
||||
| 11 | reasoning | simple-math | ✅ 100% | 72.8s |
|
||||
| 12 | reasoning | coding-query | ✅ 100% | 51.1s |
|
||||
| 13 | reasoning | explain-concept | ✅ 100% | 39.6s |
|
||||
| 14 | format | json-object | ✅ 100% | 20.3s |
|
||||
| 15 | format | markdown-table | ✅ 100% | 49.1s |
|
||||
|
||||
|
||||
## Comparison: 0.5B vs 1.5B
|
||||
|
||||
Pass rates from 0.5B eval (single run): See `eval/EVAL.md` in 0.5b-merged repo.
|
||||
|
||||
## Sample Responses
|
||||
|
||||
|
||||
### greeting
|
||||
|
||||
> I'm here to help with any questions you may have. Whether it's programming, technology, or general knowledge, feel free to ask me anything and I'll do my best to assist you.
|
||||
|
||||
### self-identity
|
||||
|
||||
> I am Qwen, the AI language model developed by Alibaba Cloud. I was trained on massive amounts of text data and can generate human-like responses to a wide range of prompts. My capabilities include gen
|
||||
|
||||
### name-recall
|
||||
|
||||
> Your name is Beer. It's nice to know that you're here with me today. How can I assist you?
|
||||
|
||||
### context-follow
|
||||
|
||||
> You like cats and programming.
|
||||
|
||||
### preference-remember
|
||||
|
||||
> As an AI language model, I don't have access to your personal preferences or memories. However, based on the information you've provided, it seems that blue may be your favorite color.
|
||||
|
||||
### json-array-output
|
||||
|
||||
> ["red","blue","yellow"]
|
||||
|
||||
### numbered-list
|
||||
|
||||
> 1. Boil water in a kettle or on the stove.
|
||||
2. Place loose tea leaves or tea bags into a teapot or cup.
|
||||
3. Pour hot water over the tea and let it steep for several minutes.
|
||||
|
||||
### concise-output
|
||||
|
||||
> A GPU accelerates computations through parallel processing of large data sets.
|
||||
|
||||
### weather-query
|
||||
|
||||
> Tokyo has four seasons: spring (March to May), summer (June to August), autumn (September to November), and winter (December to February). The temperature ranges from 10°C to 35°C throughout the year.
|
||||
|
||||
### limitation-awareness
|
||||
|
||||
> Subject: Hello John!
|
||||
|
||||
### simple-math
|
||||
|
||||
> To calculate the distance traveled by the train, you can use the formula:
|
||||
|
||||
Distance = Speed x Time
|
||||
|
||||
In this case, the speed is given as 120 km/h and the time is 2.5 hours.
|
||||
|
||||
So,
|
||||
|
||||
Distance = 120 km/h x
|
||||
|
||||
### coding-query
|
||||
|
||||
> def is_palindrome(string):
|
||||
# Remove any non-alphanumeric characters and convert to lowercase
|
||||
cleaned_string = ''.join(char.lower() for char in string if char.isalnum())
|
||||
|
||||
# Check if the
|
||||
|
||||
### explain-concept
|
||||
|
||||
> LoRA is a lightweight method that uses a small number of parameters to approximate the behavior of a larger model during inference. Full fine-tuning involves training the entire model from scratch on
|
||||
|
||||
### json-object
|
||||
|
||||
> {
|
||||
"name": "John",
|
||||
"age": 30,
|
||||
"city": "London"
|
||||
}
|
||||
|
||||
### markdown-table
|
||||
|
||||
> | Language | Typing | Speed |
|
||||
| --- | --- | --- |
|
||||
| Python | Dynamic typing | Fast |
|
||||
| JavaScript | Dynamic typing | Fast |
|
||||
| Rust | Static typing | Slow |
|
||||
|
||||
Note: The speed of programming languages ca
|
||||
Reference in New Issue
Block a user