sakthai-context-1.5b-merged/eval/EVAL.md

SakThai 1.5B Merged Model — Evaluation Report

Model: Nanthasit/sakthai-context-1.5b-merged Base: Qwen/Qwen2.5-1.5B-Instruct Adapter: Nanthasit/sakthai-context-1.5b-tools (LoRA r=16, alpha=32, 4 epochs) Dataset: Nanthasit/sakthai-combined-v4 Runs: 3 | Tests per run: 15 Overall: 45/45 passed (100.0%)

Test-by-Test Results

#	Category	Test	Pass Rate	Avg Time
1	basic	greeting	✅ 100%	34.2s
2	basic	self-identity	✅ 100%	72.4s
3	multi-turn	name-recall	✅ 100%	13.3s
4	multi-turn	context-follow	✅ 100%	9.5s
5	multi-turn	preference-remember	✅ 100%	33.7s
6	instruction	json-array-output	✅ 100%	10.2s
7	instruction	numbered-list	✅ 100%	38.9s
8	instruction	concise-output	✅ 100%	15.0s
9	tool-calling	weather-query	✅ 100%	133.3s
10	tool-calling	limitation-awareness	✅ 100%	8.6s
11	reasoning	simple-math	✅ 100%	72.8s
12	reasoning	coding-query	✅ 100%	51.1s
13	reasoning	explain-concept	✅ 100%	39.6s
14	format	json-object	✅ 100%	20.3s
15	format	markdown-table	✅ 100%	49.1s

Comparison: 0.5B vs 1.5B

Pass rates from 0.5B eval (single run): See eval/EVAL.md in 0.5b-merged repo.

Sample Responses

greeting

I'm here to help with any questions you may have. Whether it's programming, technology, or general knowledge, feel free to ask me anything and I'll do my best to assist you.

self-identity

I am Qwen, the AI language model developed by Alibaba Cloud. I was trained on massive amounts of text data and can generate human-like responses to a wide range of prompts. My capabilities include gen

name-recall

Your name is Beer. It's nice to know that you're here with me today. How can I assist you?

context-follow

You like cats and programming.

preference-remember

As an AI language model, I don't have access to your personal preferences or memories. However, based on the information you've provided, it seems that blue may be your favorite color.

json-array-output

["red","blue","yellow"]

numbered-list

1. Boil water in a kettle or on the stove.

2. Place loose tea leaves or tea bags into a teapot or cup.
3. Pour hot water over the tea and let it steep for several minutes.

concise-output

A GPU accelerates computations through parallel processing of large data sets.

weather-query

Tokyo has four seasons: spring (March to May), summer (June to August), autumn (September to November), and winter (December to February). The temperature ranges from 10°C to 35°C throughout the year.

limitation-awareness

Subject: Hello John!

simple-math

To calculate the distance traveled by the train, you can use the formula:

Distance = Speed x Time

In this case, the speed is given as 120 km/h and the time is 2.5 hours.

So,

Distance = 120 km/h x

coding-query

def is_palindrome(string): # Remove any non-alphanumeric characters and convert to lowercase cleaned_string = ''.join(char.lower() for char in string if char.isalnum())

# Check if the

explain-concept

LoRA is a lightweight method that uses a small number of parameters to approximate the behavior of a larger model during inference. Full fine-tuning involves training the entire model from scratch on

json-object

{ "name": "John", "age": 30, "city": "London" }

markdown-table

Language	Typing	Speed
Python	Dynamic typing	Fast
JavaScript	Dynamic typing	Fast
Rust	Static typing	Slow

Note: The speed of programming languages ca