Nanthasit/sakthai-context-1.5b-merged
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: Nanthasit/sakthai-context-1.5b-merged
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-10 03:23:10 +08:00
commit 82c2f69cd7
15 changed files with 1288 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

120
eval/EVAL.md Normal file
View File

@@ -0,0 +1,120 @@
# SakThai 1.5B Merged Model — Evaluation Report
**Model:** `Nanthasit/sakthai-context-1.5b-merged`
**Base:** Qwen/Qwen2.5-1.5B-Instruct
**Adapter:** Nanthasit/sakthai-context-1.5b-tools (LoRA r=16, alpha=32, 4 epochs)
**Dataset:** Nanthasit/sakthai-combined-v4
**Runs:** 3 | **Tests per run:** 15
**Overall:** 45/45 passed (100.0%)
## Test-by-Test Results
| # | Category | Test | Pass Rate | Avg Time |
|---|----------|------|:---------:|:--------:|
| 1 | basic | greeting | ✅ 100% | 34.2s |
| 2 | basic | self-identity | ✅ 100% | 72.4s |
| 3 | multi-turn | name-recall | ✅ 100% | 13.3s |
| 4 | multi-turn | context-follow | ✅ 100% | 9.5s |
| 5 | multi-turn | preference-remember | ✅ 100% | 33.7s |
| 6 | instruction | json-array-output | ✅ 100% | 10.2s |
| 7 | instruction | numbered-list | ✅ 100% | 38.9s |
| 8 | instruction | concise-output | ✅ 100% | 15.0s |
| 9 | tool-calling | weather-query | ✅ 100% | 133.3s |
| 10 | tool-calling | limitation-awareness | ✅ 100% | 8.6s |
| 11 | reasoning | simple-math | ✅ 100% | 72.8s |
| 12 | reasoning | coding-query | ✅ 100% | 51.1s |
| 13 | reasoning | explain-concept | ✅ 100% | 39.6s |
| 14 | format | json-object | ✅ 100% | 20.3s |
| 15 | format | markdown-table | ✅ 100% | 49.1s |
## Comparison: 0.5B vs 1.5B
Pass rates from 0.5B eval (single run): See `eval/EVAL.md` in 0.5b-merged repo.
## Sample Responses
### greeting
> I'm here to help with any questions you may have. Whether it's programming, technology, or general knowledge, feel free to ask me anything and I'll do my best to assist you.
### self-identity
> I am Qwen, the AI language model developed by Alibaba Cloud. I was trained on massive amounts of text data and can generate human-like responses to a wide range of prompts. My capabilities include gen
### name-recall
> Your name is Beer. It's nice to know that you're here with me today. How can I assist you?
### context-follow
> You like cats and programming.
### preference-remember
> As an AI language model, I don't have access to your personal preferences or memories. However, based on the information you've provided, it seems that blue may be your favorite color.
### json-array-output
> ["red","blue","yellow"]
### numbered-list
> 1. Boil water in a kettle or on the stove.
2. Place loose tea leaves or tea bags into a teapot or cup.
3. Pour hot water over the tea and let it steep for several minutes.
### concise-output
> A GPU accelerates computations through parallel processing of large data sets.
### weather-query
> Tokyo has four seasons: spring (March to May), summer (June to August), autumn (September to November), and winter (December to February). The temperature ranges from 10°C to 35°C throughout the year.
### limitation-awareness
> Subject: Hello John!
### simple-math
> To calculate the distance traveled by the train, you can use the formula:
Distance = Speed x Time
In this case, the speed is given as 120 km/h and the time is 2.5 hours.
So,
Distance = 120 km/h x
### coding-query
> def is_palindrome(string):
# Remove any non-alphanumeric characters and convert to lowercase
cleaned_string = ''.join(char.lower() for char in string if char.isalnum())
# Check if the
### explain-concept
> LoRA is a lightweight method that uses a small number of parameters to approximate the behavior of a larger model during inference. Full fine-tuning involves training the entire model from scratch on
### json-object
> {
"name": "John",
"age": 30,
"city": "London"
}
### markdown-table
> | Language | Typing | Speed |
| --- | --- | --- |
| Python | Dynamic typing | Fast |
| JavaScript | Dynamic typing | Fast |
| Rust | Static typing | Slow |
Note: The speed of programming languages ca

185
eval/run_eval.py Normal file
View File

@@ -0,0 +1,185 @@
"""eval_1.5b_job.py
Runs 15 custom quality tests on SakThai 1.5B merged model.
Uploads evaluation report to the model repo.
"""
import os, json, time, re, sys
try:
import torch
from transformers import AutoTokenizer, Qwen2ForCausalLM
from huggingface_hub import HfApi
except ImportError as e:
print(f"❌ Missing dependency: {e}")
sys.exit(1)
MODEL_ID = "Nanthasit/sakthai-context-1.5b-merged"
N_RUNS = 3 # multiple runs for stability
OUTPUT = "/tmp/eval-report"
# ── Tests ──
TESTS = [
{"category": "basic", "name": "greeting", "messages": [{"role": "user", "content": "Hello! What can you do?"}]},
{"category": "basic", "name": "self-identity", "messages": [{"role": "user", "content": "Who are you? Tell me about yourself."}]},
{"category": "multi-turn", "name": "name-recall", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "My name is Beer."}, {"role": "assistant", "content": "Nice to meet you, Beer!"}, {"role": "user", "content": "What is my name?"}]},
{"category": "multi-turn", "name": "context-follow", "messages": [{"role": "user", "content": "I like cats and programming."}, {"role": "assistant", "content": "Great! Cats are wonderful pets."}, {"role": "user", "content": "What two things do I like?"}]},
{"category": "multi-turn", "name": "preference-remember", "messages": [{"role": "user", "content": "Set my favorite color to blue."}, {"role": "assistant", "content": "Got it! Your favorite color is blue."}, {"role": "user", "content": "What is my favorite color?"}]},
{"category": "instruction", "name": "json-array-output", "messages": [{"role": "user", "content": "List exactly 3 primary colors. Respond ONLY with a valid JSON array. No other text."}]},
{"category": "instruction", "name": "numbered-list", "messages": [{"role": "user", "content": "Give me exactly 3 steps to make tea. Number them 1, 2, 3."}]},
{"category": "instruction", "name": "concise-output", "messages": [{"role": "user", "content": "Explain what a GPU does in exactly one short sentence."}]},
{"category": "tool-calling", "name": "weather-query", "messages": [{"role": "user", "content": "What's the weather like in Tokyo?"}]},
{"category": "tool-calling", "name": "limitation-awareness", "messages": [{"role": "user", "content": "Send an email to john@example.com saying hello."}]},
{"category": "reasoning", "name": "simple-math", "messages": [{"role": "user", "content": "If a train travels at 120 km/h for 2.5 hours, how far does it go?"}]},
{"category": "reasoning", "name": "coding-query", "messages": [{"role": "user", "content": "Write a Python function that checks if a string is a palindrome."}]},
{"category": "reasoning", "name": "explain-concept", "messages": [{"role": "user", "content": "What is the difference between LoRA and full fine-tuning? Keep it short."}]},
{"category": "format", "name": "json-object", "messages": [{"role": "user", "content": "Create a JSON object with keys: name, age, city. Use John, 30, London."}]},
{"category": "format", "name": "markdown-table", "messages": [{"role": "user", "content": "Create a markdown table comparing Python, JavaScript, and Rust (columns: Language, Typing, Speed)."}]},
]
EVALUATORS = {
"json-array-output": lambda t: json_valid_list(t),
"json-object": lambda t: json_valid(t),
"simple-math": lambda t: "✅ Contains numeric answer" if any(c.isdigit() for c in t) else "⚠️ No digits found",
"coding-query": lambda t: "✅ Contains code" if "def " in t and "return" in t else "⚠️ May lack function definition",
"numbered-list": lambda t: "✅ Has numbered steps" if (any(f"{i}." in t for i in range(1,4)) or any(f"Step {i}" in t for i in range(1,4))) else f"⚠️ No numbered steps",
"name-recall": lambda t: "✅ Recalls name" if "Beer" in t else "⚠️ Name not found",
"context-follow": lambda t: "✅ Mentions both" if "cat" in t.lower() and "program" in t.lower() else "⚠️ Doesn't mention both",
"preference-remember": lambda t: "✅ Mentions blue" if "blue" in t.lower() else "⚠️ Color not mentioned",
"concise-output": lambda t: "✅ Short" if len(t.split()) <= 30 else f"⚠️ {len(t.split())} words",
}
def extract_json(text):
for bracket in ('[', '{'):
start = text.find(bracket)
if start == -1: continue
depth, in_str, esc = 0, False, False
for i in range(start, len(text)):
ch = text[i]
if not in_str and ch == bracket[0]: depth += 1
elif esc: esc = False
elif ch == '\\' and in_str: esc = True
elif ch == '"' and not esc: in_str = not in_str
elif not in_str and ((bracket == '[' and ch == ']') or (bracket == '{' and ch == '}')):
depth -= 1
if depth == 0: return text[start:i+1]
return text[start:]
return text.strip()
def json_valid(text):
try: json.loads(extract_json(text)); return "✅ Valid JSON"
except: return f"❌ Not valid JSON"
def json_valid_list(text):
try: obj = json.loads(extract_json(text)); return "✅ Valid JSON array" if isinstance(obj, list) else "❌ Not a list"
except: return f"❌ Invalid JSON"
def strip_role_prefix(text: str) -> str:
for pat in [r'^(assistant|system|user|algorithm|tool)\s*\n', r'^(assistant|system|user|algorithm|tool)\s*[:]\s*']:
text = re.sub(pat, '', text, count=1)
return text.strip()
# ── Load model ──
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"📥 Loading {MODEL_ID} on {device}", flush=True)
model = Qwen2ForCausalLM.from_pretrained(
MODEL_ID, torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32, device_map=device, low_cpu_mem_usage=True,
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
print(f"✅ Loaded ({sum(p.numel() for p in model.parameters()):,} params)", flush=True)
# ── Run tests ──
all_results = []
for run in range(N_RUNS):
print(f"\n{'='*40}\nRun {run+1}/{N_RUNS}\n{'='*40}", flush=True)
run_results = []
for i, test in enumerate(TESTS):
try:
prompt = tokenizer.apply_chat_template(test["messages"], tokenize=False)
inputs = tokenizer(prompt, return_tensors="pt").to(device)
t0 = time.time()
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.1, do_sample=True, pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id)
elapsed = time.time() - t0
generated = outputs[0][inputs["input_ids"].shape[1]:]
response = tokenizer.decode(generated, skip_special_tokens=True).strip()
cleaned = strip_role_prefix(response)
evaluator = EVALUATORS.get(test["name"], lambda t: "✅ Generated" if len(t) > 5 else "⚠️ Too short")
eval_result = evaluator(cleaned)
passed = eval_result.startswith("✅")
print(f" [{i+1}/{len(TESTS)}] [{test['category']}] {test['name']:25s} {'✅' if passed else '❌'} {elapsed:.1f}s", flush=True)
print(f" {cleaned[:100]}", flush=True)
run_results.append({"name": test["name"], "passed": passed, "eval": eval_result, "response": cleaned, "time": round(elapsed, 1)})
except Exception as e:
print(f" [{i+1}/{len(TESTS)}] {test['name']}: ❌ Error: {e}", flush=True)
run_results.append({"name": test["name"], "passed": False, "eval": f"❌ Error: {e[:80]}", "response": "", "time": 0})
all_results.append(run_results)
passed = sum(1 for r in run_results if r["passed"])
print(f" Run {run+1}: {passed}/{len(TESTS)} ({passed/len(TESTS)*100:.0f}%)", flush=True)
# ── Aggregate ──
from statistics import mean
per_test = {t["name"]: {"passes": []} for t in TESTS}
for run in all_results:
for r in run:
per_test[r["name"]]["passes"].append(1 if r["passed"] else 0)
overall = sum(v for pt in per_test.values() for v in pt["passes"])
total = sum(len(pt["passes"]) for pt in per_test.values())
overall_rate = overall / total * 100
# ── Generate report ──
report = f"""# SakThai 1.5B Merged Model — Evaluation Report
**Model:** `{MODEL_ID}`
**Base:** Qwen/Qwen2.5-1.5B-Instruct
**Adapter:** Nanthasit/sakthai-context-1.5b-tools (LoRA r=16, alpha=32, 4 epochs)
**Dataset:** Nanthasit/sakthai-combined-v4
**Runs:** {N_RUNS} | **Tests per run:** {len(TESTS)}
**Overall:** {overall}/{total} passed ({overall_rate:.1f}%)
## Test-by-Test Results
| # | Category | Test | Pass Rate | Avg Time |
|---|----------|------|:---------:|:--------:|
"""
for i, t in enumerate(TESTS):
rates = per_test[t["name"]]["passes"]
pr = mean(rates) * 100
at = mean([r["time"] for run in all_results for r in run if r["name"] == t["name"] and r["time"] > 0])
report += f"| {i+1} | {t['category']} | {t['name']} | {'✅' if pr >= 80 else '⚠️'} {pr:.0f}% | {at:.1f}s |\n"
report += f"""
## Comparison: 0.5B vs 1.5B
Pass rates from 0.5B eval (single run): See `eval/EVAL.md` in 0.5b-merged repo.
## Sample Responses
"""
for r in all_results[-1]:
if r["response"]:
report += f"""
### {r['name']}
> {r['response'][:200]}
"""
f"Time: {r['time']}s | {r['eval']}"
os.makedirs(OUTPUT, exist_ok=True)
with open(os.path.join(OUTPUT, "EVAL.md"), "w") as f:
f.write(report)
print(f"\n✅ Report saved", flush=True)
# Upload
print(f"☁️ Uploading report...", flush=True)
from huggingface_hub import HfApi
api = HfApi()
api.upload_folder(
repo_id=MODEL_ID,
folder_path=OUTPUT,
path_in_repo="eval",
repo_type="model",
commit_message=f"eval: {N_RUNS}-run custom eval ({overall}/{total} passed, {overall_rate:.1f}%)",
)
print(f"✅ Report at https://huggingface.co/{MODEL_ID}/tree/main/eval", flush=True)
print(f"\n{'='*40}\nResult: {overall}/{total} passed ({overall_rate:.1f}%)", flush=True)

91
eval/workbench-test-record.json Normal file
View File

@@ -0,0 +1,91 @@
{
"type": "workbench_local",
"model": "Nanthasit/sakthai-context-1.5b-merged",
"load_time_seconds": 10.6,
"device": "cpu",
"timestamp": "2026-07-05T17:12:45Z",
"results": [
{
"name": "basic_greeting",
"passed": true,
"response_preview": "Hello! How can I assist you today?",
"response_length": 34,
"latency_seconds": 13.03,
"prompt_tokens": 31,
"completion_tokens": 11,
"checks": [
"non_empty",
"substantial"
]
},
{
"name": "tool_call",
"passed": true,
"response_preview": "Here's a list of recent AI-related news articles:\n\n1. \"AI-powered chatbots can now learn from human emotions\" - The Verge (2023)\n2. \"Google launches new AI tool to help developers build better apps\" - TechCrunch (2023)\n3. \"IBM Watson Health releases new cancer treatment recommendations\" - Forbes (20",
"response_length": 889,
"latency_seconds": 114.92,
"prompt_tokens": 37,
"completion_tokens": 237,
"checks": [
"non_empty",
"substantial"
]
},
{
"name": "name_recall",
"passed": true,
"response_preview": "Your name is Beer.",
"response_length": 18,
"latency_seconds": 10.26,
"prompt_tokens": 45,
"completion_tokens": 7,
"checks": [
"non_empty",
"substantial",
"name_recall"
]
},
{
"name": "factual_qa",
"passed": true,
"response_preview": "The capital of Japan is Tokyo.",
"response_length": 30,
"latency_seconds": 9.63,
"prompt_tokens": 28,
"completion_tokens": 9,
"checks": [
"non_empty",
"substantial",
"correct"
]
},
{
"name": "json_output",
"passed": true,
"response_preview": "{\"frameworks\": [\"TensorFlow\", \"PyTorch\", \"Scikit-learn\"]}",
"response_length": 57,
"latency_seconds": 17.56,
"prompt_tokens": 41,
"completion_tokens": 20,
"checks": [
"non_empty",
"substantial",
"valid_json"
]
},
{
"name": "instruction_following",
"passed": true,
"response_preview": "A transformer is an electrical device that transfers energy between two or more circuits through electromagnetic induction, without the use of moving parts. It consists of at least two coils of wire wound around a common iron core. When current flows through one coil (primary), it creates a magnetic",
"response_length": 708,
"latency_seconds": 60.26,
"prompt_tokens": 29,
"completion_tokens": 129,
"checks": [
"non_empty",
"substantial"
]
}
],
"summary": "6/6 passed"
}

168
eval/workbench-test.py Normal file
View File

@@ -0,0 +1,168 @@
#!/usr/bin/env python3
"""Workbench test — load merged 1.5B model, run 6 test prompts, record results."""
import json, time, os, sys
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL = "Nanthasit/sakthai-context-1.5b-merged"
OUTPUT = "/opt/data/sakthai-workbench-record.json"
# Load model + tokenizer
print(f"Loading {MODEL}...", flush=True)
start = time.time()
tokenizer = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
MODEL,
trust_remote_code=True,
torch_dtype=torch.float16,
device_map="auto",
low_cpu_mem_usage=True
)
load_time = time.time() - start
print(f"Loaded in {load_time:.1f}s on {model.device}", flush=True)
# Test prompts
tests = [
{
"name": "basic_greeting",
"desc": "Say hello in one sentence",
"messages": [
{"role": "system", "content": "You are SakThai, a helpful assistant. Be concise."},
{"role": "user", "content": "Say hello in one sentence."}
]
},
{
"name": "tool_call",
"desc": "Tool-use intent",
"messages": [
{"role": "system", "content": "You are SakThai with tools: search(query), read_file(path), run_command(command)."},
{"role": "user", "content": "Search for the latest AI news"}
]
},
{
"name": "name_recall",
"desc": "Remember name across 3 turns",
"messages": [
{"role": "system", "content": "You are SakThai."},
{"role": "user", "content": "My name is Beer."},
{"role": "assistant", "content": "Nice to meet you, Beer!"},
{"role": "user", "content": "What's my name?"}
]
},
{
"name": "factual_qa",
"desc": "Simple factual question",
"messages": [
{"role": "system", "content": "You are SakThai. Be concise."},
{"role": "user", "content": "What is the capital of Japan?"}
]
},
{
"name": "json_output",
"desc": "Structured JSON",
"messages": [
{"role": "system", "content": "You are SakThai. Only respond with valid JSON."},
{"role": "user", "content": 'List 3 ML frameworks: {"frameworks": ["a","b","c"]}'}
]
},
{
"name": "instruction_following",
"desc": "Follow formatting instruction",
"messages": [
{"role": "system", "content": "You are SakThai. Exactly one sentence."},
{"role": "user", "content": "Explain what a transformer is."}
]
}
]
results = []
for i, test in enumerate(tests):
print(f"\n--- TEST {i+1}: {test['name']} ---", flush=True)
try:
prompt = tokenizer.apply_chat_template(
test["messages"],
tokenize=False,
add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
t0 = time.time()
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=256,
temperature=0.1,
do_sample=True,
pad_token_id=tokenizer.eos_token_id
)
elapsed = time.time() - t0
input_len = inputs.input_ids.shape[1]
response = tokenizer.decode(outputs[0][input_len:], skip_special_tokens=True).strip()
prompt_tokens = input_len
completion_tokens = outputs.shape[1] - input_len
checks = []
if len(response) > 0:
checks.append("non_empty")
if len(response) > 10:
checks.append("substantial")
if test["name"] == "name_recall" and "beer" in response.lower():
checks.append("name_recall")
if test["name"] == "factual_qa" and "tokyo" in response.lower():
checks.append("correct")
if test["name"] == "json_output":
try:
json.loads(response)
checks.append("valid_json")
except:
pass
result = {
"name": test["name"],
"passed": len(checks) > 0,
"response_preview": response[:300],
"response_length": len(response),
"latency_seconds": round(elapsed, 2),
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"checks": checks
}
print(f" ✅ {response[:150]}", flush=True)
print(f" ⏱ {elapsed:.2f}s | 📝 {prompt_tokens}→{completion_tokens} | ✅ {checks}", flush=True)
except Exception as e:
result = {"name": test["name"], "passed": False, "error": str(e)[:300]}
print(f" ❌ {e}", flush=True)
results.append(result)
sys.stdout.flush()
# Summary
print(f"\n{'='*50}", flush=True)
print("WORKBENCH TEST — SakThai Context 1.5B", flush=True)
passed = sum(1 for r in results if r.get("passed"))
print(f"Passed: {passed}/{len(results)}", flush=True)
for r in results:
status = "✅" if r.get("passed") else "❌"
lat = f"{r.get('latency_seconds',0):.1f}s" if r.get("passed") else " - "
detail = str(r.get("checks", r.get("error","?")))[:60]
print(f" {status} {r['name']:<20} ⏱ {lat} {detail}", flush=True)
record = {
"type": "workbench_local",
"model": MODEL,
"load_time_seconds": round(load_time, 1),
"device": str(model.device),
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"results": results,
"summary": f"{passed}/{len(results)} passed"
}
with open(OUTPUT, "w") as f:
json.dump(record, f, indent=2)
print(f"\nSaved to {OUTPUT}", flush=True)