初始化项目,由ModelHub XC社区提供模型
Model: Nanthasit/sakthai-coder-browser Source: Original Platform
This commit is contained in:
93
.eval_results/benchmark-20260731_052122.yaml
Normal file
93
.eval_results/benchmark-20260731_052122.yaml
Normal file
@@ -0,0 +1,93 @@
|
||||
model: Nanthasit/sakthai-coder-browser
|
||||
benchmark_ts: '2026-07-31T05:50:33Z'
|
||||
verdict: MODEL_BROKEN_BIAS_CORRUPTION
|
||||
benchmark_valid: false
|
||||
backend: llama.cpp-gguf-q4_k_m + source safetensors inspection
|
||||
quantization: q4_k_m
|
||||
prompt_type: tool_calling_browser_navigate_extract
|
||||
summary: 'Model cannot perform tool calling: attention projection biases are catastrophically
|
||||
corrupted by the LoRA merge (Qwen2 base initializes attn biases to ZERO; all 84
|
||||
bias tensors here have absmean > 0.01, layer-0 k_proj absmean 27.7 / max 354). Degenerate
|
||||
output: whitespace-loop at temp<=0.7 on all 3 seeds (0 tool calls, 0 valid JSON);
|
||||
only at temp 1.5 does it emit any text (''Hi'' on a trivial prompt). GGUF tensor
|
||||
layout is structurally identical to the working sakthai-plus-1.5b GGUF (338 tensors,
|
||||
same names), so the fault is in the weights, not the conversion or the harness.'
|
||||
trials:
|
||||
- seed: 7
|
||||
temp: 0.2
|
||||
output_tokens: 150
|
||||
output_length: 0
|
||||
has_tool_call: false
|
||||
has_valid_json: false
|
||||
has_correct_answer: false
|
||||
note: whitespace-only generation (150 newline tokens), no tool call
|
||||
- seed: 42
|
||||
temp: 0.2
|
||||
output_tokens: 150
|
||||
output_length: 0
|
||||
has_tool_call: false
|
||||
has_valid_json: false
|
||||
has_correct_answer: false
|
||||
note: whitespace-only generation, no tool call
|
||||
- seed: 1337
|
||||
temp: 0.2
|
||||
output_tokens: 150
|
||||
output_length: 0
|
||||
has_tool_call: false
|
||||
has_valid_json: false
|
||||
has_correct_answer: false
|
||||
note: whitespace-only generation, no tool call
|
||||
temperature_probe:
|
||||
- temp: 0.2
|
||||
prompt: full browser prompt
|
||||
result: blank/whitespace loop
|
||||
- temp: 0.4
|
||||
prompt: full browser prompt
|
||||
result: blank/whitespace loop
|
||||
- temp: 0.7
|
||||
prompt: full browser prompt
|
||||
result: blank/whitespace loop
|
||||
- temp: 1.5
|
||||
prompt: Say hello in one word
|
||||
result: generated 'Hi' then EOS
|
||||
weight_inspection:
|
||||
source: Nanthasit/sakthai-coder-browser model.safetensors (3.09 GB, 338 tensors,
|
||||
1.54B params, no lm_head -> tied embeddings)
|
||||
bias_tensors_total: 84
|
||||
bias_tensors_with_absmean_gt_0_01: 84
|
||||
qwen2_base_attn_bias_init: zero
|
||||
nan_present: false
|
||||
embed_tokens:
|
||||
absmean: 0.0136
|
||||
absmax: 0.295
|
||||
normal: true
|
||||
sample_corrupted_biases:
|
||||
- tensor: model.layers.0.self_attn.k_proj.bias
|
||||
absmean: 27.6991
|
||||
absmax: 354.0
|
||||
- tensor: model.layers.0.self_attn.q_proj.bias
|
||||
absmean: 1.1716
|
||||
absmax: 28.88
|
||||
- tensor: model.layers.1.self_attn.k_proj.bias
|
||||
absmean: 3.284
|
||||
absmax: 111.0
|
||||
- tensor: model.layers.1.self_attn.q_proj.bias
|
||||
absmean: 0.5512
|
||||
absmax: 10.62
|
||||
- tensor: model.layers.10.self_attn.k_proj.bias
|
||||
absmean: 0.3383
|
||||
absmax: 6.62
|
||||
- tensor: model.layers.14.self_attn.q_proj.bias
|
||||
absmean: 0.5422
|
||||
absmax: 13.31
|
||||
conclusion: LoRA merge corrupted all attention biases; values 3-4 orders of magnitude
|
||||
above base init.
|
||||
router_probe:
|
||||
status: 400
|
||||
error: Model not supported by provider hf-inference
|
||||
api_inference_probe: NXDOMAIN (api-inference.huggingface.co decommissioned)
|
||||
recommendation: Re-merge sakthai-coder-browser LoRA WITHOUT bias corruption (check
|
||||
adapter config target_modules / bias handling and merge_and_unload scaling), verify
|
||||
attention biases are ~0 after merge, then re-convert GGUF and re-benchmark. Until
|
||||
then the model is not usable for inference. Model card should carry a BROKEN-WEIGHTS
|
||||
warning.
|
||||
Reference in New Issue
Block a user