94 lines
3.2 KiB
YAML
94 lines
3.2 KiB
YAML
model: Nanthasit/sakthai-coder-browser
|
|
benchmark_ts: '2026-07-31T05:50:33Z'
|
|
verdict: MODEL_BROKEN_BIAS_CORRUPTION
|
|
benchmark_valid: false
|
|
backend: llama.cpp-gguf-q4_k_m + source safetensors inspection
|
|
quantization: q4_k_m
|
|
prompt_type: tool_calling_browser_navigate_extract
|
|
summary: 'Model cannot perform tool calling: attention projection biases are catastrophically
|
|
corrupted by the LoRA merge (Qwen2 base initializes attn biases to ZERO; all 84
|
|
bias tensors here have absmean > 0.01, layer-0 k_proj absmean 27.7 / max 354). Degenerate
|
|
output: whitespace-loop at temp<=0.7 on all 3 seeds (0 tool calls, 0 valid JSON);
|
|
only at temp 1.5 does it emit any text (''Hi'' on a trivial prompt). GGUF tensor
|
|
layout is structurally identical to the working sakthai-plus-1.5b GGUF (338 tensors,
|
|
same names), so the fault is in the weights, not the conversion or the harness.'
|
|
trials:
|
|
- seed: 7
|
|
temp: 0.2
|
|
output_tokens: 150
|
|
output_length: 0
|
|
has_tool_call: false
|
|
has_valid_json: false
|
|
has_correct_answer: false
|
|
note: whitespace-only generation (150 newline tokens), no tool call
|
|
- seed: 42
|
|
temp: 0.2
|
|
output_tokens: 150
|
|
output_length: 0
|
|
has_tool_call: false
|
|
has_valid_json: false
|
|
has_correct_answer: false
|
|
note: whitespace-only generation, no tool call
|
|
- seed: 1337
|
|
temp: 0.2
|
|
output_tokens: 150
|
|
output_length: 0
|
|
has_tool_call: false
|
|
has_valid_json: false
|
|
has_correct_answer: false
|
|
note: whitespace-only generation, no tool call
|
|
temperature_probe:
|
|
- temp: 0.2
|
|
prompt: full browser prompt
|
|
result: blank/whitespace loop
|
|
- temp: 0.4
|
|
prompt: full browser prompt
|
|
result: blank/whitespace loop
|
|
- temp: 0.7
|
|
prompt: full browser prompt
|
|
result: blank/whitespace loop
|
|
- temp: 1.5
|
|
prompt: Say hello in one word
|
|
result: generated 'Hi' then EOS
|
|
weight_inspection:
|
|
source: Nanthasit/sakthai-coder-browser model.safetensors (3.09 GB, 338 tensors,
|
|
1.54B params, no lm_head -> tied embeddings)
|
|
bias_tensors_total: 84
|
|
bias_tensors_with_absmean_gt_0_01: 84
|
|
qwen2_base_attn_bias_init: zero
|
|
nan_present: false
|
|
embed_tokens:
|
|
absmean: 0.0136
|
|
absmax: 0.295
|
|
normal: true
|
|
sample_corrupted_biases:
|
|
- tensor: model.layers.0.self_attn.k_proj.bias
|
|
absmean: 27.6991
|
|
absmax: 354.0
|
|
- tensor: model.layers.0.self_attn.q_proj.bias
|
|
absmean: 1.1716
|
|
absmax: 28.88
|
|
- tensor: model.layers.1.self_attn.k_proj.bias
|
|
absmean: 3.284
|
|
absmax: 111.0
|
|
- tensor: model.layers.1.self_attn.q_proj.bias
|
|
absmean: 0.5512
|
|
absmax: 10.62
|
|
- tensor: model.layers.10.self_attn.k_proj.bias
|
|
absmean: 0.3383
|
|
absmax: 6.62
|
|
- tensor: model.layers.14.self_attn.q_proj.bias
|
|
absmean: 0.5422
|
|
absmax: 13.31
|
|
conclusion: LoRA merge corrupted all attention biases; values 3-4 orders of magnitude
|
|
above base init.
|
|
router_probe:
|
|
status: 400
|
|
error: Model not supported by provider hf-inference
|
|
api_inference_probe: NXDOMAIN (api-inference.huggingface.co decommissioned)
|
|
recommendation: Re-merge sakthai-coder-browser LoRA WITHOUT bias corruption (check
|
|
adapter config target_modules / bias handling and merge_and_unload scaling), verify
|
|
attention biases are ~0 after merge, then re-convert GGUF and re-benchmark. Until
|
|
then the model is not usable for inference. Model card should carry a BROKEN-WEIGHTS
|
|
warning.
|