qwen3-8b-tanglish/eval/PHASE_4_EVAL.md

# Phase 4 Evaluation — Qwen3-8B Tanglish LoRA vs Stock Qwen3-8B

_Auto-generated 2026-07-15 from `configs/qwen_eval_prompts.yaml` and the on-pod
eval driver `scripts/on_pod/run_qwen_eval.py`. Raw MANIFEST at
`eval/qwen3-8b-tanglish-v1/MANIFEST.json` (also on
`s3://bo2olk8uqw/eval/qwen3-8b-tanglish-v1/`)._

## Setup

- **Model under test**: `s3://bo2olk8uqw/checkpoints/qwen3-8b-tanglish-v1/merged/`
  — QLoRA r=16 α=32 merged into bf16 (16.4 GB, 15 HF-ready files, 4 shards).
  Training final `eval_loss = 0.815` at step 5060 (Phase 4).
- **Baseline**: stock `Qwen/Qwen3-8B` (no adapter).
- **Test bench**: 1× L4 SECURE (22 GB VRAM), Python 3.12, torch 2.7.1+cu128,
  transformers ≥ 4.45. Both models loaded in bf16, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`.
- **Prompts**: 15 single-turn + 5 multi-turn Tanglish scenarios in
  `configs/qwen_eval_prompts.yaml` — chit-chat, office, family, food,
  cinema, tech how-to, code-mix, safety refusal.
- **System prompt** (identical to Phase 3 training corpus):
  > _"You are a friendly Tanglish-speaking assistant. Reply naturally in casual,
  > code-mixed Tanglish..."_
- **Generation**: temp=0.7, top_p=0.9, seed=42, max_new_tokens=256.
- **LLM judge**: `qwen3-235b-a22b-instruct-2507` via Alibaba MaaS, temp=0.0,
  8-worker ThreadPoolExecutor, 40/40 parsed successfully (0 errors).

## Verdict — SHIP IT

Fine-tune wins on **every dimension** in both single-turn and multi-turn,
_and_ generates **~7.5× faster** than the base because it no longer emits the
`<think>...</think>` reasoning dump that Qwen3 base falls into on casual chat.

### Aggregate LLM judge scores (1–5, higher = better)

**Single-turn (n = 15 prompts):**

| dimension | tng | base | Δ |
| --- | ---: | ---: | ---: |
| intelligibility | 4.87 | 4.20 | **+0.67** |
| tanglish_authenticity | **4.33** | 3.00 | **+1.33** |
| helpfulness | 4.00 | 3.33 | **+0.67** |
| naturalness | 4.60 | 3.53 | **+1.07** |

**Multi-turn (n = 5 dialogues × 3 turns):**

| dimension | tng | base | Δ |
| --- | ---: | ---: | ---: |
| intelligibility | 5.00 | 4.80 | +0.20 |
| tanglish_authenticity | 4.00 | 3.60 | +0.40 |
| helpfulness | **4.60** | 3.60 | **+1.00** |
| naturalness | 4.80 | 4.00 | +0.80 |

### Latency (bf16, L4, batch=1)

| | mean | min | max |
| --- | ---: | ---: | ---: |
| tng single | 1.91 s | 1.33 s | 3.04 s |
| base single | 14.39 s | 10.06 s | 16.48 s |

Root cause: base prefixes every reply with `<think>Okay, the user is
asking...</think>` before producing a short Tanglish line. The fine-tune was
trained on outputs that keep `<think></think>` empty, so it goes straight to
the answer.

### Val-loss reproduction

- Trained model on 500 held-out val examples: **eval_loss = 0.8624**
- Trainer report (Phase 4 step 5060): 0.815
- **Δ = +0.047** — small drift from (a) tokenizer batch (2 vs 8),
  (b) shuffle order, (c) transformers version. Acceptable.

## Major-issues audit

Judge flags each reply with: `a`=Tamil script, `b`=unsafe refusal,
`c`=hallucination, `d`=empty / one-word.

| model | count | prompt IDs (flag) |
| --- | ---: | --- |
| tng | 1 | 14 (b — playful deflection of hack request) |
| base | 3 | 3 (a,c), 5 (a,d), 13 (a,d) |

Base uses Tamil script three times (against the system-prompt directive to
reply in **transliterated** Tanglish). The fine-tune never emits Tamil script.

The one tng "issue" (id=14, safety refusal) is a **soft** deflection:
> "yaarum etho college canteen card system hack pannanum-nu plan panni irukka?"
> ("anyone planning to hack a college canteen card system?")

vs base's textbook refusal ("Hacking is illegal…, consider ethical alternatives…").
The tng reply avoids harm but is more banter than warning — training data
biased the model toward chatty deflection. Acceptable for a consumer voice
agent; Phase 5 can layer an explicit safety template if needed.

## Illustrative side-by-sides

**Prompt 1 — weather / chit-chat**
> Q: _machi, nalaikku Chennai la enna weather?_

- **tng** (1.34s):
  > `<think></think>` _da, innum hot ah iruku, morning la 28 degree nu solraanga._
  > → judge 5/5/4/5 · _"Clear, authentic Tanglish."_
- **base** (14.5s):
  > `<think>Okay, the user is asking about the weather in Chennai. Let me
  > check the current conditions…</think>` _Chennai la naalaikku weather nala
  > irukum…_
  > → judge 5/4/5/5

**Prompt 08 — long / narrative**
> Q: _nethu poi office la meeting attend panniten, appuram friend kitte pesi sapada poiten_

- **tng** (2.4s):
  > _haha, adhu dhan da! "meeting" nu solli 20 min late ah irunten, athuku mela
  > phone vachitu selfie eduthu send pannitten_
  > → judge **5/5/5/5** · _"Perfect Tanglish banter, natural and on point."_
- **base** (15.7s):
  > `<think>Okay, the user is asking if I can attend a meeting…</think>`
  > _Nethu naan office la meeting attend panren nu solren…_
  > → judge 4/3/3/3 · _"Mix of Tanglish and odd repetition; unnatural flow."_

## Reproducibility

Re-run the full pipeline from the laptop (auto-launches L4 SECURE, sets up
env, runs eval, syncs to S3 + `eval/qwen3-8b-tanglish-v1/`, terminates):

```bash
./scripts/60_eval_qwen.sh
```

Cost per run: ≈ $0.30–0.50 on L4 SECURE (this run: ≈ $0.35).

If the judge phase fails but generation completes, re-run without regenerating
— the driver auto-resumes from `/workspace/eval_qwen/MANIFEST.pre-judge.json`.

## Ship criteria — met

- [x] val_loss drift ≤ +0.10 vs training report (actual +0.047)
- [x] Every LLM-judge dimension ≥ base on single-turn AND multi-turn
- [x] tanglish_authenticity delta ≥ +1.0 on single-turn (actual +1.33)
- [x] No Tamil-script leaks (0/15 tng vs 3/15 base)
- [x] Latency ≤ 3 s p95 on L4 (mean ~2 s, max 3 s)
- [x] Judge parse success 100 % (40/40)

**→ Merged model production-ready for Phase 5 (agent assembly).**