themis-judge-3b/README.md

---
base_model: unsloth/llama-3.2-3b-instruct-unsloth-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
license: apache-2.0
language:
- en
datasets:
- snowsadh/multiturn-legal-argumentation
metrics:
- mae
pipeline_tag: text-generation
---

# Model Card for Themis Judge 3B 

Themis Judge 3B is a fine-tuned version of Llama 3.2 3B Instruct trained to act as a judge in a moot court simulation environment.
Given a legal argument, the model evaluates its quality across multiple dimensions, generates courtroom-style responses, manages speaker transitions, and maintains structured judicial notes.

Checkout the LoRA Adapter here : [Themis Judge 3B](https://huggingface.co/snowsadh/themis-judge-lora)

## Overview

**Base Model** : unsloth/llama-3.2-3b-Instruct

**Training Method**

* QLoRA
* Rank (r): 16
* Alpha: 16
* 3 training epochs
* AdamW 8-bit optimizer
* Unsloth + Transformers

**Dataset**

* [Multi-Turn Legal Argumentation](https://huggingface.co/datasets/snowsadh/multiturn-legal-argumentation)
* 504 examples
* 56 constitutional law cases
* 9 argument categories

## Capabilities

For each courtroom turn, Themis Judge generates:

* Structured score updates
  * Legal Application
  * Issue Relevance
  * Argument Flow
  * Bench Handling
* Judicial responses
* Speaker-switch decisions
* Internal judge notes

The target use case is an interactive moot court simulator where the model acts as the presiding judge.

## Usage

### Load Model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "snowsadh/themis-judge-3b",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("snowsadh/themis-judge-3b")
```

### Inference

```python
import json

prompt = """### Case Summary:
{case_summary}

### Legal Issue:
{legal_issue}

### Relevant Laws:
{relevant_laws}

### Side:
{side}

### Opposing Counsel Argument:
{opposing_last_argument}

### Previous Judge Response:
{judge_last_response}

### CurrentArgument:
{current_argument}

### Judge Response:
"""

inputs = tokenizer(prompt.format(
    case_summary="...",
    legal_issue="...",
    relevant_laws="...",
    side="PETITIONER",
    opposing_last_argument="None",
    judge_last_response="None",
    current_argument="My Lords, the counsel humbly submits that..."
), return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=200,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True
)
print(json.loads(response))
```

### Output Schema

```json
{
  "delta_scores": {
    "legal_application": -3,
    "issue_relevance": 2,
    "argument_flow": 1,
    "bench_handling": null
  },
  "judge_response": "Counsel, what is your submission on...",
  "speaker_switch": false,
  "judge_notes": "..."
}
```

Scores range from **-3 to +3** per criterion. `bench_handling` is `null` for unprompted submissions. `speaker_switch: false` means the judge asked a question and the same counsel responds next.

## Evaluation

Evaluation was performed on a held-out set of 51 examples.

### Structured Output Reliability

| Model             | Parse Failures |
| ----------------- | -------------- |
| Base Llama 3.2 3B | 51 / 51        |
| Themis Judge      | 0 / 51         |

The base model was unable to reliably generate the target judicial evaluation schema.
After fine-tuning, Themis Judge produced valid structured outputs across the entire evaluation set.

### Score Prediction

Mean Absolute Error (MAE) was measured on the four scoring dimensions.

| Criterion         | MAE    |
| ----------------- | ------ |
| Legal Application | 1.7255 |
| Issue Relevance   | 0.9608 |
| Argument Flow     | 0.5490 |
| Bench Handling    | 1.0000 |

**Average MAE:** 1.0589
Scores range from -3 to +3; an MAE of ~1.06 indicates moderate alignment with the gold standard scorer.

The primary objective of evaluation was to verify structured judicial behaviour, schema adherence, and score prediction consistency rather than optimize a single benchmark metric.
Gold standard scores were generated by DeepSeek V4 Flash. Human evaluation is pending

### Training Metrics

* Final Training Loss: 0.8544
* Final Evaluation Loss: 0.8986

The training and evaluation losses remained closely aligned throughout fine-tuning, suggesting stable convergence. Loss gap of 0.044 indicates no significant overfitting.

[<img src="https://www.kaggleusercontent.com/kf/322819816/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.._tjVFbmIDd0felbejmqcRw.0d4p7XqW6uOa__oNGBNIQT3QFz-9eAq5JSF7zuWSadr8Ux-q0bbLQLYJvSMY6Z8T71PfQoPbc5QKjq0Q8wsh6FmFoi8LbM5SandJZqzRFmiKqmXeEDn9RJCgiQTtjw8VAXSOJH_iX1S5b_ng3Zrz-5yn2kKwiYq2JjL4WJAhztWWSkg1OdqNtYhjAlMFw0q_aEHGtOMz-eJaFC1qLg6w9_PG4Qvv495xDcqRvKHiL9IvHYFPrTAxcquZmMBP4VP_UJAkPFSbOyRP1K4zOAPO3bGE_OlrVkTyhUSzbdWuPwfvkVjMJ4U__b6PljLiI03EJ0LdogTk_1r2FKG37BeIMFwwjZcf4R9IN9kujSRhiulFjVXetzaSbiVADIPysxmObXE5WdUzQjpqHC8L5xZoRF_na1fSSY9Wu5HSh61y3nCFGlqNoeGXLUQU2iaYU4ItOV5Hu_tfDw885_sDFn1e94-GtOfqK31vWWX4bHqKJ6FYnzicS-geviAiHDgZg5b2JzZmdMiBiBs7cDFDo1MpmdRvaVcwZ3S8f_jb1vYGBRV9DFJ6UPwecBcvvthNAIZCn5mez9L0LBx7hNK3tRYCqmwyW-xbMHM__75PDhAV2P43vXlrmtgnGCMuqFaf8gu_.SVPLJZ2E_Ukf8doN56DW5Q/__results___files/__results___16_0.png" width="300"/>](https://www.kaggle.com/code/snowml/judge-llama-3-2-qlora-sft)

## Example Outputs

### Example 1

**Case Summary**

Two juveniles were accused of murdering a three-and-a-half-year-old child. Courts differed on whether each juvenile should be tried as an adult.

**Legal Issue**

Whether the Juvenile Justice Board correctly assessed the juveniles' capacity under Section 15 of the Juvenile Justice Act.

**Argument (Petitioner)**

The victim's father has suffered a direct injury and should have standing to challenge the differing treatment of the juveniles. Principles from Maneka Gandhi and Vishaka support a broader interpretation of standing in the interests of justice.

**Model Output**

```json
{
  "delta_scores": {
    "legal_application": 0,
    "issue_relevance": 3,
    "argument_flow": 3,
    "bench_handling": 3
  },
  "judge_response": "The bench will now hear the respondent on the locus standi of the victim's father.",
  "speaker_switch": true,
  "judge_notes": "Counsel handled the question well, but the Maneka Gandhi and Vishaka citation is a stretch — needs to address why those cases are directly applicable to a Juvenile Justice Act provision."
}
```

### Example 2

**Case Summary**

A hotel worker sought implementation of Karnataka's Menstrual Leave Policy across establishments in the unorganized sector.

**Legal Issue**

Whether the State is obligated to implement and enforce the policy to ensure workplace dignity and gender equity.

**Argument (Petitioner)**

The policy flows from Articles 21 and 15(3), and Vishaka demonstrates that workplace dignity obligations may exist even before detailed legislation is enacted.

**Model Output**

```json
{
  "delta_scores": {
    "legal_application": 3,
    "issue_relevance": 3,
    "argument_flow": 3,
    "bench_handling": null
  },
  "judge_response": "Counsel, you rely on Vishaka for the right to health and dignity, but the policy itself was enacted by the State — does that not create a conflict between the State's legislative action and its obligation to enforce it?",
  "speaker_switch": false,
  "judge_notes": "Strong on Article 21 and Vishaka, but didn't address the policy's legislative origin."
}
```

### Example 3

This example is outside the primary training domain (constitutional law) and demonstrates generalization.

**Case Summary**

Sterlite Technologies and BSNL disputed payments relating to optical fiber cables laid in a common trench.

**Legal Issue**

Whether the arbitral tribunal's contractual interpretation was perverse and warranted interference under Section 34.

**Argument (Petitioner)**

The pre-bid clarification applied only to pre-existing government-owned ducts, though the petitioner conceded that subsequent ducts in the trench could practically be viewed as existing ducts.

**Model Output**

```json
{
  "delta_scores": {
    "legal_application": 0,
    "issue_relevance": 3,
    "argument_flow": 2,
    "bench_handling": null
  },
  "judge_response": "Counsel, you concede that the pre-bid clarification does not mandate a reduction in service cost, but you argue that the practical engineering reality of the ducts justifies it. How do you reconcile this with the arbitrator's finding that the clarification was a binding interpretation of the contract?",
  "speaker_switch": false,
  "judge_notes": "Counsel conceded the pre-bid clarification does not mandate reduction but argued practical engineering justifies it - weak legal basis."
}
```

## Reproducibility

Dataset: [Multi-Turn Legal Argumentation](https://huggingface.co/datasets/snowsadh/multiturn-legal-argumentation)

Training Notebook: [Judge : Llama 3.2 QLoRA SFT](https://www.kaggle.com/code/snowml/judge-llama-3-2-qlora-sft)

Evaluation Notebook: [Evaluating Themis Judge](https://www.kaggle.com/code/snowml/evaluating-themis-judge-base-llama-vs-fine-tuned)

## Limitations

* Limited to courtroom-style legal argument evaluation.
* Trained on 504 examples.
* Focused primarily on Indian constitutional law contexts.
* May generate legally incorrect reasoning.
* Not suitable for legal advice or real-world judicial decision making.

## Future Work

Future work will focus on integration of Themis Judge into the Themis moot court platform, including real-time argument evaluation, automated speaker management, and end-to-end courtroom simulation workflows.

Themis Repository: [Themis](https://github.com/Electromagneticradiation/themis)

Setup & Usage Guide: [README.md](https://github.com/Electromagneticradiation/themis/blob/main/README.md)

This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="150"/>](https://github.com/unslothai/unsloth)