snowsadh/themis-judge-3b
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: snowsadh/themis-judge-3b
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-01 18:54:17 +08:00
commit bde0b59283
9 changed files with 2744 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

309
README.md Normal file
View File

@@ -0,0 +1,309 @@
---
base_model: unsloth/llama-3.2-3b-instruct-unsloth-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
license: apache-2.0
language:
- en
datasets:
- snowsadh/multiturn-legal-argumentation
metrics:
- mae
pipeline_tag: text-generation
---
# Model Card for Themis Judge 3B
Themis Judge 3B is a fine-tuned version of Llama 3.2 3B Instruct trained to act as a judge in a moot court simulation environment.
Given a legal argument, the model evaluates its quality across multiple dimensions, generates courtroom-style responses, manages speaker transitions, and maintains structured judicial notes.
Checkout the LoRA Adapter here : [Themis Judge 3B](https://huggingface.co/snowsadh/themis-judge-lora)
## Overview
**Base Model** : unsloth/llama-3.2-3b-Instruct
**Training Method**
* QLoRA
* Rank (r): 16
* Alpha: 16
* 3 training epochs
* AdamW 8-bit optimizer
* Unsloth + Transformers
**Dataset**
* [Multi-Turn Legal Argumentation](https://huggingface.co/datasets/snowsadh/multiturn-legal-argumentation)
* 504 examples
* 56 constitutional law cases
* 9 argument categories
## Capabilities
For each courtroom turn, Themis Judge generates:
* Structured score updates
* Legal Application
* Issue Relevance
* Argument Flow
* Bench Handling
* Judicial responses
* Speaker-switch decisions
* Internal judge notes
The target use case is an interactive moot court simulator where the model acts as the presiding judge.
## Usage
### Load Model
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"snowsadh/themis-judge-3b",
torch_dtype=torch.float16,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("snowsadh/themis-judge-3b")
```
### Inference
```python
import json
prompt = """### Case Summary:
{case_summary}
### Legal Issue:
{legal_issue}
### Relevant Laws:
{relevant_laws}
### Side:
{side}
### Opposing Counsel Argument:
{opposing_last_argument}
### Previous Judge Response:
{judge_last_response}
### CurrentArgument:
{current_argument}
### Judge Response:
"""
inputs = tokenizer(prompt.format(
case_summary="...",
legal_issue="...",
relevant_laws="...",
side="PETITIONER",
opposing_last_argument="None",
judge_last_response="None",
current_argument="My Lords, the counsel humbly submits that..."
), return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=200,
temperature=0.7,
do_sample=True,
pad_token_id=tokenizer.eos_token_id
)
response = tokenizer.decode(
outputs[0][inputs["input_ids"].shape[1]:],
skip_special_tokens=True
)
print(json.loads(response))
```
### Output Schema
```json
{
"delta_scores": {
"legal_application": -3,
"issue_relevance": 2,
"argument_flow": 1,
"bench_handling": null
},
"judge_response": "Counsel, what is your submission on...",
"speaker_switch": false,
"judge_notes": "..."
}
```
Scores range from **-3 to +3** per criterion. `bench_handling` is `null` for unprompted submissions. `speaker_switch: false` means the judge asked a question and the same counsel responds next.
## Evaluation
Evaluation was performed on a held-out set of 51 examples.
### Structured Output Reliability
| Model | Parse Failures |
| ----------------- | -------------- |
| Base Llama 3.2 3B | 51 / 51 |
| Themis Judge | 0 / 51 |
The base model was unable to reliably generate the target judicial evaluation schema.
After fine-tuning, Themis Judge produced valid structured outputs across the entire evaluation set.
### Score Prediction
Mean Absolute Error (MAE) was measured on the four scoring dimensions.
| Criterion | MAE |
| ----------------- | ------ |
| Legal Application | 1.7255 |
| Issue Relevance | 0.9608 |
| Argument Flow | 0.5490 |
| Bench Handling | 1.0000 |
**Average MAE:** 1.0589
Scores range from -3 to +3; an MAE of ~1.06 indicates moderate alignment with the gold standard scorer.
The primary objective of evaluation was to verify structured judicial behaviour, schema adherence, and score prediction consistency rather than optimize a single benchmark metric.
Gold standard scores were generated by DeepSeek V4 Flash. Human evaluation is pending
### Training Metrics
* Final Training Loss: 0.8544
* Final Evaluation Loss: 0.8986
The training and evaluation losses remained closely aligned throughout fine-tuning, suggesting stable convergence. Loss gap of 0.044 indicates no significant overfitting.
[<img src="https://www.kaggleusercontent.com/kf/322819816/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.._tjVFbmIDd0felbejmqcRw.0d4p7XqW6uOa__oNGBNIQT3QFz-9eAq5JSF7zuWSadr8Ux-q0bbLQLYJvSMY6Z8T71PfQoPbc5QKjq0Q8wsh6FmFoi8LbM5SandJZqzRFmiKqmXeEDn9RJCgiQTtjw8VAXSOJH_iX1S5b_ng3Zrz-5yn2kKwiYq2JjL4WJAhztWWSkg1OdqNtYhjAlMFw0q_aEHGtOMz-eJaFC1qLg6w9_PG4Qvv495xDcqRvKHiL9IvHYFPrTAxcquZmMBP4VP_UJAkPFSbOyRP1K4zOAPO3bGE_OlrVkTyhUSzbdWuPwfvkVjMJ4U__b6PljLiI03EJ0LdogTk_1r2FKG37BeIMFwwjZcf4R9IN9kujSRhiulFjVXetzaSbiVADIPysxmObXE5WdUzQjpqHC8L5xZoRF_na1fSSY9Wu5HSh61y3nCFGlqNoeGXLUQU2iaYU4ItOV5Hu_tfDw885_sDFn1e94-GtOfqK31vWWX4bHqKJ6FYnzicS-geviAiHDgZg5b2JzZmdMiBiBs7cDFDo1MpmdRvaVcwZ3S8f_jb1vYGBRV9DFJ6UPwecBcvvthNAIZCn5mez9L0LBx7hNK3tRYCqmwyW-xbMHM__75PDhAV2P43vXlrmtgnGCMuqFaf8gu_.SVPLJZ2E_Ukf8doN56DW5Q/__results___files/__results___16_0.png" width="300"/>](https://www.kaggle.com/code/snowml/judge-llama-3-2-qlora-sft)
## Example Outputs
### Example 1
**Case Summary**
Two juveniles were accused of murdering a three-and-a-half-year-old child. Courts differed on whether each juvenile should be tried as an adult.
**Legal Issue**
Whether the Juvenile Justice Board correctly assessed the juveniles' capacity under Section 15 of the Juvenile Justice Act.
**Argument (Petitioner)**
The victim's father has suffered a direct injury and should have standing to challenge the differing treatment of the juveniles. Principles from Maneka Gandhi and Vishaka support a broader interpretation of standing in the interests of justice.
**Model Output**
```json
{
"delta_scores": {
"legal_application": 0,
"issue_relevance": 3,
"argument_flow": 3,
"bench_handling": 3
},
"judge_response": "The bench will now hear the respondent on the locus standi of the victim's father.",
"speaker_switch": true,
"judge_notes": "Counsel handled the question well, but the Maneka Gandhi and Vishaka citation is a stretch — needs to address why those cases are directly applicable to a Juvenile Justice Act provision."
}
```
### Example 2
**Case Summary**
A hotel worker sought implementation of Karnataka's Menstrual Leave Policy across establishments in the unorganized sector.
**Legal Issue**
Whether the State is obligated to implement and enforce the policy to ensure workplace dignity and gender equity.
**Argument (Petitioner)**
The policy flows from Articles 21 and 15(3), and Vishaka demonstrates that workplace dignity obligations may exist even before detailed legislation is enacted.
**Model Output**
```json
{
"delta_scores": {
"legal_application": 3,
"issue_relevance": 3,
"argument_flow": 3,
"bench_handling": null
},
"judge_response": "Counsel, you rely on Vishaka for the right to health and dignity, but the policy itself was enacted by the State — does that not create a conflict between the State's legislative action and its obligation to enforce it?",
"speaker_switch": false,
"judge_notes": "Strong on Article 21 and Vishaka, but didn't address the policy's legislative origin."
}
```
### Example 3
This example is outside the primary training domain (constitutional law) and demonstrates generalization.
**Case Summary**
Sterlite Technologies and BSNL disputed payments relating to optical fiber cables laid in a common trench.
**Legal Issue**
Whether the arbitral tribunal's contractual interpretation was perverse and warranted interference under Section 34.
**Argument (Petitioner)**
The pre-bid clarification applied only to pre-existing government-owned ducts, though the petitioner conceded that subsequent ducts in the trench could practically be viewed as existing ducts.
**Model Output**
```json
{
"delta_scores": {
"legal_application": 0,
"issue_relevance": 3,
"argument_flow": 2,
"bench_handling": null
},
"judge_response": "Counsel, you concede that the pre-bid clarification does not mandate a reduction in service cost, but you argue that the practical engineering reality of the ducts justifies it. How do you reconcile this with the arbitrator's finding that the clarification was a binding interpretation of the contract?",
"speaker_switch": false,
"judge_notes": "Counsel conceded the pre-bid clarification does not mandate reduction but argued practical engineering justifies it - weak legal basis."
}
```
## Reproducibility
Dataset: [Multi-Turn Legal Argumentation](https://huggingface.co/datasets/snowsadh/multiturn-legal-argumentation)
Training Notebook: [Judge : Llama 3.2 QLoRA SFT](https://www.kaggle.com/code/snowml/judge-llama-3-2-qlora-sft)
Evaluation Notebook: [Evaluating Themis Judge](https://www.kaggle.com/code/snowml/evaluating-themis-judge-base-llama-vs-fine-tuned)
## Limitations
* Limited to courtroom-style legal argument evaluation.
* Trained on 504 examples.
* Focused primarily on Indian constitutional law contexts.
* May generate legally incorrect reasoning.
* Not suitable for legal advice or real-world judicial decision making.
## Future Work
Future work will focus on integration of Themis Judge into the Themis moot court platform, including real-time argument evaluation, automated speaker management, and end-to-end courtroom simulation workflows.
Themis Repository: [Themis](https://github.com/Electromagneticradiation/themis)
Setup & Usage Guide: [README.md](https://github.com/Electromagneticradiation/themis/blob/main/README.md)
This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="150"/>](https://github.com/unslothai/unsloth)