161 lines
6.9 KiB
Markdown
161 lines
6.9 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: DoodDood/TOMAGPT
|
|
datasets:
|
|
- DoodDood/HearsayGRPOTrainingData2
|
|
tags:
|
|
- legal
|
|
- hearsay
|
|
- classification
|
|
- grpo
|
|
- reinforcement-learning
|
|
- legalbench
|
|
- lora
|
|
pipeline_tag: text-generation
|
|
model-index:
|
|
- name: TOMAGPT
|
|
results:
|
|
- task:
|
|
type: text-classification
|
|
name: Hearsay Classification
|
|
dataset:
|
|
name: LegalBench Hearsay
|
|
type: nguha/legalbench
|
|
split: test
|
|
metrics:
|
|
- type: accuracy
|
|
value: 77.7
|
|
name: Decomposed Accuracy
|
|
---
|
|
|
|
# GGUF Files for TOMAGPT
|
|
|
|
These are the GGUF files for [DoodDood/TOMAGPT](https://huggingface.co/DoodDood/TOMAGPT).
|
|
|
|
## Downloads
|
|
|
|
| GGUF Link | Quantization | Description |
|
|
| ---- | ----- | ----------- |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.Q2_K.gguf) | Q2_K | Lowest quality |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.Q3_K_S.gguf) | Q3_K_S | |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.IQ3_S.gguf) | IQ3_S | Integer quant, preferable over Q3_K_S |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.IQ3_M.gguf) | IQ3_M | Integer quant |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.Q3_K_M.gguf) | Q3_K_M | |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.Q3_K_L.gguf) | Q3_K_L | |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.IQ4_XS.gguf) | IQ4_XS | Integer quant |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.Q4_K_S.gguf) | Q4_K_S | Fast with good performance |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.Q4_K_M.gguf) | Q4_K_M | **Recommended:** Perfect mix of speed and performance |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.Q5_K_S.gguf) | Q5_K_S | |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.Q5_K_M.gguf) | Q5_K_M | |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.Q6_K.gguf) | Q6_K | Very good quality |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.Q8_0.gguf) | Q8_0 | Best quality |
|
|
| [Download](https://huggingface.co/Flexan/DoodDood-TOMAGPT-GGUF/resolve/main/TOMAGPT.f16.gguf) | f16 | Full precision, don't bother; use a quant |
|
|
|
|
## Note from Flexan
|
|
|
|
I provide GGUFs and quantizations of publicly available models that do not have a GGUF equivalent available yet.
|
|
This process is not yet automated and I download, convert, quantize, and upload them **by hand**, usually for models **I deem interesting and wish to try out**.
|
|
|
|
If there are some quants missing that you'd like me to add, you may request one in the community tab.
|
|
If you want to request a public model to be converted, you can also request that in the community tab.
|
|
If you have questions regarding the model, please refer to the original model repo.
|
|
|
|
# TOMAGPT
|
|
|
|
A **Qwen3-4B-Instruct-2507** model fine-tuned with GRPO (Group Relative Policy Optimization) to classify legal hearsay by decomposing it into three sub-elements under the U.S. Federal Rules of Evidence.
|
|
|
|
## What It Does
|
|
|
|
TOMAGPT classifies whether a statement is hearsay by analyzing three sub-elements:
|
|
|
|
1. **Assertion** -- Is the statement an assertion?
|
|
2. **Out-of-court** -- Was the statement made out of court?
|
|
3. **TOMA** -- Is the statement offered to prove the truth of the matter asserted?
|
|
|
|
Hearsay = YES only if all three sub-elements are YES.
|
|
|
|
## Results
|
|
|
|
Evaluated on the [LegalBench hearsay test set](https://huggingface.co/datasets/nguha/legalbench) (94 examples):
|
|
|
|
| Metric | Base Model | TOMAGPT | Delta |
|
|
|--------|-----------|---------|-------|
|
|
| **Overall accuracy** | 71.3% | **77.7%** | +6.4% |
|
|
| **TOMA sub-element** | 78.0% | **95.1%** | +17.1% |
|
|
| Assertion sub-element | 90.2% | 95.1% | +4.9% |
|
|
| Non-verbal hearsay | 33.3% | 83.3% | +50.0% |
|
|
| Standard hearsay | 93.1% | 100.0% | +6.9% |
|
|
| Non-assertive conduct | 89.5% | 100.0% | +10.5% |
|
|
|
|
## Training Details
|
|
|
|
- **Method**: GRPO (Group Relative Policy Optimization)
|
|
- **Platform**: [Prime Intellect Lab](https://lab.primeintellect.ai)
|
|
- **Environment**: `smolclaims/TOMAGPT` (v0.3.0)
|
|
- **Base model**: Qwen/Qwen3-4B-Instruct-2507
|
|
- **Training data**: [DoodDood/HearsayGRPOTrainingData2](https://huggingface.co/datasets/DoodDood/HearsayGRPOTrainingData2) (3,140 examples)
|
|
- **Steps**: 500
|
|
- **Learning rate**: 1e-5
|
|
- **Batch size**: 128
|
|
- **Rollouts per example**: 16
|
|
|
|
### LoRA Configuration
|
|
|
|
- **Rank (r)**: 16
|
|
- **Alpha**: 32
|
|
- **Dropout**: 0.0
|
|
- **Target modules**: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
|
|
|
### Reward Functions
|
|
|
|
| Function | Weight | Description |
|
|
|----------|--------|-------------|
|
|
| assertion_reward | 1.5 | +1/-1 on assertion accuracy |
|
|
| out_of_court_reward | 1.0 | +1/-1 on out-of-court accuracy |
|
|
| toma_reward | 2.0 | +1/-1 on TOMA accuracy |
|
|
| consistency_penalty | 1.0 | -0.5 for contradictory outputs |
|
|
| format_compliance | 1.0 | -0.25 per missing field |
|
|
| constraint_penalty | 1.0 | -0.5 for logical violations |
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
import torch
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
"DoodDood/TOMAGPT", torch_dtype=torch.bfloat16, device_map="auto")
|
|
tokenizer = AutoTokenizer.from_pretrained("DoodDood/TOMAGPT")
|
|
|
|
system_prompt = (
|
|
"You are a legal assistant identifying hearsay. Hearsay is defined as "
|
|
"an out-of-court statement introduced to prove the truth of the matter "
|
|
"asserted.\n\n"
|
|
"Respond in EXACTLY this format (semicolon-separated):\n"
|
|
"is_hearsay: YES/NO; an_assertion: YES/NO; made_out_of_court: YES/NO; "
|
|
"is_for_toma: YES/NO"
|
|
)
|
|
|
|
scenario = "At trial, the prosecution presents testimony from a police officer who states that a bystander at the scene told him, 'The defendant ran the red light.'"
|
|
|
|
messages = [
|
|
{"role": "system", "content": system_prompt},
|
|
{"role": "user", "content": scenario}
|
|
]
|
|
|
|
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|
inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
|
|
|
with torch.no_grad():
|
|
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
|
|
|
|
response = tokenizer.decode(output[0][len(inputs.input_ids[0]):], skip_special_tokens=True)
|
|
print(response)
|
|
# Expected: is_hearsay: YES; an_assertion: YES; made_out_of_court: YES; is_for_toma: YES
|
|
```
|
|
|
|
## Links
|
|
|
|
- **Training data**: [DoodDood/HearsayGRPOTrainingData2](https://huggingface.co/datasets/DoodDood/HearsayGRPOTrainingData2)
|
|
- **GRPO environment**: `smolclaims/TOMAGPT` on [Prime Intellect](https://lab.primeintellect.ai)
|
|
- **Eval benchmark**: [nguha/legalbench](https://huggingface.co/datasets/nguha/legalbench) (hearsay subset) |