107 lines
4.3 KiB
Markdown
107 lines
4.3 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
base_model: Qwen/Qwen2.5-3B-Instruct
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
- hi
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- legal
|
||
|
|
- india
|
||
|
|
- indian-law
|
||
|
|
- bns
|
||
|
|
- retrieval-augmented-generation
|
||
|
|
- qwen2.5
|
||
|
|
---
|
||
|
|
|
||
|
|
# Nyaya-3B-v3 — Indian Legal Information Model
|
||
|
|
|
||
|
|
> **⚖️ Not legal advice.** Nyaya is a legal *information / guidance* model, **not a
|
||
|
|
> legal advisor**. The practice of law in India is reserved to enrolled advocates
|
||
|
|
> under the Advocates Act, 1961. Always consult a licensed advocate for anything
|
||
|
|
> consequential. Free legal aid is available via NALSA / DLSA (Legal Services
|
||
|
|
> Authorities Act, 1987).
|
||
|
|
|
||
|
|
Nyaya answers everyday legal questions from Indian citizens in **English, Hindi,
|
||
|
|
and Hinglish**, cites specific sections of **current law** (BNS / BNSS / BSA,
|
||
|
|
post-July-2024, with IPC↔BNS bridging), and signals when a real lawyer is needed.
|
||
|
|
|
||
|
|
- **Base model:** [`Qwen/Qwen2.5-3B-Instruct`]( https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) (Apache-2.0)
|
||
|
|
- **Method:** LoRA SFT (bf16, no quantization) using **RAFT** — teacher answers
|
||
|
|
regenerated under the inference-time RAG prompt, including deliberate
|
||
|
|
retrieval-miss demonstrations. Merged to full weights for release.
|
||
|
|
- **Designed to run with retrieval (RAG).** The gains below come *with* a dense
|
||
|
|
retriever supplying statute passages in context; used bare, it behaves close
|
||
|
|
to the base model.
|
||
|
|
|
||
|
|
## Intended use & limitations
|
||
|
|
|
||
|
|
- **Use for:** plain-language legal information, "how do I…" procedural guidance,
|
||
|
|
old→new law (IPC↔BNS) mapping, terminology.
|
||
|
|
- **Do not use for:** actual legal advice, court filings, or any decision without
|
||
|
|
a licensed advocate. Coverage is limited to the acts in the statute DB; it can
|
||
|
|
be wrong or incomplete outside them.
|
||
|
|
|
||
|
|
## Evaluation (Nyaya-Eval-v0, 500 frozen questions)
|
||
|
|
|
||
|
|
Primary metric is **citation accuracy verified against the statute DB**, not loss.
|
||
|
|
Numbers below are strict exact-match on the frozen eval, with dense RAG (k=8):
|
||
|
|
|
||
|
|
| Setting | Strict acc. | Lenient acc. |
|
||
|
|
|---|---|---|
|
||
|
|
| Base + dense RAG | 3.4% | — |
|
||
|
|
| **Nyaya-3B-v3 + dense RAG** | **3.8%** | **5.5%** |
|
||
|
|
|
||
|
|
On the held-out *dataset* eval, citation pass-rate is **90.2%** across
|
||
|
|
1068 examples.
|
||
|
|
|
||
|
|
**Honest status:** strict exact-match on the frozen benchmark is still low and the
|
||
|
|
project's human-eval ship gate has **not** been passed. This is an early research
|
||
|
|
release, **not** a validated "best" legal model. See the repo's `docs/ROADMAP.md`
|
||
|
|
go/no-go gates.
|
||
|
|
|
||
|
|
## How to use
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
tok = AutoTokenizer.from_pretrained("NyayaLabs98/nyaya-3b-v3")
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("NyayaLabs98/nyaya-3b-v3", device_map="auto")
|
||
|
|
|
||
|
|
system = (
|
||
|
|
"You are Nyaya, an Indian legal information model. You provide accurate, "
|
||
|
|
"plain-language legal guidance for Indian citizens, cite specific sections of "
|
||
|
|
"current law (BNS/BNSS/BSA and other acts in force), clearly state uncertainty, "
|
||
|
|
"and recommend consulting a licensed advocate for anything consequential. "
|
||
|
|
"You provide legal information, not legal advice."
|
||
|
|
)
|
||
|
|
# For best results, prepend retrieved statute passages to the user turn (RAG).
|
||
|
|
messages = [
|
||
|
|
{"role": "system", "content": system},
|
||
|
|
{"role": "user", "content": "Police FIR nahi likh rahi, kya karu?"},
|
||
|
|
]
|
||
|
|
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
|
||
|
|
out = model.generate(inputs, max_new_tokens=512)
|
||
|
|
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
|
||
|
|
```
|
||
|
|
|
||
|
|
## Training curves & ablations
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## Training details
|
||
|
|
|
||
|
|
See `training/` in this repo for the exact config, metric history, and all
|
||
|
|
figures (PNG + source CSV): `train_v3.yaml`, `training_history_v3.json`,
|
||
|
|
`rag_eval_rescored.json`, `checkpoint_evals.json`, `figures/`.
|
||
|
|
|
||
|
|
## License & attribution
|
||
|
|
|
||
|
|
Model weights released under **Apache-2.0**, inheriting the base model's license.
|
||
|
|
Statutory text is Government of India material (India Code / legislative.gov.in).
|
||
|
|
See the repository `NOTICE` for data provenance.
|