初始化项目，由ModelHub XC社区提供模型
Model: MrRoyaleAce/nyaya-7b Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
319
README.md
Normal file
319
README.md
Normal file
@@ -0,0 +1,319 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
- hi
|
||||
license: apache-2.0
|
||||
base_model: mistralai/Mistral-7B-Instruct-v0.3
|
||||
tags:
|
||||
- legal
|
||||
- indian-legal
|
||||
- information-extraction
|
||||
- nlp
|
||||
- qlora
|
||||
- peft
|
||||
- finetuned
|
||||
- mistral
|
||||
- json-extraction
|
||||
- legal-nlp
|
||||
datasets:
|
||||
- d0r1h/ILSum
|
||||
- law-ai/InLegalNLP
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
model-index:
|
||||
- name: nyaya-7b
|
||||
results:
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Structured Legal Information Extraction
|
||||
metrics:
|
||||
- type: f1
|
||||
value: 0.425
|
||||
name: Statute F1
|
||||
- type: accuracy
|
||||
value: 0.64
|
||||
name: Outcome Accuracy
|
||||
- type: other
|
||||
value: 0.427
|
||||
name: Hallucination Rate (lower is better)
|
||||
- type: other
|
||||
value: 0.86
|
||||
name: JSON Validity Rate
|
||||
---
|
||||
|
||||
# 🏛️ Nyaya-7B — Indian Legal Judgment Parser
|
||||
|
||||
<p align="center">
|
||||
<img src="https://img.shields.io/badge/Base%20Model-Mistral--7B--Instruct--v0.3-blue" />
|
||||
<img src="https://img.shields.io/badge/Method-QLoRA%20(4--bit%20NF4)-orange" />
|
||||
<img src="https://img.shields.io/badge/Domain-Indian%20Legal%20NLP-green" />
|
||||
<img src="https://img.shields.io/badge/License-Apache%202.0-red" />
|
||||
<img src="https://img.shields.io/badge/Language-English%20%7C%20Hindi-purple" />
|
||||
</p>
|
||||
|
||||
**Nyaya-7B** is a domain-adapted, instruction-finetuned version of [Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3), trained on **10,000+ Indian Supreme Court and High Court judgments** to extract structured legal information into clean, validated JSON — at **zero API cost**, fully offline.
|
||||
|
||||
> **"Nyaya" (न्याय)** means *justice* in Sanskrit and Hindi.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 What it does
|
||||
|
||||
Given raw Indian court judgment text, Nyaya-7B extracts a full structured JSON covering:
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| `case_name` | Petitioner v. Respondent |
|
||||
| `citation` | AIR / SCC / SCR citation |
|
||||
| `court` | Full court name (Supreme Court, High Court, etc.) |
|
||||
| `year` | Year of judgment |
|
||||
| `petitioner` / `respondent` | Party names |
|
||||
| `subject_matter` | Criminal / Civil / Constitutional / Tax / ... |
|
||||
| `statutes_cited` | List of Acts + Sections + descriptions |
|
||||
| `precedents_cited` | AIR/SCC citations with case names |
|
||||
| `legal_issues` | Issues framed by the court |
|
||||
| `holding` | Court's decision and reasoning (1–3 sentences) |
|
||||
| `outcome` | dismissed / allowed / disposed / remanded / modified |
|
||||
|
||||
---
|
||||
|
||||
## 📊 Benchmark Results
|
||||
|
||||
Evaluated on a **50-case held-out test set** of Indian SC/HC judgments, benchmarked head-to-head against Gemini 2.5 Flash:
|
||||
|
||||
| Metric | Gemini 2.5 Flash | **Nyaya-7B** | Winner |
|
||||
|---|---|---|---|
|
||||
| **Statute F1** | 0.227 | **0.425** | 🏆 Nyaya-7B (+87%) |
|
||||
| **Outcome Accuracy** | 0.20 | **0.64** | 🏆 Nyaya-7B (+220%) |
|
||||
| **Hallucination Rate** ↓ | 0.775 | **0.427** | 🏆 Nyaya-7B (−45%) |
|
||||
| **JSON Validity** | 0.98 | 0.86 | Gemini Flash |
|
||||
| **Field Coverage** | 0.893 | 0.855 | Gemini Flash |
|
||||
| **Cost per Judgment** | ~$0.0001 | **$0.00** | 🏆 Nyaya-7B |
|
||||
|
||||
> Nyaya-7B achieves **3.2× higher outcome classification accuracy** and **45% lower hallucination rate** than Gemini 2.5 Flash, while running entirely offline at zero cost.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
pip install transformers torch accelerate bitsandbytes
|
||||
```
|
||||
|
||||
### Load & Run (GPU, 4-bit quantized)
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, BitsAndBytesConfig
|
||||
import torch, json
|
||||
|
||||
MODEL_ID = "mrroyaleace/nyaya-7b" # replace with your HuggingFace repo
|
||||
|
||||
bnb_config = BitsAndBytesConfig(
|
||||
load_in_4bit=True,
|
||||
bnb_4bit_quant_type="nf4",
|
||||
bnb_4bit_compute_dtype=torch.float16,
|
||||
)
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
|
||||
tokenizer.pad_token = tokenizer.eos_token
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
MODEL_ID,
|
||||
quantization_config=bnb_config,
|
||||
device_map="auto",
|
||||
)
|
||||
|
||||
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
|
||||
```
|
||||
|
||||
### Extract structured data from a judgment
|
||||
|
||||
```python
|
||||
judgment_text = """
|
||||
IN THE SUPREME COURT OF INDIA
|
||||
Criminal Appeal No. 1234 of 2022
|
||||
|
||||
State of Punjab ...Appellant
|
||||
Versus
|
||||
Gurpreet Singh ...Respondent
|
||||
|
||||
JUDGMENT
|
||||
|
||||
The appellant challenges the High Court's order acquitting the respondent
|
||||
of charges under Section 302 IPC read with Section 34 IPC...
|
||||
"""
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "You are Nyaya, a specialized Indian legal extraction model."},
|
||||
{"role": "user", "content": f"Extract structured data from this judgment and return JSON:\n\n{judgment_text}"}
|
||||
]
|
||||
|
||||
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
|
||||
raw_output = pipe(
|
||||
prompt,
|
||||
max_new_tokens=512,
|
||||
do_sample=False,
|
||||
return_full_text=False,
|
||||
pad_token_id=tokenizer.eos_token_id,
|
||||
)[0]["generated_text"]
|
||||
|
||||
result = json.loads(raw_output.strip())
|
||||
print(result)
|
||||
```
|
||||
|
||||
### Expected Output
|
||||
|
||||
```json
|
||||
{
|
||||
"case_name": "State of Punjab v. Gurpreet Singh",
|
||||
"citation": null,
|
||||
"court": "Supreme Court of India",
|
||||
"year": 2022,
|
||||
"petitioner": "State of Punjab",
|
||||
"respondent": "Gurpreet Singh",
|
||||
"subject_matter": "Criminal",
|
||||
"statutes_cited": [
|
||||
{"act": "Indian Penal Code", "section": "302", "description": "Punishment for murder"},
|
||||
{"act": "Indian Penal Code", "section": "34", "description": "Acts done by several persons in furtherance of common intention"}
|
||||
],
|
||||
"precedents_cited": [],
|
||||
"legal_issues": [
|
||||
"Whether the High Court was justified in acquitting the respondent under Section 302 IPC?"
|
||||
],
|
||||
"holding": "The Supreme Court examined the evidence and found the High Court's reasoning sound...",
|
||||
"outcome": "dismissed"
|
||||
}
|
||||
```
|
||||
|
||||
### CPU Inference (no GPU required)
|
||||
|
||||
```python
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
MODEL_ID,
|
||||
torch_dtype=torch.float32,
|
||||
device_map="cpu",
|
||||
low_cpu_mem_usage=True,
|
||||
)
|
||||
# Note: CPU inference is significantly slower (~5–15 min per judgment)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Training Details
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| **Base model** | mistralai/Mistral-7B-Instruct-v0.3 |
|
||||
| **Fine-tuning method** | QLoRA (Quantized Low-Rank Adaptation) |
|
||||
| **Quantization** | 4-bit NF4 with double quantization |
|
||||
| **LoRA rank** | 16 |
|
||||
| **LoRA alpha** | 32 |
|
||||
| **LoRA dropout** | 0.05 |
|
||||
| **Target modules** | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
|
||||
| **Training samples** | ~10,000 Indian SC/HC judgment pairs |
|
||||
| **Epochs** | 3 |
|
||||
| **Effective batch size** | 16 (batch 2 × grad_accum 8) |
|
||||
| **Learning rate** | 2e-4 |
|
||||
| **LR scheduler** | Cosine |
|
||||
| **Optimizer** | paged_adamw_8bit |
|
||||
| **Max sequence length** | 2048 tokens |
|
||||
| **Hardware** | Kaggle T4 × 2 (32 GB VRAM total) |
|
||||
| **Training time** | ~8 hours |
|
||||
| **Compute dtype** | float16 |
|
||||
|
||||
### Training Infrastructure
|
||||
|
||||
- Fine-tuned on **Kaggle T4 × 2** GPU notebooks
|
||||
- Monitored with **Weights & Biases** (JSON validity rate logged per epoch)
|
||||
- Adapter merged into base model weights with `merge_and_unload()` for zero inference overhead
|
||||
|
||||
---
|
||||
|
||||
## 📚 Training Data
|
||||
|
||||
The model was trained on ~2,000 Indian court judgment pairs curated and labeled from:
|
||||
|
||||
- **[ILSum](https://huggingface.co/datasets/d0r1h/ILSum)** — Indian Legal Summarization dataset (Supreme Court judgments)
|
||||
- **[InLegalNLP](https://huggingface.co/datasets/law-ai/InLegalNLP)** — Indian Legal NLP benchmark corpus
|
||||
|
||||
Labels were auto-generated using **Gemini 2.5 Flash** as a labelling oracle on the raw judgment texts, following the canonical extraction schema, then validated for JSON structure and field completeness.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Limitations
|
||||
|
||||
- **Not for legal advice:** This model extracts structured information only. It does not provide legal opinions or advice. Always consult a qualified lawyer for legal matters.
|
||||
- **Pre-1950 judgments:** May perform poorly on archaic legal language from older judgments.
|
||||
- **Hindi/regional language text:** Primarily trained on English-language judgments; performance degrades on mixed-language or vernacular text.
|
||||
- **Scanned/handwritten PDFs:** Model accepts only clean text input — OCR preprocessing is required for scanned documents.
|
||||
- **Citation hallucination:** Significantly reduced (42.7% vs 77.5% baseline), but the model can still occasionally generate plausible-but-incorrect section numbers. Always validate critical citations against primary sources.
|
||||
- **Novel statutes:** Statutes not well-represented in training data (e.g., recent 2023–24 acts) may have lower extraction accuracy.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Intended Use
|
||||
|
||||
- ⚖️ Legal research and document processing automation
|
||||
- 🤖 Paralegal workflow tools and legal analytics dashboards
|
||||
- 📖 Academic research on Indian legal NLP
|
||||
- 🔍 Building legal search and knowledge graph systems
|
||||
- 📊 Bulk digitization of case records
|
||||
|
||||
## ❌ Out-of-Scope Use
|
||||
|
||||
- Providing legal advice to individuals
|
||||
- Making or influencing judicial decisions
|
||||
- Use in actual legal proceedings without qualified human review
|
||||
- Any high-stakes decision-making without validation
|
||||
|
||||
---
|
||||
|
||||
## 📄 Output Schema
|
||||
|
||||
```python
|
||||
{
|
||||
"case_name": str, # "Petitioner v. Respondent"
|
||||
"citation": str | None, # "AIR 1997 SC 3986" or null
|
||||
"court": str, # Full court name
|
||||
"year": int | None, # 4-digit year
|
||||
"petitioner": str,
|
||||
"respondent": str,
|
||||
"subject_matter": str | None, # Criminal | Civil | Constitutional | ...
|
||||
"statutes_cited": [{"act": str, "section": str, "description": str}],
|
||||
"precedents_cited": [{"citation": str, "case_name": str | None}],
|
||||
"legal_issues": [str],
|
||||
"holding": str, # 1-3 sentence summary
|
||||
"outcome": str # dismissed | allowed | disposed | remanded | modified
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Related Resources
|
||||
|
||||
- 🐙 **GitHub:** [nyaya-legal-ai](https://github.com/shubace/nyaya-legal-ai) — Full project source code including FastAPI backend, LangGraph pipeline, and Streamlit UI
|
||||
- 📊 **Base Model:** [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)
|
||||
- 📁 **Dataset (ILSum):** [d0r1h/ILSum](https://huggingface.co/datasets/d0r1h/ILSum)
|
||||
|
||||
---
|
||||
|
||||
## 📝 Citation
|
||||
|
||||
If you use Nyaya-7B in your research or applications, please cite:
|
||||
|
||||
```bibtex
|
||||
@misc{nyaya7b2026,
|
||||
title = {Nyaya-7B: A QLoRA Fine-tuned LLM for Indian Legal Judgment Parsing},
|
||||
author = {Shubham Suman},
|
||||
year = {2026},
|
||||
url = {https://huggingface.co/mrroyaleace/nyaya-7b},
|
||||
note = {Fine-tuned from mistralai/Mistral-7B-Instruct-v0.3 on 2,000+ Indian SC/HC judgments}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Built with ❤️ for the Indian legal research community. Nyaya-7B is open-source and free to use under the Apache 2.0 license.*
|
||||
87
chat_template.jinja
Normal file
87
chat_template.jinja
Normal file
@@ -0,0 +1,87 @@
|
||||
{%- if messages[0]["role"] == "system" %}
|
||||
{%- set system_message = messages[0]["content"] %}
|
||||
{%- set loop_messages = messages[1:] %}
|
||||
{%- else %}
|
||||
{%- set loop_messages = messages %}
|
||||
{%- endif %}
|
||||
{%- if not tools is defined %}
|
||||
{%- set tools = none %}
|
||||
{%- endif %}
|
||||
{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
|
||||
|
||||
{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
|
||||
{%- set ns = namespace() %}
|
||||
{%- set ns.index = 0 %}
|
||||
{%- for message in loop_messages %}
|
||||
{%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
|
||||
{%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
|
||||
{{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
|
||||
{%- endif %}
|
||||
{%- set ns.index = ns.index + 1 %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
|
||||
{{- bos_token }}
|
||||
{%- for message in loop_messages %}
|
||||
{%- if message["role"] == "user" %}
|
||||
{%- if tools is not none and (message == user_messages[-1]) %}
|
||||
{{- "[AVAILABLE_TOOLS] [" }}
|
||||
{%- for tool in tools %}
|
||||
{%- set tool = tool.function %}
|
||||
{{- '{"type": "function", "function": {' }}
|
||||
{%- for key, val in tool.items() if key != "return" %}
|
||||
{%- if val is string %}
|
||||
{{- '"' + key + '": "' + val + '"' }}
|
||||
{%- else %}
|
||||
{{- '"' + key + '": ' + val|tojson }}
|
||||
{%- endif %}
|
||||
{%- if not loop.last %}
|
||||
{{- ", " }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{{- "}}" }}
|
||||
{%- if not loop.last %}
|
||||
{{- ", " }}
|
||||
{%- else %}
|
||||
{{- "]" }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{{- "[/AVAILABLE_TOOLS]" }}
|
||||
{%- endif %}
|
||||
{%- if loop.last and system_message is defined %}
|
||||
{{- "[INST] " + system_message + "\n\n" + message["content"] + "[/INST]" }}
|
||||
{%- else %}
|
||||
{{- "[INST] " + message["content"] + "[/INST]" }}
|
||||
{%- endif %}
|
||||
{%- elif message.tool_calls is defined and message.tool_calls is not none %}
|
||||
{{- "[TOOL_CALLS] [" }}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- set out = tool_call.function|tojson %}
|
||||
{{- out[:-1] }}
|
||||
{%- if not tool_call.id is defined or tool_call.id|length != 9 %}
|
||||
{{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
|
||||
{%- endif %}
|
||||
{{- ', "id": "' + tool_call.id + '"}' }}
|
||||
{%- if not loop.last %}
|
||||
{{- ", " }}
|
||||
{%- else %}
|
||||
{{- "]" + eos_token }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif message["role"] == "assistant" %}
|
||||
{{- " " + message["content"]|trim + eos_token}}
|
||||
{%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
|
||||
{%- if message.content is defined and message.content.content is defined %}
|
||||
{%- set content = message.content.content %}
|
||||
{%- else %}
|
||||
{%- set content = message.content %}
|
||||
{%- endif %}
|
||||
{{- '[TOOL_RESULTS] {"content": ' + content|string + ", " }}
|
||||
{%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
|
||||
{{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
|
||||
{%- endif %}
|
||||
{{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
|
||||
{%- else %}
|
||||
{{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
30
config.json
Normal file
30
config.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"architectures": [
|
||||
"MistralForCausalLM"
|
||||
],
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"dtype": "float16",
|
||||
"eos_token_id": 2,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 14336,
|
||||
"max_position_embeddings": 32768,
|
||||
"model_type": "mistral",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 8,
|
||||
"pad_token_id": null,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_parameters": {
|
||||
"rope_theta": 1000000.0,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"sliding_window": null,
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "5.0.0",
|
||||
"use_cache": true,
|
||||
"vocab_size": 32768
|
||||
}
|
||||
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"transformers_version": "5.0.0"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d57fab5123e04c27b28b3a0187e967829b861824577c2623e465214f1efb80c1
|
||||
size 14496080848
|
||||
275733
tokenizer.json
Normal file
275733
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
16
tokenizer_config.json
Normal file
16
tokenizer_config.json
Normal file
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"add_prefix_space": true,
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<s>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "</s>",
|
||||
"is_local": true,
|
||||
"legacy": false,
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"pad_token": "</s>",
|
||||
"sp_model_kwargs": {},
|
||||
"spaces_between_special_tokens": false,
|
||||
"tokenizer_class": "TokenizersBackend",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
Reference in New Issue
Block a user