初始化项目，由ModelHub XC社区提供模型
Model: mii-llm/open-zagreus-0.4B Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
330
README.md
Normal file
330
README.md
Normal file
@@ -0,0 +1,330 @@
|
|||||||
|
---
|
||||||
|
language:
|
||||||
|
- it
|
||||||
|
- en
|
||||||
|
license: apache-2.0
|
||||||
|
tags:
|
||||||
|
- small-language-model
|
||||||
|
- slm
|
||||||
|
- edge-ai
|
||||||
|
- italian
|
||||||
|
- bilingual
|
||||||
|
- instruction-following
|
||||||
|
- open-source
|
||||||
|
- fully-reproducible
|
||||||
|
- llama
|
||||||
|
- nanotron
|
||||||
|
- axolotl
|
||||||
|
base_model: mii-llm/zagreus-0.4B-ita
|
||||||
|
model_type: llama
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
library_name: transformers
|
||||||
|
datasets:
|
||||||
|
- DeepMount00/OpenItalianData
|
||||||
|
---
|
||||||
|
|
||||||
|
# Open-Zagreus-0.4B
|
||||||
|
|
||||||
|
**Open-Zagreus-0.4B** is a fully open-source bilingual English/Italian Small Language Model (SLM) — open data, open weights, open recipe. It is post-trained on top of [Zagreus-0.4B-ita](https://huggingface.co/mii-llm/zagreus-0.4B-ita) using the publicly available [OpenItalianData](https://huggingface.co/datasets/DeepMount00/OpenItalianData) dataset published by Michele Montebovi, making the entire pipeline — from pre-training data to final weights — **fully reproducible by anyone**.
|
||||||
|
|
||||||
|
This model is released by the [mii-llm](https://mii-llm.ai) community (*Made in Italy – Large Language Model*) as a contribution to the open-source Italian NLP ecosystem, demonstrating that it is possible to build competitive English/Italian language models using exclusively open resources.
|
||||||
|
|
||||||
|
> ✅ **Fully open**: all training data, model weights, and training recipes are publicly available and reproducible.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Model Details
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|---|---|
|
||||||
|
| **Architecture** | Modified Llama-3.2 (fully dense) |
|
||||||
|
| **Parameters** | ~400M |
|
||||||
|
| **Hidden size** | 960 |
|
||||||
|
| **Layers** | 32 |
|
||||||
|
| **Attention heads** | 15 (KV heads: 5) |
|
||||||
|
| **Context length** | 4096 tokens |
|
||||||
|
| **Tokenizer** | Llama-3.2 (`vocab_size`: 128,256) |
|
||||||
|
| **Precision** | BF16 |
|
||||||
|
| **Languages** | English, Italian |
|
||||||
|
| **Base model** | mii-llm/zagreus-0.4B-ita |
|
||||||
|
| **SFT dataset** | [DeepMount00/OpenItalianData](https://huggingface.co/datasets/DeepMount00/OpenItalianData) |
|
||||||
|
| **Post-training framework** | Axolotl + FSDP |
|
||||||
|
| **Chat template** | ChatML |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Training Details
|
||||||
|
|
||||||
|
### Base Model Pre-training
|
||||||
|
|
||||||
|
`Open-Zagreus-0.4B` is built on `Zagreus-0.4B-ita`, pre-trained on approximately **1 trillion tokens**:
|
||||||
|
|
||||||
|
| Dataset | Description |
|
||||||
|
|---|---|
|
||||||
|
| [FineWeb (350BT sample)](https://huggingface.co/datasets/HuggingFaceFW/fineweb/viewer/sample-350BT) | ~350B tokens of English web text |
|
||||||
|
| [FineWeb-2 (ita_Latn)](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2/viewer/ita_Latn) | Italian web text |
|
||||||
|
| [FinePDFs (ita_Latn)](https://huggingface.co/datasets/HuggingFaceFW/finepdfs/viewer/ita_Latn) | Italian PDF documents |
|
||||||
|
| [StarCoder Data](https://huggingface.co/datasets/bigcode/starcoderdata) | ~250B tokens of code |
|
||||||
|
|
||||||
|
**Token distribution**: ~400B English + ~400B Italian + ~200B Code
|
||||||
|
**Infrastructure**: 64× NVIDIA A100 GPUs (8 nodes × 8 GPUs) on Seeweb HPC
|
||||||
|
**Pre-training framework**: [Nanotron (mii-llm fork)](https://github.com/mii-llm/nanotron)
|
||||||
|
|
||||||
|
### Post-training (SFT)
|
||||||
|
|
||||||
|
Post-training was performed using **Axolotl** with FSDP across 4 nodes (32× A100 GPUs), using the fully public **OpenItalianData** dataset.
|
||||||
|
|
||||||
|
**SFT dataset**: [DeepMount00/OpenItalianData](https://huggingface.co/datasets/DeepMount00/OpenItalianData)
|
||||||
|
**Dataset author**: Michele Montebovi
|
||||||
|
|
||||||
|
**Key hyperparameters:**
|
||||||
|
|
||||||
|
| Hyperparameter | Value |
|
||||||
|
|---|---|
|
||||||
|
| Optimizer | AdamW (fused) |
|
||||||
|
| Learning rate | `1e-3` |
|
||||||
|
| LR scheduler | Cosine (constant ratio: 0.8, min ratio: 0.3) |
|
||||||
|
| Epochs | 3 |
|
||||||
|
| Micro batch size | 1 |
|
||||||
|
| Gradient accumulation steps | 8 |
|
||||||
|
| Sequence length | 4096 |
|
||||||
|
| Max grad norm | 1.0 |
|
||||||
|
| Precision | BF16 + Flash Attention |
|
||||||
|
| FSDP strategy | FULL_SHARD |
|
||||||
|
|
||||||
|
### Full Axolotl Configuration
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
base_model: giux78/zagreus-0.4B-ita
|
||||||
|
strict: false
|
||||||
|
output_dir: ./ale_outputs/opendata-zagreus-sft-final
|
||||||
|
seed: 42
|
||||||
|
chat_template_jinja: "{%- for message in messages -%}\n {{- \"<|im_start|>\" + message.role + \"\\n\" + message.content + \"<|im_end|>\" + \"\\n\" -}}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n\t{{- \"<|im_start|>assistant\\n\" -}}\n{%- endif -%}"
|
||||||
|
datasets:
|
||||||
|
- path: /training/openitaliandata
|
||||||
|
type: chat_template
|
||||||
|
field_messages: conversation
|
||||||
|
roles_to_train: ["assistant"]
|
||||||
|
train_on_eos: turn
|
||||||
|
|
||||||
|
dataset_prepared_path: ./ale_outputs/dataset_cache/opendata-zagreus-sft
|
||||||
|
|
||||||
|
sequence_len: 4096
|
||||||
|
sample_packing: true
|
||||||
|
eval_sample_packing: true
|
||||||
|
pad_to_sequence_len: true
|
||||||
|
|
||||||
|
cosine_constant_lr_ratio: 0.8
|
||||||
|
cosine_min_lr_ratio: 0.3
|
||||||
|
|
||||||
|
optimizer: adamw_torch_fused
|
||||||
|
lr_scheduler: constant
|
||||||
|
learning_rate: 1.0e-03
|
||||||
|
|
||||||
|
max_grad_norm: 1.0
|
||||||
|
micro_batch_size: 1
|
||||||
|
gradient_accumulation_steps: 8
|
||||||
|
num_epochs: 3
|
||||||
|
|
||||||
|
bf16: auto
|
||||||
|
flash_attention: true
|
||||||
|
gradient_checkpointing: true
|
||||||
|
|
||||||
|
logging_steps: 10
|
||||||
|
eval_strategy: steps
|
||||||
|
eval_steps: 300
|
||||||
|
save_strategy: steps
|
||||||
|
save_steps: 500
|
||||||
|
save_total_limit: 3
|
||||||
|
val_set_size: 10000
|
||||||
|
|
||||||
|
fsdp_config:
|
||||||
|
fsdp_sharding_strategy: FULL_SHARD
|
||||||
|
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
|
||||||
|
fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
|
||||||
|
fsdp_backward_prefetch_policy: BACKWARD_PRE
|
||||||
|
fsdp_state_dict_type: FULL_STATE_DICT
|
||||||
|
|
||||||
|
special_tokens:
|
||||||
|
pad_token: <|im_end|>
|
||||||
|
eos_token: <|im_end|>
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Chat Template
|
||||||
|
|
||||||
|
This model uses the **ChatML** format:
|
||||||
|
|
||||||
|
```
|
||||||
|
<|im_start|>system
|
||||||
|
Sei un assistente utile.<|im_end|>
|
||||||
|
<|im_start|>user
|
||||||
|
Ciao! Come posso imparare l'italiano?<|im_end|>
|
||||||
|
<|im_start|>assistant
|
||||||
|
```
|
||||||
|
|
||||||
|
Special tokens:
|
||||||
|
- `pad_token`: `<|im_end|>`
|
||||||
|
- `eos_token`: `<|im_end|>`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||||
|
import torch
|
||||||
|
|
||||||
|
model_id = "mii-llm/open-zagreus-0.4B"
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
model_id,
|
||||||
|
torch_dtype=torch.bfloat16,
|
||||||
|
device_map="auto"
|
||||||
|
)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": "Sei un assistente utile e preciso."},
|
||||||
|
{"role": "user", "content": "Raccontami qualcosa di interessante sull'Italia."}
|
||||||
|
]
|
||||||
|
|
||||||
|
input_ids = tokenizer.apply_chat_template(
|
||||||
|
messages,
|
||||||
|
add_generation_prompt=True,
|
||||||
|
return_tensors="pt"
|
||||||
|
).to(model.device)
|
||||||
|
|
||||||
|
output = model.generate(
|
||||||
|
input_ids,
|
||||||
|
max_new_tokens=512,
|
||||||
|
temperature=0.7,
|
||||||
|
do_sample=True
|
||||||
|
)
|
||||||
|
|
||||||
|
print(tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True))
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Evaluation
|
||||||
|
|
||||||
|
### Standard Benchmarks
|
||||||
|
|
||||||
|
#### Evaluation Command
|
||||||
|
|
||||||
|
```bash
|
||||||
|
lm-eval --model hf --model_args pretrained=giux78/Open-Zagreus-0.4B \
|
||||||
|
--tasks m_mmlu_it,arc_it,hellaswag_it --device cuda:0 --batch_size 1
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Results
|
||||||
|
|
||||||
|
| Model | MMLU IT ↑ | ARC IT ↑ | HellaSwag IT ↑ | **Average** |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| **Open-Zagreus-0.4B** | 0.2530 | 0.3020 | 0.3608 | **0.3053** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Evalita Benchmark
|
||||||
|
|
||||||
|
Evalita is a comprehensive Italian NLP evaluation suite covering a wide range of linguistic tasks. We evaluate Open-Zagreus-0.4B using the [evalita-mp](https://github.com/evalita) tasks and compare it directly against its base model (`Zagreus-0.4B-ita`) to measure the impact of SFT.
|
||||||
|
|
||||||
|
#### Evaluation Command
|
||||||
|
|
||||||
|
```bash
|
||||||
|
lm_eval --model hf \
|
||||||
|
--model_args pretrained=giux78/Open-Zagreus-0.4B \
|
||||||
|
--tasks evalita-mp \
|
||||||
|
--device cuda:0 \
|
||||||
|
--batch_size 1
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Results: Open-Zagreus-0.4B vs. Zagreus-0.4B-ita (Base)
|
||||||
|
|
||||||
|
| Task | Metric | Zagreus-0.4B-ita (base) | **Open-Zagreus-0.4B (SFT)** | Δ |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| **Overall** | acc | 0.3226 | **0.3313** | **+0.0087** |
|
||||||
|
| Admission Test | acc | **0.2137** | 0.2083 | -0.0054 |
|
||||||
|
| FAQ | acc | **0.2681** | 0.2672 | -0.0009 |
|
||||||
|
| Hate Speech Detection | f1 | **0.6056** | 0.4340 | -0.1716 |
|
||||||
|
| Lexical Substitution | f1 | 0.0000 | 0.0000 | = |
|
||||||
|
| NER | f1 | **0.1611** | 0.1357 | -0.0254 |
|
||||||
|
| Relation Extraction | f1 | **0.1244** | 0.0000 | -0.1244 |
|
||||||
|
| Sentiment Analysis | f1 | 0.3660 | **0.3712** | +0.0052 |
|
||||||
|
| Summarization (Fanpage) | rouge1 | 0.1947 | **0.2305** | +0.0358 |
|
||||||
|
| Text Entailment | acc | 0.5133 | **0.5492** | +0.0359 |
|
||||||
|
| Word in Context | f1 | 0.4697 | **0.4880** | +0.0183 |
|
||||||
|
|
||||||
|
#### Discussion
|
||||||
|
|
||||||
|
The SFT stage delivers a net **+0.0087 overall improvement** on Evalita. Gains are most significant in generative and semantic tasks:
|
||||||
|
|
||||||
|
- **Summarization** (+0.0358): the model produces more coherent and relevant summaries after instruction tuning
|
||||||
|
- **Text Entailment** (+0.0359): improved language understanding and reasoning
|
||||||
|
- **Word in Context** (+0.0183): better contextual semantic disambiguation
|
||||||
|
- **Sentiment Analysis** (+0.0052): marginal improvement in affective understanding
|
||||||
|
|
||||||
|
Some structured classification tasks (Hate Speech Detection, Relation Extraction, NER) regress after SFT — a known phenomenon when general-purpose instruction tuning shifts the model away from the specific output format expected by these extractive tasks. This is expected behavior and not indicative of degraded general language quality.
|
||||||
|
|
||||||
|
Overall, these results confirm that **a fully open-source pipeline — using only publicly available data and tools — is sufficient to produce a competitive Italian SLM**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Reproducibility
|
||||||
|
|
||||||
|
This is the only model in the Nesso/Zagreus family where **every component is fully open and reproducible**:
|
||||||
|
|
||||||
|
| Component | Resource |
|
||||||
|
|---|---|
|
||||||
|
| Pre-training data | FineWeb, FineWeb-2, FinePDFs, StarCoder (all public) |
|
||||||
|
| Pre-training framework | [mii-llm/nanotron](https://github.com/mii-llm/nanotron) |
|
||||||
|
| SFT data | [DeepMount00/OpenItalianData](https://huggingface.co/datasets/DeepMount00/OpenItalianData) |
|
||||||
|
| SFT framework | [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) |
|
||||||
|
| Evaluation | [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) |
|
||||||
|
| Model weights | This repository |
|
||||||
|
| Training config | See Axolotl configuration above |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related Models
|
||||||
|
|
||||||
|
| Model | Description |
|
||||||
|
|---|---|
|
||||||
|
| [Zagreus-0.4B-ita](https://huggingface.co/mii-llm/zagreus-0.4B-ita) | Base pre-trained model (this model's foundation) |
|
||||||
|
| [Nesso-0.4B-instruct](https://huggingface.co/mii-llm/nesso-0.4B-instruct) | Proprietary SFT — optimized for instruction following |
|
||||||
|
| [Nesso-0.4B-agentic](https://huggingface.co/mii-llm/nesso-0.4B-agentic) | Proprietary SFT — optimized for function calling and agentic tasks |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Citation
|
||||||
|
|
||||||
|
If you use this model in your research, please cite:
|
||||||
|
|
||||||
|
```bibtex
|
||||||
|
@misc{nesso2025,
|
||||||
|
title = {The Joy and Pain of Training an LLM from Scratch:
|
||||||
|
A Technical Report on the Zagreus and Nesso Model Families},
|
||||||
|
author = {mii-llm community},
|
||||||
|
year = {2025},
|
||||||
|
howpublished = {\url{https://github.com/mii-llm/zagreus-nesso-slm}},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Acknowledgements
|
||||||
|
|
||||||
|
- **Antonio Baldassarra** (CEO, Seeweb) and **Marco Cristofanilli** (Head of AI, Seeweb) for infrastructure sponsorship
|
||||||
|
- **Michele Montebovi** for publishing the [OpenItalianData](https://huggingface.co/datasets/DeepMount00/OpenItalianData) SFT dataset that makes this model fully reproducible
|
||||||
|
- The **Hugging Face** team for Nanotron, datatrove, FineWeb, and FineWeb-2
|
||||||
|
- The **mii-llm** open-source community
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Released under the **Apache 2.0** license.
|
||||||
|
|
||||||
|
> Made with ❤️ in Italy by [mii-llm](https://mii-llm.ai)
|
||||||
6
chat_template.jinja
Normal file
6
chat_template.jinja
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
{%- for message in messages -%}
|
||||||
|
{{- "<|im_start|>" + message.role + "\n" + message.content + "<|im_end|>" + "\n" -}}
|
||||||
|
{%- endfor -%}
|
||||||
|
{%- if add_generation_prompt -%}
|
||||||
|
{{- "<|im_start|>assistant\n" -}}
|
||||||
|
{%- endif -%}
|
||||||
30
config.json
Normal file
30
config.json
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"LlamaForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 128000,
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"eos_token_id": 128256,
|
||||||
|
"head_dim": 64,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 960,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 2560,
|
||||||
|
"max_position_embeddings": 4096,
|
||||||
|
"mlp_bias": false,
|
||||||
|
"model_type": "llama",
|
||||||
|
"num_attention_heads": 15,
|
||||||
|
"num_hidden_layers": 32,
|
||||||
|
"num_key_value_heads": 5,
|
||||||
|
"pad_token_id": 128256,
|
||||||
|
"pretraining_tp": 1,
|
||||||
|
"rms_norm_eps": 1e-05,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 10000.0,
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"transformers_version": "4.56.2",
|
||||||
|
"use_cache": false,
|
||||||
|
"vocab_size": 128264
|
||||||
|
}
|
||||||
11
generation_config.json
Normal file
11
generation_config.json
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
{
|
||||||
|
"_from_model_config": true,
|
||||||
|
"bos_token_id": 128000,
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
128256,
|
||||||
|
128001
|
||||||
|
],
|
||||||
|
"pad_token_id": 128256,
|
||||||
|
"transformers_version": "4.56.2"
|
||||||
|
}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:e8b1657d4822768a9365bad79efb82ff5708ae4edf055805f51ef866b2ed3d84
|
||||||
|
size 875569960
|
||||||
23
special_tokens_map.json
Normal file
23
special_tokens_map.json
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"bos_token": {
|
||||||
|
"content": "<|begin_of_text|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:3d621ce551ab3c6b3181223d69422d7c9dfbda768bcce27f77a11c63b6c16584
|
||||||
|
size 17211433
|
||||||
2127
tokenizer_config.json
Normal file
2127
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user