初始化项目，由ModelHub XC社区提供模型
Model: exnivo/tinybrain-100m-base Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
assets/tinybrain-100m-base-banner.png filter=lfs diff=lfs merge=lfs -text
|
||||
448
README.md
Normal file
448
README.md
Normal file
@@ -0,0 +1,448 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- llama
|
||||
- text-generation
|
||||
- causal-lm
|
||||
- tinybrain
|
||||
- from-scratch
|
||||
- 100m
|
||||
- base-model
|
||||
- small-language-model
|
||||
- tiny-llm
|
||||
- english
|
||||
- pretraining
|
||||
- transformers
|
||||
datasets:
|
||||
- exnivo/tinybrain-pretrain-corpus-2b
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
<img
|
||||
src="https://huggingface.co/exnivo/tinybrain-100m-base/resolve/main/assets/tinybrain-100m-base-banner.png"
|
||||
alt="TinyBrain-100M Base — Base language model for small LLMs"
|
||||
width="100%"
|
||||
/>
|
||||
</p>
|
||||
|
||||
# TinyBrain-100M Base
|
||||
|
||||
**A 103M parameter English causal language model trained from scratch.**
|
||||
|
||||
TinyBrain-100M Base is a small LLaMA-style causal language model trained from scratch on the [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) dataset.
|
||||
|
||||
This is a **base model**, not an instruct/chat model. It is intended for language modeling experiments, continued pretraining, supervised fine-tuning, and small-model research.
|
||||
|
||||
For chat or instruction-following behavior, use the instruction-tuned version:
|
||||
|
||||
[`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct)
|
||||
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
model_id = "exnivo/tinybrain-100m-base"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||||
|
||||
prompt = "Photosynthesis is the process by which plants"
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=80,
|
||||
temperature=0.8,
|
||||
top_p=0.9,
|
||||
repetition_penalty=1.08,
|
||||
do_sample=True,
|
||||
pad_token_id=tokenizer.eos_token_id,
|
||||
)
|
||||
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## At a Glance
|
||||
|
||||
| Item | Details |
|
||||
|---|---|
|
||||
| Model type | Base causal language model |
|
||||
| Parameters | 103,385,856 |
|
||||
| Approx. size | 103.4M |
|
||||
| Architecture | LLaMA-style causal transformer |
|
||||
| Language | English |
|
||||
| Context length | 2048 tokens |
|
||||
| Vocabulary size | 24,000 |
|
||||
| Tokenizer | Custom TinyBrain tokenizer |
|
||||
| Training style | From scratch pretraining |
|
||||
| Pretraining dataset | [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) |
|
||||
| Instruct dataset | [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k) |
|
||||
| Instruct model | [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct) |
|
||||
|
||||
## Model Details
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Parameters | 103.4M |
|
||||
| Architecture | `llama` / `LlamaForCausalLM` |
|
||||
| Vocabulary size | 24,000 |
|
||||
| Context length | 2048 tokens |
|
||||
| Hidden size | 768 |
|
||||
| Intermediate size | 2048 |
|
||||
| Layers | 12 |
|
||||
| Attention heads | 12 |
|
||||
| Key/value heads | 12 |
|
||||
| Activation | SiLU |
|
||||
| RMS norm epsilon | `1e-05` |
|
||||
| Tied embeddings | true |
|
||||
| BOS token | `<|bos|>` |
|
||||
| EOS token | `<|eos|>` |
|
||||
| PAD token | `<|pad|>` |
|
||||
| Dataset | [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) |
|
||||
|
||||
## Intended Use
|
||||
|
||||
TinyBrain-100M Base is intended for:
|
||||
|
||||
- small language model research
|
||||
- causal language modeling experiments
|
||||
- continued pretraining
|
||||
- supervised fine-tuning
|
||||
- instruction tuning
|
||||
- tokenizer/model experiments
|
||||
- educational small-model projects
|
||||
- comparing base vs instruct behavior
|
||||
- lightweight local model experiments
|
||||
|
||||
This model is best used as a **base checkpoint** for further training.
|
||||
|
||||
## Not Intended For
|
||||
|
||||
This model is not intended to be used directly as a finished assistant.
|
||||
|
||||
Do not rely on the base model for:
|
||||
|
||||
- polished chat behavior
|
||||
- instruction following
|
||||
- safety-critical answers
|
||||
- factual authority
|
||||
- medical, legal, or financial advice
|
||||
- live/current information
|
||||
- advanced reasoning
|
||||
- production use without evaluation
|
||||
|
||||
For assistant-style behavior, use [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct) instead.
|
||||
|
||||
## Training Data
|
||||
|
||||
TinyBrain-100M Base was trained on:
|
||||
|
||||
[`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b)
|
||||
|
||||
The pretraining corpus is a mixed-source English dataset containing factual text, educational text, math reasoning data, code data, conversation-style data, and clean web text.
|
||||
|
||||
The pretraining corpus scan found:
|
||||
|
||||
| Metric | Value |
|
||||
|---|---:|
|
||||
| Rows | 3,013,308 |
|
||||
| Characters | 7,767,447,861 |
|
||||
| Words | 1,249,832,587 |
|
||||
| Approx. tokens | ~1.81B tokenizer-independent estimate |
|
||||
|
||||
The training run used an estimated **~2.1B training tokens**. Token counts may differ depending on tokenizer, packing, filtering, and training pipeline details.
|
||||
|
||||
## Dataset Mix
|
||||
|
||||
The pretraining corpus includes these broad categories:
|
||||
|
||||
| Category | Rows | Percent |
|
||||
|---|---:|---:|
|
||||
| `factual` | 773,492 | 25.67% |
|
||||
| `educational` | 752,625 | 24.98% |
|
||||
| `math_reasoning` | 633,341 | 21.02% |
|
||||
| `code` | 326,019 | 10.82% |
|
||||
| `conversation` | 296,728 | 9.85% |
|
||||
| `clean_web` | 231,103 | 7.67% |
|
||||
|
||||
## Relationship to TinyBrain
|
||||
|
||||
TinyBrain is a small LLM project focused on compact datasets, small base models, and instruction-tuned models.
|
||||
|
||||
| Stage | Repository | Purpose |
|
||||
|---|---|---|
|
||||
| Pretraining corpus | [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b) | Base language model training data |
|
||||
| Base model | [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base) | Small causal LM trained from scratch |
|
||||
| SFT dataset | [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k) | Instruction/chat fine-tuning data |
|
||||
| Instruct model | [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct) | Chat/instruct model fine-tuned from the base model |
|
||||
|
||||
Pipeline:
|
||||
|
||||
```text
|
||||
TinyBrain Pretrain Corpus 2B
|
||||
↓
|
||||
TinyBrain-100M Base
|
||||
↓
|
||||
TinyBrain Instruct 200K
|
||||
↓
|
||||
TinyBrain-100M Instruct
|
||||
```
|
||||
|
||||
## Evaluation
|
||||
|
||||
A quick WikiText-2 evaluation was run on the base model.
|
||||
|
||||
| Metric | Value |
|
||||
|---|---:|
|
||||
| Eval tokens | 38,138 |
|
||||
| Eval text chars | 159,791 |
|
||||
| Loss | 3.7440 |
|
||||
| Perplexity | 42.27 |
|
||||
|
||||
This is a lightweight evaluation, not a full benchmark suite. Results may vary depending on evaluation script, tokenizer settings, context length, and dataset preprocessing.
|
||||
|
||||
## Base Model Behavior
|
||||
|
||||
TinyBrain-100M Base is a raw pretrained model. It can complete text, but it is not tuned to follow instructions.
|
||||
|
||||
Example base prompt:
|
||||
|
||||
```text
|
||||
Photosynthesis is the process by which plants
|
||||
```
|
||||
|
||||
The base model may continue with partially useful text, but it can also repeat, drift, hallucinate, or produce broken completions. This is expected for a small base model and is one reason instruction tuning is needed.
|
||||
|
||||
For better chat behavior, use:
|
||||
|
||||
[`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct)
|
||||
|
||||
## Recommended Generation Settings
|
||||
|
||||
For raw base-model text completion:
|
||||
|
||||
```python
|
||||
temperature = 0.8
|
||||
top_p = 0.9
|
||||
max_new_tokens = 80
|
||||
repetition_penalty = 1.08
|
||||
```
|
||||
|
||||
For more stable completions:
|
||||
|
||||
```python
|
||||
temperature = 0.5
|
||||
top_p = 0.85
|
||||
max_new_tokens = 80
|
||||
repetition_penalty = 1.1
|
||||
```
|
||||
|
||||
For deterministic testing:
|
||||
|
||||
```python
|
||||
do_sample = False
|
||||
max_new_tokens = 80
|
||||
```
|
||||
|
||||
## Example: Text Completion
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
model_id = "exnivo/tinybrain-100m-base"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||||
|
||||
prompt = "Gravity is the force that"
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=80,
|
||||
do_sample=True,
|
||||
temperature=0.8,
|
||||
top_p=0.9,
|
||||
repetition_penalty=1.08,
|
||||
pad_token_id=tokenizer.eos_token_id,
|
||||
)
|
||||
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Example: Fine-Tuning Starting Point
|
||||
|
||||
TinyBrain-100M Base can be fine-tuned on the TinyBrain SFT dataset:
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
from trl import SFTTrainer, SFTConfig
|
||||
|
||||
base_model = "exnivo/tinybrain-100m-base"
|
||||
dataset_id = "exnivo/tinybrain-instruct-sft-200k"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(base_model)
|
||||
model = AutoModelForCausalLM.from_pretrained(base_model)
|
||||
|
||||
ds = load_dataset(dataset_id, split="train")
|
||||
|
||||
def format_example(example):
|
||||
text = ""
|
||||
|
||||
for message in example["messages"]:
|
||||
role = message["role"]
|
||||
content = message["content"].strip()
|
||||
|
||||
if role == "user":
|
||||
text += f"User: {content}\n"
|
||||
elif role == "assistant":
|
||||
text += f"Assistant: {content}\n"
|
||||
|
||||
return {"text": text.strip()}
|
||||
|
||||
ds = ds.map(format_example)
|
||||
|
||||
config = SFTConfig(
|
||||
output_dir="tinybrain-100m-instruct-sft",
|
||||
dataset_text_field="text",
|
||||
max_seq_length=512,
|
||||
per_device_train_batch_size=8,
|
||||
gradient_accumulation_steps=4,
|
||||
learning_rate=2e-5,
|
||||
num_train_epochs=1,
|
||||
logging_steps=20,
|
||||
save_steps=500,
|
||||
)
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model=model,
|
||||
tokenizer=tokenizer,
|
||||
train_dataset=ds,
|
||||
args=config,
|
||||
)
|
||||
|
||||
trainer.train()
|
||||
```
|
||||
|
||||
## Training
|
||||
|
||||
TinyBrain-100M Base was trained from scratch on the TinyBrain pretraining corpus.
|
||||
|
||||
Training details:
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Training type | From-scratch causal language modeling |
|
||||
| Dataset | TinyBrain Pretrain Corpus 2B |
|
||||
| Approx. training tokens | ~2.1B |
|
||||
| Reported best validation loss | 2.6779 |
|
||||
| Training precision | bf16 |
|
||||
| Hardware | NVIDIA RTX PRO 6000 Blackwell Server Edition |
|
||||
|
||||
## Strengths
|
||||
|
||||
TinyBrain-100M Base is useful because it is:
|
||||
|
||||
- small and lightweight
|
||||
- trained from scratch
|
||||
- easy to inspect
|
||||
- easy to fine-tune
|
||||
- based on an open TinyBrain data pipeline
|
||||
- trained on a compact mixed-source corpus
|
||||
- suitable for small-model experiments
|
||||
- useful as a base checkpoint for SFT
|
||||
|
||||
## Limitations
|
||||
|
||||
TinyBrain-100M Base has important limitations.
|
||||
|
||||
The model may:
|
||||
|
||||
- hallucinate facts
|
||||
- produce broken or repetitive text
|
||||
- fail at math
|
||||
- fail at instruction following
|
||||
- misunderstand prompts
|
||||
- generate incomplete code
|
||||
- produce outdated or incorrect information
|
||||
- drift off-topic
|
||||
- repeat web/data artifacts
|
||||
|
||||
This is expected for a small **base** model. It has not been tuned to reliably follow user instructions.
|
||||
|
||||
For chat and assistant behavior, use the instruction-tuned model instead.
|
||||
|
||||
## Suggested Evaluation
|
||||
|
||||
Recommended checks:
|
||||
|
||||
- validation loss / perplexity
|
||||
- text completion quality
|
||||
- repetition behavior
|
||||
- short factual completions
|
||||
- simple math completions
|
||||
- code completion sanity checks
|
||||
- hallucination checks
|
||||
- before/after SFT comparison
|
||||
- downstream instruction-following after fine-tuning
|
||||
|
||||
Example base-model prompts:
|
||||
|
||||
```text
|
||||
Paris is the capital city of
|
||||
```
|
||||
|
||||
```text
|
||||
The Netherlands is a country in
|
||||
```
|
||||
|
||||
```text
|
||||
A cat is an animal that
|
||||
```
|
||||
|
||||
```text
|
||||
One plus one equals
|
||||
```
|
||||
|
||||
```text
|
||||
Photosynthesis is the process by which plants
|
||||
```
|
||||
|
||||
## Citation
|
||||
|
||||
If you use this model, you can cite it as:
|
||||
|
||||
```bibtex
|
||||
@misc{tinybrain_100m_base,
|
||||
title = {TinyBrain-100M Base},
|
||||
author = {exnivo},
|
||||
year = {2026},
|
||||
publisher = {Hugging Face},
|
||||
howpublished = {\url{https://huggingface.co/exnivo/tinybrain-100m-base}}
|
||||
}
|
||||
```
|
||||
|
||||
## Related Repositories
|
||||
|
||||
- Pretraining corpus: [`exnivo/tinybrain-pretrain-corpus-2b`](https://huggingface.co/datasets/exnivo/tinybrain-pretrain-corpus-2b)
|
||||
- Base model: [`exnivo/tinybrain-100m-base`](https://huggingface.co/exnivo/tinybrain-100m-base)
|
||||
- SFT dataset: [`exnivo/tinybrain-instruct-sft-200k`](https://huggingface.co/datasets/exnivo/tinybrain-instruct-sft-200k)
|
||||
- Instruct model: [`exnivo/tinybrain-100m-instruct`](https://huggingface.co/exnivo/tinybrain-100m-instruct)
|
||||
|
||||
## License
|
||||
|
||||
This model is released under the Apache 2.0 license.
|
||||
|
||||
The training dataset is mixed-source and currently listed under `license: other`. Users should review the upstream dataset licenses and source metadata before commercial use of models trained or fine-tuned from this checkpoint.
|
||||
|
||||
## Disclaimer
|
||||
|
||||
TinyBrain-100M Base is an experimental small base language model. It may produce incorrect, biased, unsafe, nonsensical, or misleading outputs.
|
||||
|
||||
Do not use it for high-stakes applications without additional training, filtering, evaluation, and safeguards.
|
||||
0
assets/.gitkeep
Normal file
0
assets/.gitkeep
Normal file
3
assets/tinybrain-100m-base-banner.png
Normal file
3
assets/tinybrain-100m-base-banner.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:026046aeca29069b800a10aa57ce460f13f6ab4b662b603ce401ccab1fbdf407
|
||||
size 1094620
|
||||
32
config.json
Normal file
32
config.json
Normal file
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"dtype": "float32",
|
||||
"eos_token_id": 2,
|
||||
"head_dim": 64,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 768,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 2048,
|
||||
"max_position_embeddings": 2048,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 12,
|
||||
"num_hidden_layers": 12,
|
||||
"num_key_value_heads": 12,
|
||||
"pad_token_id": 0,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_parameters": {
|
||||
"rope_theta": 10000.0,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "5.12.1",
|
||||
"use_cache": false,
|
||||
"vocab_size": 24000
|
||||
}
|
||||
10
generation_config.json
Normal file
10
generation_config.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"pad_token_id": 0,
|
||||
"transformers_version": "5.12.1",
|
||||
"use_cache": false
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1411777f034283d3dd6878ac969689eb6069be9e943878d14f1c5ea2377de8e5
|
||||
size 413555632
|
||||
14
special_tokens_map.json
Normal file
14
special_tokens_map.json
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"pad_token": "<|pad|>",
|
||||
"bos_token": "<|bos|>",
|
||||
"eos_token": "<|eos|>",
|
||||
"additional_special_tokens": [
|
||||
"<|user|>",
|
||||
"<|assistant|>",
|
||||
"<|system|>",
|
||||
"<|end|>",
|
||||
"<|tool|>",
|
||||
"<|tool_call|>",
|
||||
"<|tool_result|>"
|
||||
]
|
||||
}
|
||||
119061
tokenizer.json
Normal file
119061
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
19
tokenizer_config.json
Normal file
19
tokenizer_config.json
Normal file
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"tokenizer_class": "PreTrainedTokenizerFast",
|
||||
"model_max_length": 2048,
|
||||
"padding_side": "right",
|
||||
"truncation_side": "right",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"bos_token": "<|bos|>",
|
||||
"eos_token": "<|eos|>",
|
||||
"pad_token": "<|pad|>",
|
||||
"additional_special_tokens": [
|
||||
"<|user|>",
|
||||
"<|assistant|>",
|
||||
"<|system|>",
|
||||
"<|end|>",
|
||||
"<|tool|>",
|
||||
"<|tool_call|>",
|
||||
"<|tool_result|>"
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user