初始化项目，由ModelHub XC社区提供模型
Model: Itaking/itakura_v4-model Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
216
README.md
Normal file
216
README.md
Normal file
@@ -0,0 +1,216 @@
|
|||||||
|
---
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
- ja
|
||||||
|
license: apache-2.0
|
||||||
|
tags:
|
||||||
|
- olmo
|
||||||
|
- causal-lm
|
||||||
|
- bilingual
|
||||||
|
- japanese
|
||||||
|
- from-scratch
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
---
|
||||||
|
|
||||||
|
# tokenizerから自作した英語＆日本語対応モデル（アーキテクチャーはOLMo-2-386Mベース）
|
||||||
|
|
||||||
|
LLM学習の勉強用に開発したモデルです。会話にはなりません。
|
||||||
|
|
||||||
|
英日バイリンガルの小規模言語モデル。OLMo-2 アーキテクチャをベースに、**ランダム初期化から** Pretrain → CPT → SFT → DPO の4ステージパイプラインで学習しました。
|
||||||
|
|
||||||
|
事前学習済みモデルの重みをダウンロードせず、AllenAI が公開しているアーキテクチャ設定（`config.json`）のみを使用してゼロから学習しています。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## モデル概要
|
||||||
|
|
||||||
|
| 項目 | 値 |
|
||||||
|
|------|-----|
|
||||||
|
| アーキテクチャ | OLMo-2（縮小版） |
|
||||||
|
| パラメータ数 | ~386M |
|
||||||
|
| 語彙数 | 65,536（英日 BPE） |
|
||||||
|
| コンテキスト長 | 2,048 tokens |
|
||||||
|
| 精度 | bfloat16 |
|
||||||
|
| 対応言語 | 英語 / 日本語 |
|
||||||
|
|
||||||
|
### アーキテクチャ詳細
|
||||||
|
|
||||||
|
| ハイパーパラメータ | 値 |
|
||||||
|
|---|---|
|
||||||
|
| hidden_size | 1,024 |
|
||||||
|
| num_hidden_layers | 16 |
|
||||||
|
| num_attention_heads | 16 |
|
||||||
|
| num_key_value_heads | 8（GQA） |
|
||||||
|
| intermediate_size | 4,096 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 学習パイプライン
|
||||||
|
|
||||||
|
```
|
||||||
|
ランダム重み初期化
|
||||||
|
↓
|
||||||
|
Stage 1: Pretraining (~2B tokens, EN 70% / JA 30%)
|
||||||
|
↓
|
||||||
|
Stage 2: CPT / Annealing (高品質データで継続学習)
|
||||||
|
↓
|
||||||
|
Stage 3: SFT (~115K instruction samples)
|
||||||
|
↓
|
||||||
|
Stage 4: DPO (UltraFeedback preference alignment)
|
||||||
|
↓
|
||||||
|
本モデル
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stage 1: Pretraining
|
||||||
|
|
||||||
|
| 項目 | 詳細 |
|
||||||
|
|------|------|
|
||||||
|
| データ | FineWeb (EN 70%) + Wikipedia JA (JA 30%) |
|
||||||
|
| トークン数 | ~2B tokens |
|
||||||
|
| LR | 3e-4（cosine + min-LR 10%） |
|
||||||
|
| Effective batch | 128 seq ≈ 262K tokens |
|
||||||
|
| Optimizer | AdamW (weight_decay=0.1) |
|
||||||
|
|
||||||
|
### Stage 2: CPT（Continued Pretraining）
|
||||||
|
|
||||||
|
| 項目 | 詳細 |
|
||||||
|
|------|------|
|
||||||
|
| データ | FineWeb-Edu (EN 60%) + CulturaX JA (JA 40%) |
|
||||||
|
| ドキュメント数 | 500K docs |
|
||||||
|
| LR | 1e-4（cosine） |
|
||||||
|
|
||||||
|
### Stage 3: SFT（Supervised Fine-Tuning）
|
||||||
|
|
||||||
|
| データセット | サンプル数 | 言語 |
|
||||||
|
|---|---|---|
|
||||||
|
| allenai/tulu-3-sft-mixture | 70,000 | 英語 |
|
||||||
|
| kunishou/databricks-dolly-15k-ja | ~15,000 | 日本語 |
|
||||||
|
| lightblue/tagengo-gpt4（日本語のみ） | 10,000 | 日本語 |
|
||||||
|
| izumi-lab/llm-japanese-dataset | 20,000 | 日本語 |
|
||||||
|
| **合計** | **~115,000** | |
|
||||||
|
|
||||||
|
| 項目 | 値 |
|
||||||
|
|---|---|
|
||||||
|
| LR | 5e-6（linear） |
|
||||||
|
| Epochs | 2 |
|
||||||
|
| Effective batch | 64 |
|
||||||
|
| Loss | Cross-Entropy（assistant turns のみ） |
|
||||||
|
|
||||||
|
### Stage 4: DPO（Direct Preference Optimization）
|
||||||
|
|
||||||
|
| 項目 | 詳細 |
|
||||||
|
|------|------|
|
||||||
|
| データ | allenai/ultrafeedback_binarized_cleaned |
|
||||||
|
| Loss | DPO sigmoid（β=0.1） |
|
||||||
|
| LR | 5e-7（linear） |
|
||||||
|
| Epochs | 1 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## トークナイザー
|
||||||
|
|
||||||
|
英日バイリンガル BPE トークナイザーをゼロから学習しました。
|
||||||
|
|
||||||
|
| 項目 | 値 |
|
||||||
|
|---|---|
|
||||||
|
| 種類 | ByteLevel BPE |
|
||||||
|
| 語彙数 | 65,536（= 2¹⁶） |
|
||||||
|
| 学習データ | FineWeb 60K docs + Wikipedia JA 60K docs |
|
||||||
|
| 特殊トークン | `<\|endoftext\|>`, `<\|padding\|>`, `<\|user\|>`, `<\|assistant\|>`, `<\|system\|>` |
|
||||||
|
|
||||||
|
OLMo-2 のオリジナルトークナイザー（vocab=50,280）と比べて日本語トークン効率が大幅に向上しています。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 使い方
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||||
|
import torch
|
||||||
|
|
||||||
|
model_id = "itacora/olmo2-386m-ja"
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
model_id,
|
||||||
|
torch_dtype=torch.bfloat16,
|
||||||
|
device_map="auto",
|
||||||
|
)
|
||||||
|
|
||||||
|
messages = [{"role": "user", "content": "機械学習とは何ですか？"}]
|
||||||
|
input_text = tokenizer.apply_chat_template(
|
||||||
|
messages, tokenize=False, add_generation_prompt=True
|
||||||
|
)
|
||||||
|
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
|
||||||
|
inputs.pop("token_type_ids", None)
|
||||||
|
|
||||||
|
with torch.inference_mode():
|
||||||
|
output = model.generate(
|
||||||
|
**inputs,
|
||||||
|
max_new_tokens=200,
|
||||||
|
temperature=0.7,
|
||||||
|
top_p=0.9,
|
||||||
|
do_sample=True,
|
||||||
|
repetition_penalty=1.3,
|
||||||
|
)
|
||||||
|
|
||||||
|
response = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
|
||||||
|
print(response)
|
||||||
|
```
|
||||||
|
|
||||||
|
### チャットテンプレート
|
||||||
|
|
||||||
|
```
|
||||||
|
<|user|>
|
||||||
|
{ユーザーの入力}
|
||||||
|
<|assistant|>
|
||||||
|
{モデルの返答}<|endoftext|>
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 学習環境
|
||||||
|
|
||||||
|
| 項目 | 詳細 |
|
||||||
|
|------|------|
|
||||||
|
| GPU | NVIDIA RTX 4090 (24GB VRAM) |
|
||||||
|
| フレームワーク | PyTorch 2.4 + Transformers + TRL |
|
||||||
|
| 精度 | bfloat16 + gradient checkpointing |
|
||||||
|
| プラットフォーム | RunPod |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 限界と注意事項
|
||||||
|
|
||||||
|
- **Pretrain データ量**: Chinchilla 最適値（386M × 20 = ~8B tokens）に対して約 2B tokens のみ。言語能力・知識量ともに制限があります。
|
||||||
|
- **繰り返し**: 長い生成では同じ文が繰り返されることがあります。`repetition_penalty=1.3` の使用を推奨します。
|
||||||
|
- **事実の誤り**: 知識が不十分なため、誤った情報を生成する場合があります。重要な用途には使用しないでください。
|
||||||
|
- **日本語品質**: SFT データの一部（izumi-lab）に `【P】` 等のフォーマットマーカーが含まれており、日本語生成に影響することがあります。
|
||||||
|
- **研究・実験目的**: 本モデルはフルパイプラインの動作確認・学習実験を目的として作成されました。プロダクション用途には適しません。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ライセンス
|
||||||
|
|
||||||
|
Apache 2.0
|
||||||
|
|
||||||
|
学習データに使用した各データセットのライセンスも確認してください：
|
||||||
|
- [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb): ODC-By
|
||||||
|
- [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia): CC BY-SA 4.0
|
||||||
|
- [Tülu-3 SFT mixture](https://huggingface.co/datasets/allenai/tulu-3-sft-mixture): ODC-BY
|
||||||
|
- [UltraFeedback](https://huggingface.co/datasets/allenai/ultrafeedback_binarized_cleaned): MIT
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 引用
|
||||||
|
|
||||||
|
本モデルは以下の研究・成果物をベースにしています：
|
||||||
|
|
||||||
|
```bibtex
|
||||||
|
@article{olmo2,
|
||||||
|
title={OLMo 2: The best fully open language model to date},
|
||||||
|
author={Team OLMo},
|
||||||
|
year={2025},
|
||||||
|
url={https://arxiv.org/abs/2501.00656}
|
||||||
|
}
|
||||||
|
```
|
||||||
28
checkpoint-3801/config.json
Normal file
28
checkpoint-3801/config.json
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
{
|
||||||
|
"_name_or_path": "checkpoints/sft",
|
||||||
|
"architectures": [
|
||||||
|
"Olmo2ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 0,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 1024,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 4096,
|
||||||
|
"max_position_embeddings": 4096,
|
||||||
|
"model_type": "olmo2",
|
||||||
|
"num_attention_heads": 16,
|
||||||
|
"num_hidden_layers": 16,
|
||||||
|
"num_key_value_heads": 8,
|
||||||
|
"pad_token_id": 1,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 500000,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"transformers_version": "4.47.0",
|
||||||
|
"use_cache": false,
|
||||||
|
"vocab_size": 65536
|
||||||
|
}
|
||||||
7
checkpoint-3801/generation_config.json
Normal file
7
checkpoint-3801/generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"_from_model_config": true,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 0,
|
||||||
|
"pad_token_id": 1,
|
||||||
|
"transformers_version": "4.47.0"
|
||||||
|
}
|
||||||
3
checkpoint-3801/model.safetensors
Normal file
3
checkpoint-3801/model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:1c835569fc6868581fcb1a71c0fcbef34a4d650c31e3ca5e414cb358adb32e26
|
||||||
|
size 771889040
|
||||||
3
checkpoint-3801/optimizer.pt
Normal file
3
checkpoint-3801/optimizer.pt
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:f9f76f8e03ed1a9ce329c7c9bcbaa0e4c59f3ad13e5b81731ee889695b116cdc
|
||||||
|
size 1543887930
|
||||||
3
checkpoint-3801/rng_state.pth
Normal file
3
checkpoint-3801/rng_state.pth
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:386fcc8cc1089aade9450d86fb239ea3483f455fd2d78d8378645feecfec9d69
|
||||||
|
size 14244
|
||||||
3
checkpoint-3801/scheduler.pt
Normal file
3
checkpoint-3801/scheduler.pt
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:11199a2056eae24a7b539975c36ca618910c6dd69adada7b32cafa7546a58268
|
||||||
|
size 1064
|
||||||
35
checkpoint-3801/special_tokens_map.json
Normal file
35
checkpoint-3801/special_tokens_map.json
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
{
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|user|>",
|
||||||
|
"<|assistant|>",
|
||||||
|
"<|system|>"
|
||||||
|
],
|
||||||
|
"bos_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<|padding|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"unk_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
326716
checkpoint-3801/tokenizer.json
Normal file
326716
checkpoint-3801/tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
58
checkpoint-3801/tokenizer_config.json
Normal file
58
checkpoint-3801/tokenizer_config.json
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
{
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"0": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"1": {
|
||||||
|
"content": "<|padding|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"2": {
|
||||||
|
"content": "<|user|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"3": {
|
||||||
|
"content": "<|assistant|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"4": {
|
||||||
|
"content": "<|system|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|user|>",
|
||||||
|
"<|assistant|>",
|
||||||
|
"<|system|>"
|
||||||
|
],
|
||||||
|
"bos_token": "<|endoftext|>",
|
||||||
|
"chat_template": "{% for message in messages %}{% if message['role'] == 'system' %}<|system|>\n{{ message['content'] }}\n{% elif message['role'] == 'user' %}<|user|>\n{{ message['content'] }}\n{% elif message['role'] == 'assistant' %}<|assistant|>\n{{ message['content'] }}{{ eos_token }}\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|assistant|>\n{% endif %}",
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|endoftext|>",
|
||||||
|
"extra_special_tokens": {},
|
||||||
|
"model_max_length": 1000000000000000019884624838656,
|
||||||
|
"pad_token": "<|padding|>",
|
||||||
|
"tokenizer_class": "PreTrainedTokenizerFast",
|
||||||
|
"unk_token": "<|endoftext|>"
|
||||||
|
}
|
||||||
57048
checkpoint-3801/trainer_state.json
Normal file
57048
checkpoint-3801/trainer_state.json
Normal file
File diff suppressed because it is too large
Load Diff
3
checkpoint-3801/training_args.bin
Normal file
3
checkpoint-3801/training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:af42ef9bfcda66ea7d72451bdfc9db238c592215d63e2850bd2f59c36d29b64a
|
||||||
|
size 6200
|
||||||
28
config.json
Normal file
28
config.json
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
{
|
||||||
|
"_name_or_path": "checkpoints/sft",
|
||||||
|
"architectures": [
|
||||||
|
"Olmo2ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 0,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 1024,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 4096,
|
||||||
|
"max_position_embeddings": 4096,
|
||||||
|
"model_type": "olmo2",
|
||||||
|
"num_attention_heads": 16,
|
||||||
|
"num_hidden_layers": 16,
|
||||||
|
"num_key_value_heads": 8,
|
||||||
|
"pad_token_id": 1,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 500000,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"transformers_version": "4.47.0",
|
||||||
|
"use_cache": false,
|
||||||
|
"vocab_size": 65536
|
||||||
|
}
|
||||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"_from_model_config": true,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 0,
|
||||||
|
"pad_token_id": 1,
|
||||||
|
"transformers_version": "4.47.0"
|
||||||
|
}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:1c835569fc6868581fcb1a71c0fcbef34a4d650c31e3ca5e414cb358adb32e26
|
||||||
|
size 771889040
|
||||||
35
special_tokens_map.json
Normal file
35
special_tokens_map.json
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
{
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|user|>",
|
||||||
|
"<|assistant|>",
|
||||||
|
"<|system|>"
|
||||||
|
],
|
||||||
|
"bos_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<|padding|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"unk_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
326716
tokenizer.json
Normal file
326716
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
58
tokenizer_config.json
Normal file
58
tokenizer_config.json
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
{
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"0": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"1": {
|
||||||
|
"content": "<|padding|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"2": {
|
||||||
|
"content": "<|user|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"3": {
|
||||||
|
"content": "<|assistant|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"4": {
|
||||||
|
"content": "<|system|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|user|>",
|
||||||
|
"<|assistant|>",
|
||||||
|
"<|system|>"
|
||||||
|
],
|
||||||
|
"bos_token": "<|endoftext|>",
|
||||||
|
"chat_template": "{% for message in messages %}{% if message['role'] == 'system' %}<|system|>\n{{ message['content'] }}\n{% elif message['role'] == 'user' %}<|user|>\n{{ message['content'] }}\n{% elif message['role'] == 'assistant' %}<|assistant|>\n{{ message['content'] }}{{ eos_token }}\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|assistant|>\n{% endif %}",
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|endoftext|>",
|
||||||
|
"extra_special_tokens": {},
|
||||||
|
"model_max_length": 1000000000000000019884624838656,
|
||||||
|
"pad_token": "<|padding|>",
|
||||||
|
"tokenizer_class": "PreTrainedTokenizerFast",
|
||||||
|
"unk_token": "<|endoftext|>"
|
||||||
|
}
|
||||||
3
training_args.bin
Normal file
3
training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:af42ef9bfcda66ea7d72451bdfc9db238c592215d63e2850bd2f59c36d29b64a
|
||||||
|
size 6200
|
||||||
Reference in New Issue
Block a user