初始化项目，由ModelHub XC社区提供模型
Model: patrickcmd/orpheus-3b-tts-multilingual Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
658
README.md
Normal file
658
README.md
Normal file
@@ -0,0 +1,658 @@
|
|||||||
|
---
|
||||||
|
license: apache-2.0
|
||||||
|
language:
|
||||||
|
- ach
|
||||||
|
- af
|
||||||
|
- en
|
||||||
|
- ee
|
||||||
|
- ff
|
||||||
|
- ha
|
||||||
|
- ig
|
||||||
|
- ki
|
||||||
|
- rw
|
||||||
|
- lgg
|
||||||
|
- ln
|
||||||
|
- lg
|
||||||
|
- luo
|
||||||
|
- nyn
|
||||||
|
- st
|
||||||
|
- sw
|
||||||
|
- teo
|
||||||
|
- tn
|
||||||
|
- xh
|
||||||
|
- yo
|
||||||
|
library_name: transformers
|
||||||
|
pipeline_tag: text-to-speech
|
||||||
|
tags:
|
||||||
|
- text-to-speech
|
||||||
|
- tts
|
||||||
|
- orpheus
|
||||||
|
- multilingual
|
||||||
|
- multi-speaker
|
||||||
|
- african-languages
|
||||||
|
- low-resource-language
|
||||||
|
- sunbird
|
||||||
|
- snac
|
||||||
|
- unsloth
|
||||||
|
- llama
|
||||||
|
datasets:
|
||||||
|
- Sunbird/tts
|
||||||
|
base_model: unsloth/orpheus-3b-0.1-pretrained
|
||||||
|
---
|
||||||
|
|
||||||
|
# Orpheus-3B Sunbird Multilingual TTS
|
||||||
|
|
||||||
|
A multilingual, multi-speaker text-to-speech model fine-tuned from
|
||||||
|
[`unsloth/orpheus-3b-0.1-pretrained`](https://huggingface.co/unsloth/orpheus-3b-0.1-pretrained)
|
||||||
|
on the **full** [`Sunbird/tts`](https://huggingface.co/datasets/Sunbird/tts)
|
||||||
|
corpus — 20 language configurations and every speaker present in the
|
||||||
|
dataset.
|
||||||
|
|
||||||
|
The model accepts arbitrary text and emits 24 kHz mono speech via the
|
||||||
|
[SNAC](https://huggingface.co/hubertsiuzdak/snac_24khz) audio codec.
|
||||||
|
Voice selection happens at the prompt level: prepend the chosen
|
||||||
|
`speaker_id` followed by `": "` to your text, and the model produces
|
||||||
|
audio in that speaker's voice.
|
||||||
|
|
||||||
|
## Quick links
|
||||||
|
|
||||||
|
- **Base model:** [`unsloth/orpheus-3b-0.1-pretrained`](https://huggingface.co/unsloth/orpheus-3b-0.1-pretrained) (Llama-3 architecture)
|
||||||
|
- **Audio codec:** [`hubertsiuzdak/snac_24khz`](https://huggingface.co/hubertsiuzdak/snac_24khz) (24 kHz, 7 codes per ~12 ms frame)
|
||||||
|
- **Training dataset:** [`Sunbird/tts`](https://huggingface.co/datasets/Sunbird/tts) — all 20 configs, all speakers
|
||||||
|
- **Training framework:** [Unsloth](https://github.com/unslothai/unsloth) + HuggingFace Trainer
|
||||||
|
|
||||||
|
## Languages covered
|
||||||
|
|
||||||
|
Speaker IDs encode both the source corpus (`salt_*`, `waxal_*`, `slr32_*`,
|
||||||
|
`slr129_*`, `bateesa_*`) and the language. Languages marked with an em dash
|
||||||
|
in the Speaker IDs column are present in the model's training mix but do
|
||||||
|
not currently expose individual voice IDs in this checkpoint.
|
||||||
|
|
||||||
|
| Config | Language | ISO 639-1 | Region | Speaker IDs |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `ach` | Acholi | — | Uganda, South Sudan | `salt_ach_0001`<br>`waxal_ach_0001`<br>`waxal_ach_0005`<br>`waxal_ach_0006`<br>`waxal_ach_0008` |
|
||||||
|
| `afr` | Afrikaans | af | South Africa, Namibia | `slr32_afr_0009` |
|
||||||
|
| `eng` | English | en | (control language) | `salt_eng_0001`<br>`salt_eng_0002`<br>`salt_eng_0003` |
|
||||||
|
| `ewe` | Ewe | ee | Ghana, Togo | `slr129_ewe_0001` |
|
||||||
|
| `ful` | Fulah | ff | West Africa (Sahel) | `waxal_ful_0003`<br>`waxal_ful_0004`<br>`waxal_ful_0006` |
|
||||||
|
| `hau` | Hausa | ha | Nigeria, Niger, Chad | `waxal_hau_0004`<br>`waxal_hau_0006`<br>`waxal_hau_0007`<br>`waxal_hau_0008` |
|
||||||
|
| `ibo` | Igbo | ig | Nigeria | `waxal_ibo_0003`<br>`waxal_ibo_0005`<br>`waxal_ibo_0008` |
|
||||||
|
| `kik` | Kikuyu | ki | Kenya | `waxal_kik_0003`<br>`waxal_kik_0004` |
|
||||||
|
| `kin` | Kinyarwanda | rw | Rwanda | `bateesa_kin_0001` |
|
||||||
|
| `lgg` | Lugbara | — | Uganda, DRC | — |
|
||||||
|
| `lin` | Lingala | ln | DRC, Republic of Congo | `slr129_lin_0001` |
|
||||||
|
| `lug` | Luganda | lg | Uganda | `salt_lug_0001`<br>`waxal_lug_0002`<br>`waxal_lug_0003`<br>`waxal_lug_0004`<br>`waxal_lug_0005`<br>`waxal_lug_0006`<br>`waxal_lug_0007`<br>`waxal_lug_0008` |
|
||||||
|
| `luo` | Luo (Dholuo) | — | Kenya, Tanzania | `waxal_luo_0001`<br>`waxal_luo_0002`<br>`waxal_luo_0003`<br>`waxal_luo_0004` |
|
||||||
|
| `nyn` | Runyankole | — | Uganda | `salt_nyn_0001`<br>`waxal_nyn_0003`<br>`waxal_nyn_0004`<br>`waxal_nyn_0007`<br>`waxal_nyn_0008` |
|
||||||
|
| `sot` | Sesotho | st | Lesotho, South Africa | — |
|
||||||
|
| `swa` | Swahili | sw | East Africa | `waxal_swa_0006`<br>`waxal_swa_0007` |
|
||||||
|
| `teo` | Ateso | — | Uganda, Kenya | `salt_teo_0001` |
|
||||||
|
| `tsn` | Setswana | tn | Botswana, South Africa | — |
|
||||||
|
| `xho` | Xhosa | xh | South Africa | `slr32_xho_0012` |
|
||||||
|
| `yor` | Yoruba | yo | Nigeria, Benin | `waxal_yor_0002`<br>`waxal_yor_0006`<br>`waxal_yor_0008` |
|
||||||
|
|
||||||
|
Per-language quality scales with the amount of training data Sunbird
|
||||||
|
collected for that language; some configs have many more speaker hours
|
||||||
|
than others. **Audition the test split** for each language before relying
|
||||||
|
on a particular speaker — see the discovery snippet below.
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
```python
|
||||||
|
# After installing the dependencies (see "Inference" below)
|
||||||
|
wav = synthesize("Mwattu, oli otya?", speaker_id="salt_lug_0001") # Luganda
|
||||||
|
wav = synthesize("Habari yako rafiki.", speaker_id="salt_swa_0001") # Swahili
|
||||||
|
wav = synthesize("Bawo ni, ọrẹ mi?", speaker_id="salt_yor_0001") # Yoruba
|
||||||
|
```
|
||||||
|
|
||||||
|
The model has no explicit "language" knob — the language identity
|
||||||
|
travels via the speaker tag, since each `salt_<lang>_<NNNN>` voice was
|
||||||
|
recorded in exactly one language.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Discovering speaker IDs
|
||||||
|
|
||||||
|
The exact speaker_ids in each config can be enumerated from the dataset:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from collections import defaultdict
|
||||||
|
from datasets import load_dataset, get_dataset_config_names
|
||||||
|
|
||||||
|
CONFIGS = get_dataset_config_names("Sunbird/tts") # the 20 languages
|
||||||
|
|
||||||
|
speakers_by_lang = defaultdict(set)
|
||||||
|
for cfg in CONFIGS:
|
||||||
|
ds = load_dataset("Sunbird/tts", cfg, split="train")
|
||||||
|
for sid in ds["speaker_id"]:
|
||||||
|
speakers_by_lang[cfg].add(sid)
|
||||||
|
|
||||||
|
for cfg, sids in sorted(speakers_by_lang.items()):
|
||||||
|
print(f"{cfg}: {len(sids)} speaker(s) — {sorted(sids)[:3]}{'...' if len(sids) > 3 else ''}")
|
||||||
|
```
|
||||||
|
|
||||||
|
Speaker IDs follow the pattern `salt_<lang>_<NNNN>` (e.g.,
|
||||||
|
`salt_lug_0001`, `salt_ach_0007`). Pass any one of them as
|
||||||
|
`speaker_id` to either inference function below.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Inference
|
||||||
|
|
||||||
|
The model wraps every prompt in a multi-speaker tagged format:
|
||||||
|
|
||||||
|
```
|
||||||
|
[SOH] + tokenize("<speaker_id>: <your text>") + [EOT, EOH]
|
||||||
|
```
|
||||||
|
|
||||||
|
and the model autoregressively emits Llama-3 special tokens followed by
|
||||||
|
SNAC audio codes that decode to a 24 kHz waveform. Two reference
|
||||||
|
implementations follow.
|
||||||
|
|
||||||
|
### Option A — `transformers` + `unsloth` (single request)
|
||||||
|
|
||||||
|
Best for development, notebook-driven iteration, and small batch sizes.
|
||||||
|
|
||||||
|
**Install:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install unsloth snac soundfile torchcodec "datasets>=3.4.1,<4.0.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Run:**
|
||||||
|
|
||||||
|
```python
|
||||||
|
import os
|
||||||
|
import numpy as np
|
||||||
|
import torch
|
||||||
|
import soundfile as sf
|
||||||
|
from unsloth import FastLanguageModel
|
||||||
|
from snac import SNAC
|
||||||
|
|
||||||
|
MODEL_ID = "sunbird/orpheus-3b-tts-multilingual"
|
||||||
|
|
||||||
|
# Special tokens — must match the training format
|
||||||
|
END_OF_TEXT = 128009
|
||||||
|
START_OF_SPEECH = 128257
|
||||||
|
END_OF_SPEECH = 128258
|
||||||
|
START_OF_HUMAN = 128259
|
||||||
|
END_OF_HUMAN = 128260
|
||||||
|
PAD_TOKEN = 128263
|
||||||
|
AUDIO_TOKEN_LO = 128266
|
||||||
|
AUDIO_TOKEN_HI = 128266 + 7 * 4096 # exclusive
|
||||||
|
|
||||||
|
# 1) Load the LM (LoRA already merged into 16-bit weights at training time)
|
||||||
|
model, tokenizer = FastLanguageModel.from_pretrained(
|
||||||
|
model_name = MODEL_ID,
|
||||||
|
max_seq_length = 4096,
|
||||||
|
dtype = None, # auto bf16 / fp16
|
||||||
|
load_in_4bit = False, # set True to halve VRAM at slight quality cost
|
||||||
|
token = os.environ.get("HF_TOKEN"),
|
||||||
|
)
|
||||||
|
FastLanguageModel.for_inference(model)
|
||||||
|
|
||||||
|
# 2) Load SNAC decoder (CPU is fine — frees GPU for the LM)
|
||||||
|
snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").to("cpu")
|
||||||
|
|
||||||
|
|
||||||
|
def _redistribute_codes(code_list: list[int]) -> torch.Tensor:
|
||||||
|
layer_1, layer_2, layer_3 = [], [], []
|
||||||
|
for i in range(len(code_list) // 7):
|
||||||
|
layer_1.append(code_list[7*i])
|
||||||
|
layer_2.append(code_list[7*i + 1] - 4096)
|
||||||
|
layer_3.append(code_list[7*i + 2] - 2*4096)
|
||||||
|
layer_3.append(code_list[7*i + 3] - 3*4096)
|
||||||
|
layer_2.append(code_list[7*i + 4] - 4*4096)
|
||||||
|
layer_3.append(code_list[7*i + 5] - 5*4096)
|
||||||
|
layer_3.append(code_list[7*i + 6] - 6*4096)
|
||||||
|
if not layer_1:
|
||||||
|
return torch.zeros(1, 1, 12000) # ~0.5s silence fallback
|
||||||
|
clamp = lambda vals: [max(0, min(4095, v)) for v in vals]
|
||||||
|
codes = [torch.tensor(clamp(layer_1)).unsqueeze(0),
|
||||||
|
torch.tensor(clamp(layer_2)).unsqueeze(0),
|
||||||
|
torch.tensor(clamp(layer_3)).unsqueeze(0)]
|
||||||
|
return snac_model.decode(codes)
|
||||||
|
|
||||||
|
|
||||||
|
def synthesize(text: str, speaker_id: str,
|
||||||
|
*, max_new_tokens: int = 1200,
|
||||||
|
temperature: float = 0.6, top_p: float = 0.95,
|
||||||
|
repetition_penalty: float = 1.1,
|
||||||
|
seed: int | None = None) -> np.ndarray:
|
||||||
|
"""Synthesize speech for `text` in the voice of `speaker_id`.
|
||||||
|
|
||||||
|
`speaker_id` must be one of the speakers seen during training,
|
||||||
|
e.g. "salt_lug_0001" (Luganda) or "salt_swa_0003" (Swahili).
|
||||||
|
"""
|
||||||
|
if seed is not None:
|
||||||
|
torch.manual_seed(seed)
|
||||||
|
|
||||||
|
tagged = f"{speaker_id}: {text}"
|
||||||
|
text_ids = tokenizer(tagged, return_tensors="pt").input_ids
|
||||||
|
soh = torch.tensor([[START_OF_HUMAN]], dtype=torch.int64)
|
||||||
|
end = torch.tensor([[END_OF_TEXT, END_OF_HUMAN]], dtype=torch.int64)
|
||||||
|
input_ids = torch.cat([soh, text_ids, end], dim=1).to("cuda")
|
||||||
|
attention_mask = torch.ones_like(input_ids)
|
||||||
|
|
||||||
|
generated = model.generate(
|
||||||
|
input_ids = input_ids, attention_mask = attention_mask,
|
||||||
|
max_new_tokens = max_new_tokens,
|
||||||
|
do_sample = True,
|
||||||
|
temperature = temperature, top_p = top_p,
|
||||||
|
repetition_penalty = repetition_penalty,
|
||||||
|
eos_token_id = END_OF_SPEECH, use_cache = True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Crop on last SOS, filter to audio token range, redistribute, decode
|
||||||
|
sos_indices = (generated == START_OF_SPEECH).nonzero(as_tuple=True)
|
||||||
|
cropped = generated[:, sos_indices[1][-1].item() + 1:] if len(sos_indices[1]) > 0 else generated
|
||||||
|
row = cropped[0]
|
||||||
|
audio_only = row[(row >= AUDIO_TOKEN_LO) & (row < AUDIO_TOKEN_HI)]
|
||||||
|
n = (audio_only.size(0) // 7) * 7
|
||||||
|
code_list = [t.item() - AUDIO_TOKEN_LO for t in audio_only[:n]]
|
||||||
|
waveform = _redistribute_codes(code_list)
|
||||||
|
return waveform.detach().squeeze().to("cpu").numpy().astype(np.float32)
|
||||||
|
|
||||||
|
|
||||||
|
# 3) Use it — pick a speaker per language
|
||||||
|
wav = synthesize("Mwattu, Mukama yeebazibwe.", speaker_id="salt_lug_0001", seed=42)
|
||||||
|
sf.write("luganda.wav", wav, 24000)
|
||||||
|
|
||||||
|
wav = synthesize("Habari yako rafiki.", speaker_id="salt_swa_0001", seed=42)
|
||||||
|
sf.write("swahili.wav", wav, 24000)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option B — `vllm` (high throughput, batched, deployment)
|
||||||
|
|
||||||
|
Best for serving traffic. PagedAttention + continuous batching gives
|
||||||
|
roughly **5–10× faster** single-request latency and **10–100× higher**
|
||||||
|
throughput on batched requests vs. the `transformers` path. Multi-speaker
|
||||||
|
batching (different `speaker_id`s in one call) gets the full benefit.
|
||||||
|
|
||||||
|
> **Important:** vLLM ships its own torch/transformers and conflicts
|
||||||
|
> with Unsloth's pinned versions. Use a fresh Python environment for
|
||||||
|
> vLLM serving — do not install on top of an Unsloth env.
|
||||||
|
|
||||||
|
**Install:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install vllm snac soundfile torchcodec "datasets>=3.4.1,<4.0.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Run:**
|
||||||
|
|
||||||
|
```python
|
||||||
|
import os
|
||||||
|
import numpy as np
|
||||||
|
import torch
|
||||||
|
import soundfile as sf
|
||||||
|
from snac import SNAC
|
||||||
|
from transformers import AutoTokenizer
|
||||||
|
from vllm import LLM, SamplingParams
|
||||||
|
|
||||||
|
MODEL_ID = "sunbird/orpheus-3b-tts-multilingual"
|
||||||
|
|
||||||
|
END_OF_TEXT = 128009
|
||||||
|
START_OF_SPEECH = 128257
|
||||||
|
END_OF_SPEECH = 128258
|
||||||
|
START_OF_HUMAN = 128259
|
||||||
|
END_OF_HUMAN = 128260
|
||||||
|
AUDIO_TOKEN_LO = 128266
|
||||||
|
AUDIO_TOKEN_HI = 128266 + 7 * 4096
|
||||||
|
|
||||||
|
# 1) Load
|
||||||
|
llm = LLM(
|
||||||
|
model = MODEL_ID,
|
||||||
|
dtype = "bfloat16",
|
||||||
|
max_model_len = 4096,
|
||||||
|
gpu_memory_utilization = 0.85,
|
||||||
|
)
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=os.environ.get("HF_TOKEN"))
|
||||||
|
snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").to("cpu")
|
||||||
|
|
||||||
|
|
||||||
|
def _build_prompt_token_ids(text: str, speaker_id: str) -> list[int]:
|
||||||
|
tagged = f"{speaker_id}: {text}"
|
||||||
|
text_ids = tokenizer.encode(tagged, add_special_tokens=True)
|
||||||
|
return [START_OF_HUMAN] + text_ids + [END_OF_TEXT, END_OF_HUMAN]
|
||||||
|
|
||||||
|
|
||||||
|
def _codes_to_waveform(generated_token_ids: list[int]) -> np.ndarray:
|
||||||
|
ids = torch.tensor(generated_token_ids, dtype=torch.int64)
|
||||||
|
sos_pos = (ids == START_OF_SPEECH).nonzero(as_tuple=True)[0]
|
||||||
|
if len(sos_pos) > 0:
|
||||||
|
ids = ids[sos_pos[-1].item() + 1:]
|
||||||
|
audio = ids[(ids >= AUDIO_TOKEN_LO) & (ids < AUDIO_TOKEN_HI)]
|
||||||
|
n = (audio.size(0) // 7) * 7
|
||||||
|
cl = [t.item() - AUDIO_TOKEN_LO for t in audio[:n]]
|
||||||
|
l1, l2, l3 = [], [], []
|
||||||
|
for i in range(len(cl) // 7):
|
||||||
|
l1.append(cl[7*i])
|
||||||
|
l2.append(cl[7*i+1] - 4096); l3.append(cl[7*i+2] - 2*4096)
|
||||||
|
l3.append(cl[7*i+3] - 3*4096); l2.append(cl[7*i+4] - 4*4096)
|
||||||
|
l3.append(cl[7*i+5] - 5*4096); l3.append(cl[7*i+6] - 6*4096)
|
||||||
|
if not l1:
|
||||||
|
return np.zeros(12000, dtype=np.float32)
|
||||||
|
cb = lambda v: [max(0, min(4095, x)) for x in v]
|
||||||
|
codes = [torch.tensor(cb(l1)).unsqueeze(0),
|
||||||
|
torch.tensor(cb(l2)).unsqueeze(0),
|
||||||
|
torch.tensor(cb(l3)).unsqueeze(0)]
|
||||||
|
return snac_model.decode(codes).detach().squeeze().cpu().numpy().astype(np.float32)
|
||||||
|
|
||||||
|
|
||||||
|
def synthesize(text: str, speaker_id: str,
|
||||||
|
*, max_tokens: int = 1200,
|
||||||
|
temperature: float = 0.6, top_p: float = 0.95,
|
||||||
|
repetition_penalty: float = 1.1,
|
||||||
|
seed: int | None = None) -> np.ndarray:
|
||||||
|
sp = SamplingParams(
|
||||||
|
temperature = temperature, top_p = top_p,
|
||||||
|
repetition_penalty = repetition_penalty,
|
||||||
|
max_tokens = max_tokens,
|
||||||
|
stop_token_ids = [END_OF_SPEECH],
|
||||||
|
skip_special_tokens = False,
|
||||||
|
seed = seed,
|
||||||
|
)
|
||||||
|
pids = _build_prompt_token_ids(text, speaker_id)
|
||||||
|
out = llm.generate([{"prompt_token_ids": pids}], sp)
|
||||||
|
return _codes_to_waveform(list(out[0].outputs[0].token_ids))
|
||||||
|
|
||||||
|
|
||||||
|
def synthesize_batch(items: list[dict], **kwargs) -> list[np.ndarray]:
|
||||||
|
"""items: list of {"text": str, "speaker_id": str} — different speakers
|
||||||
|
can be mixed in one batch."""
|
||||||
|
sp = SamplingParams(
|
||||||
|
temperature = kwargs.get("temperature", 0.6),
|
||||||
|
top_p = kwargs.get("top_p", 0.95),
|
||||||
|
repetition_penalty = kwargs.get("repetition_penalty", 1.1),
|
||||||
|
max_tokens = kwargs.get("max_tokens", 1200),
|
||||||
|
stop_token_ids = [END_OF_SPEECH],
|
||||||
|
skip_special_tokens = False,
|
||||||
|
seed = kwargs.get("seed"),
|
||||||
|
)
|
||||||
|
prompts = [{"prompt_token_ids": _build_prompt_token_ids(it["text"], it["speaker_id"])}
|
||||||
|
for it in items]
|
||||||
|
outputs = llm.generate(prompts, sp)
|
||||||
|
return [_codes_to_waveform(list(o.outputs[0].token_ids)) for o in outputs]
|
||||||
|
|
||||||
|
|
||||||
|
# 2) Single — pick a speaker per language
|
||||||
|
wav = synthesize("Mwattu, oli otya?", speaker_id="salt_lug_0001", seed=42)
|
||||||
|
sf.write("luganda.wav", wav, 24000)
|
||||||
|
|
||||||
|
# 3) Batched — different languages and speakers in one GPU pass
|
||||||
|
items = [
|
||||||
|
{"text": "Mwattu, oli otya?", "speaker_id": "salt_lug_0001"},
|
||||||
|
{"text": "Habari yako rafiki.", "speaker_id": "waxal_swa_0006"},
|
||||||
|
{"text": "Bawo ni, ọrẹ mi?", "speaker_id": "waxal_yor_0002"},
|
||||||
|
{"text": "Sannu, ina kwana?", "speaker_id": "waxal_hau_0004"},
|
||||||
|
{"text": "Goeie môre, hoe gaan dit?", "speaker_id": "slr32_afr_0009"},
|
||||||
|
]
|
||||||
|
wavs = synthesize_batch(items, seed=123)
|
||||||
|
for i, (it, w) in enumerate(zip(items, wavs)):
|
||||||
|
sf.write(f"batch_{i:02d}_{it['speaker_id']}.wav", w, 24000)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Generation parameters
|
||||||
|
|
||||||
|
| Param | Default | What it does |
|
||||||
|
|---|---|---|
|
||||||
|
| `temperature` | 0.6 | Lower = more deterministic, slightly flatter prosody. |
|
||||||
|
| `top_p` | 0.95 | Nucleus sampling. Don't drop below 0.9 — produces robotic audio. |
|
||||||
|
| `repetition_penalty` | 1.1 | Discourages stuck-on-one-frame artefacts. 1.0 disables it. |
|
||||||
|
| `max_new_tokens` / `max_tokens` | 1200 | ≈ 9–10 s of audio. Raise for longer utterances. |
|
||||||
|
| `seed` | `None` | Pass an int for reproducible output across runs. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Token format
|
||||||
|
|
||||||
|
The tokenizer is Llama-3's, with Orpheus's audio-codebook special tokens
|
||||||
|
laid out above the standard text vocabulary:
|
||||||
|
|
||||||
|
| Token | ID | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `<\|begin_of_text\|>` | 128000 | Llama-3 BOS (auto-prepended by tokenizer) |
|
||||||
|
| `<\|end_of_text\|>` | 128009 | end of human turn (text portion) |
|
||||||
|
| `START_OF_SPEECH` | 128257 | model emits this just before audio codes |
|
||||||
|
| `END_OF_SPEECH` | 128258 | model emits this when it finishes — used as `eos_token_id` / `stop_token_ids` |
|
||||||
|
| `START_OF_HUMAN` | 128259 | wrap the text prompt |
|
||||||
|
| `END_OF_HUMAN` | 128260 | wrap the text prompt |
|
||||||
|
| `START_OF_AI` | 128261 | model emits this to begin its response |
|
||||||
|
| `END_OF_AI` | 128262 | model emits this when fully done |
|
||||||
|
| `PAD_TOKEN` | 128263 | left-padding for batched generation |
|
||||||
|
| audio codebook | 128266 + N·4096 | SNAC codes, N ∈ {0..6} for 7-frame layout |
|
||||||
|
|
||||||
|
**Training prompt structure** (and what the model expects at inference):
|
||||||
|
|
||||||
|
```
|
||||||
|
[SOH] + tokenize("salt_<lang>_<NNNN>: <text>") + [EOT] + [EOH]
|
||||||
|
↳ model autoregressively emits:
|
||||||
|
[SOA] + [SOS] + audio_codes... + [EOS] + [EOA]
|
||||||
|
```
|
||||||
|
|
||||||
|
To recover audio: find the **last** `START_OF_SPEECH` (128257) in the
|
||||||
|
output, take everything after it, drop any token outside the audio
|
||||||
|
codebook range, group into 7-token frames, undo the per-position offsets,
|
||||||
|
and feed the three layers to `SNAC.decode`. Both inference snippets above
|
||||||
|
implement this end-to-end.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Training details
|
||||||
|
|
||||||
|
| Setting | Value |
|
||||||
|
|---|---|
|
||||||
|
| Base model | `unsloth/orpheus-3b-0.1-pretrained` (raw pretrained, not the `-ft` voice-actor variant) |
|
||||||
|
| Adapter | LoRA r=64, α=64, dropout=0, bias=none |
|
||||||
|
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
|
||||||
|
| Optimizer | `adamw_8bit`, weight decay 0.001 |
|
||||||
|
| LR schedule | linear, lr=2e-4, warmup steps=5 |
|
||||||
|
| Per-device batch size | 1 (with `gradient_accumulation_steps=4`, effective batch = 4) |
|
||||||
|
| Epochs | 3 |
|
||||||
|
| `max_seq_length` | 4096 |
|
||||||
|
| `save_total_limit` | 2 |
|
||||||
|
| Precision | bfloat16 weights, 16-bit LoRA |
|
||||||
|
| Seed | 3407 |
|
||||||
|
| Hardware | single NVIDIA RTX 4090 (24 GB) |
|
||||||
|
| Gradient checkpointing | Unsloth's optimised variant |
|
||||||
|
| Final save | LoRA merged into 16-bit weights via `save_pretrained_merged(save_method="merged_16bit")` |
|
||||||
|
|
||||||
|
The pretrained variant of Orpheus was chosen over the `-ft` voice-actor
|
||||||
|
variant because that variant has a strong English-voice-actor prior that
|
||||||
|
fights low-resource-language fine-tuning.
|
||||||
|
|
||||||
|
### Data prep summary
|
||||||
|
|
||||||
|
1. Load all 20 configs of `Sunbird/tts` (`get_dataset_config_names`)
|
||||||
|
and `concatenate_datasets` their `train` and `test` splits into one
|
||||||
|
training set and one held-out evaluation set. **No** speaker filter.
|
||||||
|
2. Tag each row with `source = example["speaker_id"]` (per-row, not
|
||||||
|
constant) — the model learns the multi-speaker prompt format
|
||||||
|
`f"{speaker_id}: {text}"` across every speaker it sees.
|
||||||
|
3. Cast `audio` to 24 kHz via `Audio(sampling_rate=24000)`.
|
||||||
|
4. Drop rows whose tokenised text alone exceeds `max_seq_length` —
|
||||||
|
saves expensive SNAC encoding on rows that would be filtered out
|
||||||
|
downstream.
|
||||||
|
5. Encode each remaining audio clip with `hubertsiuzdak/snac_24khz` →
|
||||||
|
7 codes per frame, flattened with per-layer offsets
|
||||||
|
`(+128266, +4096, +2·4096, …)`.
|
||||||
|
6. Filter out rows with empty/None codes; drop consecutive duplicate
|
||||||
|
frames.
|
||||||
|
7. Build `input_ids = [SOH] + text_ids + [EOT] + [EOH] + [SOA] + [SOS] + audio_codes + [EOS] + [EOA]`.
|
||||||
|
8. Drop rows whose total tokenised length exceeds `max_seq_length`
|
||||||
|
(safety net for rows where text fits but text + audio together
|
||||||
|
overflow the budget).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Evaluation
|
||||||
|
|
||||||
|
Quality was evaluated qualitatively on a diverse held-out test sample:
|
||||||
|
during training, up to 10 utterances are pulled from
|
||||||
|
`ds_test.shuffle(seed=42)` covering as many distinct speaker_ids as
|
||||||
|
possible. Generated audio is saved next to the ground-truth recording
|
||||||
|
under `inference_samples/sample_<idx>_<speaker_id>.wav` so each language
|
||||||
|
/ voice combination can be auditioned individually.
|
||||||
|
|
||||||
|
We did **not** run automated metrics (WER on a downstream STT, MOS
|
||||||
|
prediction, language-confusion eval, etc.) for this release. Numbers
|
||||||
|
will be added if/when those become part of the evaluation pipeline.
|
||||||
|
|
||||||
|
**Important caveat — quality varies by language.** The training corpus
|
||||||
|
is unbalanced across the 20 configs; languages with more speaker hours
|
||||||
|
in `Sunbird/tts` get more training signal and produce more natural
|
||||||
|
speech. Audition the per-language samples before relying on a specific
|
||||||
|
voice for production traffic.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Intended uses & out-of-scope
|
||||||
|
|
||||||
|
**Intended:**
|
||||||
|
|
||||||
|
- Multilingual voice synthesis for accessibility, language learning,
|
||||||
|
human–computer interaction, audio content creation, and downstream
|
||||||
|
speech research on the 20 covered languages.
|
||||||
|
- A reference checkpoint for the Sunbird/tts → Orpheus-3B multilingual
|
||||||
|
fine-tuning pipeline; reproducible training recipe in
|
||||||
|
[`Orpheus_3B_Sunbird_Multilingual.ipynb`](https://github.com/SunbirdAI/Qwen3-TTS/blob/main/orpheus-3B/Orpheus_3B_Sunbird_Multilingual.ipynb).
|
||||||
|
|
||||||
|
**Out of scope:**
|
||||||
|
|
||||||
|
- **Voice impersonation / deception.** The model imitates the timbres
|
||||||
|
of consenting Sunbird voice donors. Do not use the generated audio
|
||||||
|
to impersonate identifiable real persons or to produce content that
|
||||||
|
could mislead listeners about who is speaking.
|
||||||
|
- **High-stakes decisions.** Generated speech may contain pronunciation
|
||||||
|
errors, prosodic artefacts, or hallucinated phrases — do not deploy
|
||||||
|
in safety-critical contexts (medical, legal, emergency) without
|
||||||
|
human review.
|
||||||
|
- **Languages outside the 20 configs.** The model has no signal for
|
||||||
|
languages not present in `Sunbird/tts`; sending German text to any
|
||||||
|
speaker will produce garbled output, not "German with a Luganda
|
||||||
|
accent".
|
||||||
|
- **Code-switching.** Each speaker_id was recorded in a single language;
|
||||||
|
the model has not seen mixed-language utterances and will likely
|
||||||
|
produce phonetic artefacts at language boundaries within one prompt.
|
||||||
|
- **Cross-language voice transfer.** Sending Acholi text to
|
||||||
|
`salt_lug_0001` (a Luganda speaker_id) is undefined behaviour. The
|
||||||
|
model has no language-conditioning input separate from the speaker
|
||||||
|
tag, so language identity travels via the speaker_id. Use a speaker
|
||||||
|
whose `salt_<lang>_NNNN` prefix matches the language of your text.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Limitations & risks
|
||||||
|
|
||||||
|
- **Quality varies by language.** Per-language data volume in
|
||||||
|
`Sunbird/tts` is unbalanced. Languages with fewer hours produce
|
||||||
|
noticeably less natural speech. Run the per-language test-split
|
||||||
|
audit (script below) before committing to a particular voice.
|
||||||
|
- **No language conditioning.** There is no `language` token; the
|
||||||
|
model relies entirely on the speaker_id to disambiguate. Mismatching
|
||||||
|
speaker_id and text language is undefined behaviour (see above).
|
||||||
|
- **Vocabulary coverage.** Limited to the lexicon present in each
|
||||||
|
config's training subset. Unfamiliar words, code-switching, and
|
||||||
|
out-of-distribution proper nouns may produce artefacts.
|
||||||
|
- **Long utterances.** The model was trained on utterances up to ~16 s
|
||||||
|
of audio (`max_seq_length=4096`). Generation may degrade or
|
||||||
|
truncate beyond ~10 s of speech.
|
||||||
|
- **Sampling variance.** With `do_sample=True`, identical prompts can
|
||||||
|
produce noticeably different deliveries between runs. Pass `seed=`
|
||||||
|
for reproducibility.
|
||||||
|
- **No emotion/style control.** Unlike the upstream `orpheus-3b-0.1-ft`,
|
||||||
|
this fine-tune was not exposed to in-text emotion tags
|
||||||
|
(`<laugh>`, `<sigh>`, …). Such tags will be tokenised as ordinary
|
||||||
|
text and produce no special prosodic effect.
|
||||||
|
- **Bias.** Inherits any biases present in the Sunbird/tts corpus and
|
||||||
|
in Llama-3's pretraining; we have not audited these systematically
|
||||||
|
per language.
|
||||||
|
|
||||||
|
### Quick per-language audit script
|
||||||
|
|
||||||
|
```python
|
||||||
|
from datasets import load_dataset, Audio, get_dataset_config_names
|
||||||
|
import soundfile as sf
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
CONFIGS = get_dataset_config_names("Sunbird/tts")
|
||||||
|
out_dir = Path("language_audit"); out_dir.mkdir(exist_ok=True)
|
||||||
|
|
||||||
|
for cfg in CONFIGS:
|
||||||
|
ds = load_dataset("Sunbird/tts", cfg, split="test")
|
||||||
|
ds = ds.cast_column("audio", Audio(sampling_rate=24000))
|
||||||
|
row = ds[0]
|
||||||
|
sid, text = row["speaker_id"], row["text"]
|
||||||
|
print(f"{cfg}: {sid} -> {text[:80]}")
|
||||||
|
wav = synthesize(text, speaker_id=sid, seed=0)
|
||||||
|
sf.write(out_dir / f"{cfg}_{sid}.wav", wav, 24000)
|
||||||
|
sf.write(out_dir / f"{cfg}_{sid}_groundtruth.wav",
|
||||||
|
row["audio"]["array"], 24000)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Hardware requirements
|
||||||
|
|
||||||
|
| Mode | Min VRAM | Recommended |
|
||||||
|
|---|---|---|
|
||||||
|
| `transformers` + Unsloth, fp16 | 8 GB (with `load_in_4bit=True`) | 16 GB |
|
||||||
|
| `transformers` + Unsloth, bf16 | 14 GB | 24 GB |
|
||||||
|
| vLLM, bf16, `max_model_len=4096` | 14 GB | 24 GB |
|
||||||
|
|
||||||
|
Audio decoding via SNAC runs on CPU and adds ~50–150 ms per utterance.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## License & attribution
|
||||||
|
|
||||||
|
This fine-tune is released under **Apache-2.0**, matching the upstream
|
||||||
|
[`unsloth/orpheus-3b-0.1-pretrained`](https://huggingface.co/unsloth/orpheus-3b-0.1-pretrained)
|
||||||
|
license. It transitively inherits obligations from:
|
||||||
|
|
||||||
|
- The [Orpheus-TTS](https://github.com/canopyai/Orpheus-TTS) project (CanopyAI).
|
||||||
|
- The [Llama-3](https://llama.meta.com/llama3/) base architecture and weights — Meta Llama 3 Community License.
|
||||||
|
- The [SNAC](https://github.com/hubertsiuzdak/snac) audio codec (Hubert Siuzdak, MIT).
|
||||||
|
- The [`Sunbird/tts`](https://huggingface.co/datasets/Sunbird/tts) dataset and the SALT voice donors who contributed recordings.
|
||||||
|
|
||||||
|
If you redistribute the merged weights, please carry these attributions
|
||||||
|
forward.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Citation
|
||||||
|
|
||||||
|
If you use this model in your work, please cite both the dataset and the
|
||||||
|
fine-tuning project:
|
||||||
|
|
||||||
|
```bibtex
|
||||||
|
@misc{sunbird_orpheus3b_multilingual_2026,
|
||||||
|
title = {Orpheus-3B Sunbird Multilingual TTS},
|
||||||
|
author = {Sunbird AI},
|
||||||
|
year = {2026},
|
||||||
|
howpublished = {\url{https://huggingface.co/sunbird/orpheus-3b-tts-multilingual}},
|
||||||
|
}
|
||||||
|
|
||||||
|
@misc{sunbird_tts_dataset,
|
||||||
|
title = {Sunbird Speech Dataset},
|
||||||
|
author = {Sunbird AI},
|
||||||
|
howpublished = {\url{https://huggingface.co/datasets/Sunbird/tts}},
|
||||||
|
}
|
||||||
|
|
||||||
|
@misc{orpheus_tts_2025,
|
||||||
|
title = {Orpheus-TTS},
|
||||||
|
author = {Canopy Labs},
|
||||||
|
year = {2025},
|
||||||
|
howpublished = {\url{https://github.com/canopyai/Orpheus-TTS}},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Single-speaker variant
|
||||||
|
|
||||||
|
If you only need one specific voice and want a smaller, more focused
|
||||||
|
checkpoint, see
|
||||||
|
[`sunbird/orpheus-3b-tts-salt-lug-0001`](https://huggingface.co/sunbird/orpheus-3b-tts-salt-lug-0001)
|
||||||
|
— same recipe, scoped to a single Luganda speaker.
|
||||||
93
chat_template.jinja
Normal file
93
chat_template.jinja
Normal file
@@ -0,0 +1,93 @@
|
|||||||
|
{{- bos_token }}
|
||||||
|
{%- if custom_tools is defined %}
|
||||||
|
{%- set tools = custom_tools %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if not tools_in_user_message is defined %}
|
||||||
|
{%- set tools_in_user_message = true %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if not date_string is defined %}
|
||||||
|
{%- if strftime_now is defined %}
|
||||||
|
{%- set date_string = strftime_now("%d %b %Y") %}
|
||||||
|
{%- else %}
|
||||||
|
{%- set date_string = "26 Jul 2024" %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if not tools is defined %}
|
||||||
|
{%- set tools = none %}
|
||||||
|
{%- endif %}
|
||||||
|
|
||||||
|
{#- This block extracts the system message, so we can slot it into the right place. #}
|
||||||
|
{%- if messages[0]['role'] == 'system' %}
|
||||||
|
{%- set system_message = messages[0]['content']|trim %}
|
||||||
|
{%- set messages = messages[1:] %}
|
||||||
|
{%- else %}
|
||||||
|
{%- set system_message = "" %}
|
||||||
|
{%- endif %}
|
||||||
|
|
||||||
|
{#- System message #}
|
||||||
|
{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
|
||||||
|
{%- if tools is not none %}
|
||||||
|
{{- "Environment: ipython\n" }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- "Cutting Knowledge Date: December 2023\n" }}
|
||||||
|
{{- "Today Date: " + date_string + "\n\n" }}
|
||||||
|
{%- if tools is not none and not tools_in_user_message %}
|
||||||
|
{{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
|
||||||
|
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
|
||||||
|
{{- "Do not use variables.\n\n" }}
|
||||||
|
{%- for t in tools %}
|
||||||
|
{{- t | tojson(indent=4) }}
|
||||||
|
{{- "\n\n" }}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- system_message }}
|
||||||
|
{{- "<|eot_id|>" }}
|
||||||
|
|
||||||
|
{#- Custom tools are passed in a user message with some extra guidance #}
|
||||||
|
{%- if tools_in_user_message and not tools is none %}
|
||||||
|
{#- Extract the first user message so we can plug it in here #}
|
||||||
|
{%- if messages | length != 0 %}
|
||||||
|
{%- set first_user_message = messages[0]['content']|trim %}
|
||||||
|
{%- set messages = messages[1:] %}
|
||||||
|
{%- else %}
|
||||||
|
{{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
|
||||||
|
{{- "Given the following functions, please respond with a JSON for a function call " }}
|
||||||
|
{{- "with its proper arguments that best answers the given prompt.\n\n" }}
|
||||||
|
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
|
||||||
|
{{- "Do not use variables.\n\n" }}
|
||||||
|
{%- for t in tools %}
|
||||||
|
{{- t | tojson(indent=4) }}
|
||||||
|
{{- "\n\n" }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- first_user_message + "<|eot_id|>"}}
|
||||||
|
{%- endif %}
|
||||||
|
|
||||||
|
{%- for message in messages %}
|
||||||
|
{%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
|
||||||
|
{{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
|
||||||
|
{%- elif 'tool_calls' in message %}
|
||||||
|
{%- if not message.tool_calls|length == 1 %}
|
||||||
|
{{- raise_exception("This model only supports single tool-calls at once!") }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- set tool_call = message.tool_calls[0].function %}
|
||||||
|
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
|
||||||
|
{{- '{"name": "' + tool_call.name + '", ' }}
|
||||||
|
{{- '"parameters": ' }}
|
||||||
|
{{- tool_call.arguments | tojson }}
|
||||||
|
{{- "}" }}
|
||||||
|
{{- "<|eot_id|>" }}
|
||||||
|
{%- elif message.role == "tool" or message.role == "ipython" %}
|
||||||
|
{{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
|
||||||
|
{%- if message.content is mapping or message.content is iterable %}
|
||||||
|
{{- message.content | tojson }}
|
||||||
|
{%- else %}
|
||||||
|
{{- message.content }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- "<|eot_id|>" }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- if add_generation_prompt %}
|
||||||
|
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
|
||||||
|
{%- endif %}
|
||||||
38
config.json
Normal file
38
config.json
Normal file
@@ -0,0 +1,38 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"LlamaForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 128000,
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"eos_token_id": 128009,
|
||||||
|
"head_dim": 128,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 3072,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 8192,
|
||||||
|
"max_position_embeddings": 131072,
|
||||||
|
"mlp_bias": false,
|
||||||
|
"model_type": "llama",
|
||||||
|
"num_attention_heads": 24,
|
||||||
|
"num_hidden_layers": 28,
|
||||||
|
"num_key_value_heads": 8,
|
||||||
|
"pad_token_id": 128004,
|
||||||
|
"pretraining_tp": 1,
|
||||||
|
"rms_norm_eps": 1e-05,
|
||||||
|
"rope_scaling": {
|
||||||
|
"factor": 32.0,
|
||||||
|
"high_freq_factor": 4.0,
|
||||||
|
"low_freq_factor": 1.0,
|
||||||
|
"original_max_position_embeddings": 8192,
|
||||||
|
"rope_type": "llama3"
|
||||||
|
},
|
||||||
|
"rope_theta": 500000.0,
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"unsloth_fixed": true,
|
||||||
|
"unsloth_optimized": true,
|
||||||
|
"unsloth_version": "2026.4.8",
|
||||||
|
"use_cache": true,
|
||||||
|
"vocab_size": 156939
|
||||||
|
}
|
||||||
3
model-00001-of-00002.safetensors
Normal file
3
model-00001-of-00002.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:9bdb7698468040694fa7577304b5c4d21e7b6229502e8f8b3bdf20a81673373b
|
||||||
|
size 4991031824
|
||||||
3
model-00002-of-00002.safetensors
Normal file
3
model-00002-of-00002.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:69b09255384b5ea8d5958b4e458b0209e6f37132568b4d1b4d473bb2176385fb
|
||||||
|
size 1610725592
|
||||||
261
model.safetensors.index.json
Normal file
261
model.safetensors.index.json
Normal file
@@ -0,0 +1,261 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"total_size": 6601728000
|
||||||
|
},
|
||||||
|
"weight_map": {
|
||||||
|
"model.embed_tokens.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.20.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.23.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.24.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.25.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.26.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.27.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
|
||||||
|
"model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||||
|
"model.norm.weight": "model-00002-of-00002.safetensors"
|
||||||
|
}
|
||||||
|
}
|
||||||
26
special_tokens_map.json
Normal file
26
special_tokens_map.json
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
{
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|audio|>"
|
||||||
|
],
|
||||||
|
"bos_token": {
|
||||||
|
"content": "<|begin_of_text|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"eos_token": {
|
||||||
|
"content": "<|eot_id|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<|finetune_right_pad_id|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:fc3fecb199b4170636dbfab986d25f628157268d37b861f9cadaca60b1353bce
|
||||||
|
size 22849547
|
||||||
231542
tokenizer_config.json
Normal file
231542
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user