初始化项目，由ModelHub XC社区提供模型

Model: patrickcmd/orpheus-3b-tts-multilingual Source: Original Platform
2026-08-02 00:43:22 +08:00
commit 8f536afbfb
10 changed files with 232663 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,658 @@
+---
+license: apache-2.0
+language:
+  - ach
+  - af
+  - en
+  - ee
+  - ff
+  - ha
+  - ig
+  - ki
+  - rw
+  - lgg
+  - ln
+  - lg
+  - luo
+  - nyn
+  - st
+  - sw
+  - teo
+  - tn
+  - xh
+  - yo
+library_name: transformers
+pipeline_tag: text-to-speech
+tags:
+  - text-to-speech
+  - tts
+  - orpheus
+  - multilingual
+  - multi-speaker
+  - african-languages
+  - low-resource-language
+  - sunbird
+  - snac
+  - unsloth
+  - llama
+datasets:
+  - Sunbird/tts
+base_model: unsloth/orpheus-3b-0.1-pretrained
+---
+
+# Orpheus-3B Sunbird Multilingual TTS
+
+A multilingual, multi-speaker text-to-speech model fine-tuned from
+[`unsloth/orpheus-3b-0.1-pretrained`](https://huggingface.co/unsloth/orpheus-3b-0.1-pretrained)
+on the **full** [`Sunbird/tts`](https://huggingface.co/datasets/Sunbird/tts)
+corpus — 20 language configurations and every speaker present in the
+dataset.
+
+The model accepts arbitrary text and emits 24 kHz mono speech via the
+[SNAC](https://huggingface.co/hubertsiuzdak/snac_24khz) audio codec.
+Voice selection happens at the prompt level: prepend the chosen
+`speaker_id` followed by `": "` to your text, and the model produces
+audio in that speaker's voice.
+
+## Quick links
+
+- **Base model:** [`unsloth/orpheus-3b-0.1-pretrained`](https://huggingface.co/unsloth/orpheus-3b-0.1-pretrained) (Llama-3 architecture)
+- **Audio codec:** [`hubertsiuzdak/snac_24khz`](https://huggingface.co/hubertsiuzdak/snac_24khz) (24 kHz, 7 codes per ~12 ms frame)
+- **Training dataset:** [`Sunbird/tts`](https://huggingface.co/datasets/Sunbird/tts) — all 20 configs, all speakers
+- **Training framework:** [Unsloth](https://github.com/unslothai/unsloth) + HuggingFace Trainer
+
+## Languages covered
+
+Speaker IDs encode both the source corpus (`salt_*`, `waxal_*`, `slr32_*`,
+`slr129_*`, `bateesa_*`) and the language. Languages marked with an em dash
+in the Speaker IDs column are present in the model's training mix but do
+not currently expose individual voice IDs in this checkpoint.
+
+| Config | Language | ISO 639-1 | Region | Speaker IDs |
+|---|---|---|---|---|
+| `ach` | Acholi | — | Uganda, South Sudan | `salt_ach_0001`<br>`waxal_ach_0001`<br>`waxal_ach_0005`<br>`waxal_ach_0006`<br>`waxal_ach_0008` |
+| `afr` | Afrikaans | af | South Africa, Namibia | `slr32_afr_0009` |
+| `eng` | English | en | (control language) | `salt_eng_0001`<br>`salt_eng_0002`<br>`salt_eng_0003` |
+| `ewe` | Ewe | ee | Ghana, Togo | `slr129_ewe_0001` |
+| `ful` | Fulah | ff | West Africa (Sahel) | `waxal_ful_0003`<br>`waxal_ful_0004`<br>`waxal_ful_0006` |
+| `hau` | Hausa | ha | Nigeria, Niger, Chad | `waxal_hau_0004`<br>`waxal_hau_0006`<br>`waxal_hau_0007`<br>`waxal_hau_0008` |
+| `ibo` | Igbo | ig | Nigeria | `waxal_ibo_0003`<br>`waxal_ibo_0005`<br>`waxal_ibo_0008` |
+| `kik` | Kikuyu | ki | Kenya | `waxal_kik_0003`<br>`waxal_kik_0004` |
+| `kin` | Kinyarwanda | rw | Rwanda | `bateesa_kin_0001` |
+| `lgg` | Lugbara | — | Uganda, DRC | — |
+| `lin` | Lingala | ln | DRC, Republic of Congo | `slr129_lin_0001` |
+| `lug` | Luganda | lg | Uganda | `salt_lug_0001`<br>`waxal_lug_0002`<br>`waxal_lug_0003`<br>`waxal_lug_0004`<br>`waxal_lug_0005`<br>`waxal_lug_0006`<br>`waxal_lug_0007`<br>`waxal_lug_0008` |
+| `luo` | Luo (Dholuo) | — | Kenya, Tanzania | `waxal_luo_0001`<br>`waxal_luo_0002`<br>`waxal_luo_0003`<br>`waxal_luo_0004` |
+| `nyn` | Runyankole | — | Uganda | `salt_nyn_0001`<br>`waxal_nyn_0003`<br>`waxal_nyn_0004`<br>`waxal_nyn_0007`<br>`waxal_nyn_0008` |
+| `sot` | Sesotho | st | Lesotho, South Africa | — |
+| `swa` | Swahili | sw | East Africa | `waxal_swa_0006`<br>`waxal_swa_0007` |
+| `teo` | Ateso | — | Uganda, Kenya | `salt_teo_0001` |
+| `tsn` | Setswana | tn | Botswana, South Africa | — |
+| `xho` | Xhosa | xh | South Africa | `slr32_xho_0012` |
+| `yor` | Yoruba | yo | Nigeria, Benin | `waxal_yor_0002`<br>`waxal_yor_0006`<br>`waxal_yor_0008` |
+
+Per-language quality scales with the amount of training data Sunbird
+collected for that language; some configs have many more speaker hours
+than others. **Audition the test split** for each language before relying
+on a particular speaker — see the discovery snippet below.
+
+## TL;DR
+
+```python
+# After installing the dependencies (see "Inference" below)
+wav = synthesize("Mwattu, oli otya?",  speaker_id="salt_lug_0001")  # Luganda
+wav = synthesize("Habari yako rafiki.", speaker_id="salt_swa_0001")  # Swahili
+wav = synthesize("Bawo ni, ọrẹ mi?",     speaker_id="salt_yor_0001")  # Yoruba
+```
+
+The model has no explicit "language" knob — the language identity
+travels via the speaker tag, since each `salt_<lang>_<NNNN>` voice was
+recorded in exactly one language.
+
+---
+
+## Discovering speaker IDs
+
+The exact speaker_ids in each config can be enumerated from the dataset:
+
+```python
+from collections import defaultdict
+from datasets import load_dataset, get_dataset_config_names
+
+CONFIGS = get_dataset_config_names("Sunbird/tts")  # the 20 languages
+
+speakers_by_lang = defaultdict(set)
+for cfg in CONFIGS:
+    ds = load_dataset("Sunbird/tts", cfg, split="train")
+    for sid in ds["speaker_id"]:
+        speakers_by_lang[cfg].add(sid)
+
+for cfg, sids in sorted(speakers_by_lang.items()):
+    print(f"{cfg}: {len(sids)} speaker(s) — {sorted(sids)[:3]}{'...' if len(sids) > 3 else ''}")
+```
+
+Speaker IDs follow the pattern `salt_<lang>_<NNNN>` (e.g.,
+`salt_lug_0001`, `salt_ach_0007`). Pass any one of them as
+`speaker_id` to either inference function below.
+
+---
+
+## Inference
+
+The model wraps every prompt in a multi-speaker tagged format:
+
+```
+[SOH] + tokenize("<speaker_id>: <your text>") + [EOT, EOH]
+```
+
+and the model autoregressively emits Llama-3 special tokens followed by
+SNAC audio codes that decode to a 24 kHz waveform. Two reference
+implementations follow.
+
+### Option A — `transformers` + `unsloth` (single request)
+
+Best for development, notebook-driven iteration, and small batch sizes.
+
+**Install:**
+
+```bash
+pip install unsloth snac soundfile torchcodec "datasets>=3.4.1,<4.0.0"
+```
+
+**Run:**
+
+```python
+import os
+import numpy as np
+import torch
+import soundfile as sf
+from unsloth import FastLanguageModel
+from snac import SNAC
+
+MODEL_ID = "sunbird/orpheus-3b-tts-multilingual"
+
+# Special tokens — must match the training format
+END_OF_TEXT     = 128009
+START_OF_SPEECH = 128257
+END_OF_SPEECH   = 128258
+START_OF_HUMAN  = 128259
+END_OF_HUMAN    = 128260
+PAD_TOKEN       = 128263
+AUDIO_TOKEN_LO  = 128266
+AUDIO_TOKEN_HI  = 128266 + 7 * 4096   # exclusive
+
+# 1) Load the LM (LoRA already merged into 16-bit weights at training time)
+model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name     = MODEL_ID,
+    max_seq_length = 4096,
+    dtype          = None,         # auto bf16 / fp16
+    load_in_4bit   = False,        # set True to halve VRAM at slight quality cost
+    token          = os.environ.get("HF_TOKEN"),
+)
+FastLanguageModel.for_inference(model)
+
+# 2) Load SNAC decoder (CPU is fine — frees GPU for the LM)
+snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").to("cpu")
+
+
+def _redistribute_codes(code_list: list[int]) -> torch.Tensor:
+    layer_1, layer_2, layer_3 = [], [], []
+    for i in range(len(code_list) // 7):
+        layer_1.append(code_list[7*i])
+        layer_2.append(code_list[7*i + 1] - 4096)
+        layer_3.append(code_list[7*i + 2] - 2*4096)
+        layer_3.append(code_list[7*i + 3] - 3*4096)
+        layer_2.append(code_list[7*i + 4] - 4*4096)
+        layer_3.append(code_list[7*i + 5] - 5*4096)
+        layer_3.append(code_list[7*i + 6] - 6*4096)
+    if not layer_1:
+        return torch.zeros(1, 1, 12000)   # ~0.5s silence fallback
+    clamp = lambda vals: [max(0, min(4095, v)) for v in vals]
+    codes = [torch.tensor(clamp(layer_1)).unsqueeze(0),
+             torch.tensor(clamp(layer_2)).unsqueeze(0),
+             torch.tensor(clamp(layer_3)).unsqueeze(0)]
+    return snac_model.decode(codes)
+
+
+def synthesize(text: str, speaker_id: str,
+               *, max_new_tokens: int = 1200,
+               temperature: float = 0.6, top_p: float = 0.95,
+               repetition_penalty: float = 1.1,
+               seed: int | None = None) -> np.ndarray:
+    """Synthesize speech for `text` in the voice of `speaker_id`.
+
+    `speaker_id` must be one of the speakers seen during training,
+    e.g. "salt_lug_0001" (Luganda) or "salt_swa_0003" (Swahili).
+    """
+    if seed is not None:
+        torch.manual_seed(seed)
+
+    tagged   = f"{speaker_id}: {text}"
+    text_ids = tokenizer(tagged, return_tensors="pt").input_ids
+    soh = torch.tensor([[START_OF_HUMAN]], dtype=torch.int64)
+    end = torch.tensor([[END_OF_TEXT, END_OF_HUMAN]], dtype=torch.int64)
+    input_ids      = torch.cat([soh, text_ids, end], dim=1).to("cuda")
+    attention_mask = torch.ones_like(input_ids)
+
+    generated = model.generate(
+        input_ids = input_ids, attention_mask = attention_mask,
+        max_new_tokens = max_new_tokens,
+        do_sample = True,
+        temperature = temperature, top_p = top_p,
+        repetition_penalty = repetition_penalty,
+        eos_token_id = END_OF_SPEECH, use_cache = True,
+    )
+
+    # Crop on last SOS, filter to audio token range, redistribute, decode
+    sos_indices = (generated == START_OF_SPEECH).nonzero(as_tuple=True)
+    cropped = generated[:, sos_indices[1][-1].item() + 1:] if len(sos_indices[1]) > 0 else generated
+    row = cropped[0]
+    audio_only = row[(row >= AUDIO_TOKEN_LO) & (row < AUDIO_TOKEN_HI)]
+    n = (audio_only.size(0) // 7) * 7
+    code_list = [t.item() - AUDIO_TOKEN_LO for t in audio_only[:n]]
+    waveform  = _redistribute_codes(code_list)
+    return waveform.detach().squeeze().to("cpu").numpy().astype(np.float32)
+
+
+# 3) Use it — pick a speaker per language
+wav = synthesize("Mwattu, Mukama yeebazibwe.", speaker_id="salt_lug_0001", seed=42)
+sf.write("luganda.wav", wav, 24000)
+
+wav = synthesize("Habari yako rafiki.", speaker_id="salt_swa_0001", seed=42)
+sf.write("swahili.wav", wav, 24000)
+```
+
+### Option B — `vllm` (high throughput, batched, deployment)
+
+Best for serving traffic. PagedAttention + continuous batching gives
+roughly **5–10× faster** single-request latency and **10–100× higher**
+throughput on batched requests vs. the `transformers` path. Multi-speaker
+batching (different `speaker_id`s in one call) gets the full benefit.
+
+> **Important:** vLLM ships its own torch/transformers and conflicts
+> with Unsloth's pinned versions. Use a fresh Python environment for
+> vLLM serving — do not install on top of an Unsloth env.
+
+**Install:**
+
+```bash
+pip install vllm snac soundfile torchcodec "datasets>=3.4.1,<4.0.0"
+```
+
+**Run:**
+
+```python
+import os
+import numpy as np
+import torch
+import soundfile as sf
+from snac import SNAC
+from transformers import AutoTokenizer
+from vllm import LLM, SamplingParams
+
+MODEL_ID = "sunbird/orpheus-3b-tts-multilingual"
+
+END_OF_TEXT     = 128009
+START_OF_SPEECH = 128257
+END_OF_SPEECH   = 128258
+START_OF_HUMAN  = 128259
+END_OF_HUMAN    = 128260
+AUDIO_TOKEN_LO  = 128266
+AUDIO_TOKEN_HI  = 128266 + 7 * 4096
+
+# 1) Load
+llm = LLM(
+    model = MODEL_ID,
+    dtype = "bfloat16",
+    max_model_len = 4096,
+    gpu_memory_utilization = 0.85,
+)
+tokenizer  = AutoTokenizer.from_pretrained(MODEL_ID, token=os.environ.get("HF_TOKEN"))
+snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").to("cpu")
+
+
+def _build_prompt_token_ids(text: str, speaker_id: str) -> list[int]:
+    tagged = f"{speaker_id}: {text}"
+    text_ids = tokenizer.encode(tagged, add_special_tokens=True)
+    return [START_OF_HUMAN] + text_ids + [END_OF_TEXT, END_OF_HUMAN]
+
+
+def _codes_to_waveform(generated_token_ids: list[int]) -> np.ndarray:
+    ids = torch.tensor(generated_token_ids, dtype=torch.int64)
+    sos_pos = (ids == START_OF_SPEECH).nonzero(as_tuple=True)[0]
+    if len(sos_pos) > 0:
+        ids = ids[sos_pos[-1].item() + 1:]
+    audio = ids[(ids >= AUDIO_TOKEN_LO) & (ids < AUDIO_TOKEN_HI)]
+    n = (audio.size(0) // 7) * 7
+    cl = [t.item() - AUDIO_TOKEN_LO for t in audio[:n]]
+    l1, l2, l3 = [], [], []
+    for i in range(len(cl) // 7):
+        l1.append(cl[7*i])
+        l2.append(cl[7*i+1] - 4096); l3.append(cl[7*i+2] - 2*4096)
+        l3.append(cl[7*i+3] - 3*4096); l2.append(cl[7*i+4] - 4*4096)
+        l3.append(cl[7*i+5] - 5*4096); l3.append(cl[7*i+6] - 6*4096)
+    if not l1:
+        return np.zeros(12000, dtype=np.float32)
+    cb = lambda v: [max(0, min(4095, x)) for x in v]
+    codes = [torch.tensor(cb(l1)).unsqueeze(0),
+             torch.tensor(cb(l2)).unsqueeze(0),
+             torch.tensor(cb(l3)).unsqueeze(0)]
+    return snac_model.decode(codes).detach().squeeze().cpu().numpy().astype(np.float32)
+
+
+def synthesize(text: str, speaker_id: str,
+               *, max_tokens: int = 1200,
+               temperature: float = 0.6, top_p: float = 0.95,
+               repetition_penalty: float = 1.1,
+               seed: int | None = None) -> np.ndarray:
+    sp = SamplingParams(
+        temperature = temperature, top_p = top_p,
+        repetition_penalty = repetition_penalty,
+        max_tokens = max_tokens,
+        stop_token_ids = [END_OF_SPEECH],
+        skip_special_tokens = False,
+        seed = seed,
+    )
+    pids = _build_prompt_token_ids(text, speaker_id)
+    out  = llm.generate([{"prompt_token_ids": pids}], sp)
+    return _codes_to_waveform(list(out[0].outputs[0].token_ids))
+
+
+def synthesize_batch(items: list[dict], **kwargs) -> list[np.ndarray]:
+    """items: list of {"text": str, "speaker_id": str} — different speakers
+    can be mixed in one batch."""
+    sp = SamplingParams(
+        temperature = kwargs.get("temperature", 0.6),
+        top_p = kwargs.get("top_p", 0.95),
+        repetition_penalty = kwargs.get("repetition_penalty", 1.1),
+        max_tokens = kwargs.get("max_tokens", 1200),
+        stop_token_ids = [END_OF_SPEECH],
+        skip_special_tokens = False,
+        seed = kwargs.get("seed"),
+    )
+    prompts = [{"prompt_token_ids": _build_prompt_token_ids(it["text"], it["speaker_id"])}
+               for it in items]
+    outputs = llm.generate(prompts, sp)
+    return [_codes_to_waveform(list(o.outputs[0].token_ids)) for o in outputs]
+
+
+# 2) Single — pick a speaker per language
+wav = synthesize("Mwattu, oli otya?", speaker_id="salt_lug_0001", seed=42)
+sf.write("luganda.wav", wav, 24000)
+
+# 3) Batched — different languages and speakers in one GPU pass
+items = [
+    {"text": "Mwattu, oli otya?",        "speaker_id": "salt_lug_0001"},
+    {"text": "Habari yako rafiki.",      "speaker_id": "waxal_swa_0006"},
+    {"text": "Bawo ni, ọrẹ mi?",          "speaker_id": "waxal_yor_0002"},
+    {"text": "Sannu, ina kwana?",        "speaker_id": "waxal_hau_0004"},
+    {"text": "Goeie môre, hoe gaan dit?", "speaker_id": "slr32_afr_0009"},
+]
+wavs = synthesize_batch(items, seed=123)
+for i, (it, w) in enumerate(zip(items, wavs)):
+    sf.write(f"batch_{i:02d}_{it['speaker_id']}.wav", w, 24000)
+```
+
+### Generation parameters
+
+| Param | Default | What it does |
+|---|---|---|
+| `temperature` | 0.6 | Lower = more deterministic, slightly flatter prosody. |
+| `top_p` | 0.95 | Nucleus sampling. Don't drop below 0.9 — produces robotic audio. |
+| `repetition_penalty` | 1.1 | Discourages stuck-on-one-frame artefacts. 1.0 disables it. |
+| `max_new_tokens` / `max_tokens` | 1200 | ≈ 9–10 s of audio. Raise for longer utterances. |
+| `seed` | `None` | Pass an int for reproducible output across runs. |
+
+---
+
+## Token format
+
+The tokenizer is Llama-3's, with Orpheus's audio-codebook special tokens
+laid out above the standard text vocabulary:
+
+| Token | ID | Purpose |
+|---|---|---|
+| `<\|begin_of_text\|>` | 128000 | Llama-3 BOS (auto-prepended by tokenizer) |
+| `<\|end_of_text\|>` | 128009 | end of human turn (text portion) |
+| `START_OF_SPEECH` | 128257 | model emits this just before audio codes |
+| `END_OF_SPEECH` | 128258 | model emits this when it finishes — used as `eos_token_id` / `stop_token_ids` |
+| `START_OF_HUMAN` | 128259 | wrap the text prompt |
+| `END_OF_HUMAN` | 128260 | wrap the text prompt |
+| `START_OF_AI` | 128261 | model emits this to begin its response |
+| `END_OF_AI` | 128262 | model emits this when fully done |
+| `PAD_TOKEN` | 128263 | left-padding for batched generation |
+| audio codebook | 128266 + N·4096 | SNAC codes, N ∈ {0..6} for 7-frame layout |
+
+**Training prompt structure** (and what the model expects at inference):
+
+```
+[SOH] + tokenize("salt_<lang>_<NNNN>: <text>") + [EOT] + [EOH]
+↳ model autoregressively emits:
+[SOA] + [SOS] + audio_codes... + [EOS] + [EOA]
+```
+
+To recover audio: find the **last** `START_OF_SPEECH` (128257) in the
+output, take everything after it, drop any token outside the audio
+codebook range, group into 7-token frames, undo the per-position offsets,
+and feed the three layers to `SNAC.decode`. Both inference snippets above
+implement this end-to-end.
+
+---
+
+## Training details
+
+| Setting | Value |
+|---|---|
+| Base model | `unsloth/orpheus-3b-0.1-pretrained` (raw pretrained, not the `-ft` voice-actor variant) |
+| Adapter | LoRA r=64, α=64, dropout=0, bias=none |
+| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
+| Optimizer | `adamw_8bit`, weight decay 0.001 |
+| LR schedule | linear, lr=2e-4, warmup steps=5 |
+| Per-device batch size | 1 (with `gradient_accumulation_steps=4`, effective batch = 4) |
+| Epochs | 3 |
+| `max_seq_length` | 4096 |
+| `save_total_limit` | 2 |
+| Precision | bfloat16 weights, 16-bit LoRA |
+| Seed | 3407 |
+| Hardware | single NVIDIA RTX 4090 (24 GB) |
+| Gradient checkpointing | Unsloth's optimised variant |
+| Final save | LoRA merged into 16-bit weights via `save_pretrained_merged(save_method="merged_16bit")` |
+
+The pretrained variant of Orpheus was chosen over the `-ft` voice-actor
+variant because that variant has a strong English-voice-actor prior that
+fights low-resource-language fine-tuning.
+
+### Data prep summary
+
+1. Load all 20 configs of `Sunbird/tts` (`get_dataset_config_names`)
+   and `concatenate_datasets` their `train` and `test` splits into one
+   training set and one held-out evaluation set. **No** speaker filter.
+2. Tag each row with `source = example["speaker_id"]` (per-row, not
+   constant) — the model learns the multi-speaker prompt format
+   `f"{speaker_id}: {text}"` across every speaker it sees.
+3. Cast `audio` to 24 kHz via `Audio(sampling_rate=24000)`.
+4. Drop rows whose tokenised text alone exceeds `max_seq_length` —
+   saves expensive SNAC encoding on rows that would be filtered out
+   downstream.
+5. Encode each remaining audio clip with `hubertsiuzdak/snac_24khz` →
+   7 codes per frame, flattened with per-layer offsets
+   `(+128266, +4096, +2·4096, …)`.
+6. Filter out rows with empty/None codes; drop consecutive duplicate
+   frames.
+7. Build `input_ids = [SOH] + text_ids + [EOT] + [EOH] + [SOA] + [SOS] + audio_codes + [EOS] + [EOA]`.
+8. Drop rows whose total tokenised length exceeds `max_seq_length`
+   (safety net for rows where text fits but text + audio together
+   overflow the budget).
+
+---
+
+## Evaluation
+
+Quality was evaluated qualitatively on a diverse held-out test sample:
+during training, up to 10 utterances are pulled from
+`ds_test.shuffle(seed=42)` covering as many distinct speaker_ids as
+possible. Generated audio is saved next to the ground-truth recording
+under `inference_samples/sample_<idx>_<speaker_id>.wav` so each language
+/ voice combination can be auditioned individually.
+
+We did **not** run automated metrics (WER on a downstream STT, MOS
+prediction, language-confusion eval, etc.) for this release. Numbers
+will be added if/when those become part of the evaluation pipeline.
+
+**Important caveat — quality varies by language.** The training corpus
+is unbalanced across the 20 configs; languages with more speaker hours
+in `Sunbird/tts` get more training signal and produce more natural
+speech. Audition the per-language samples before relying on a specific
+voice for production traffic.
+
+---
+
+## Intended uses & out-of-scope
+
+**Intended:**
+
+- Multilingual voice synthesis for accessibility, language learning,
+  human–computer interaction, audio content creation, and downstream
+  speech research on the 20 covered languages.
+- A reference checkpoint for the Sunbird/tts → Orpheus-3B multilingual
+  fine-tuning pipeline; reproducible training recipe in
+  [`Orpheus_3B_Sunbird_Multilingual.ipynb`](https://github.com/SunbirdAI/Qwen3-TTS/blob/main/orpheus-3B/Orpheus_3B_Sunbird_Multilingual.ipynb).
+
+**Out of scope:**
+
+- **Voice impersonation / deception.** The model imitates the timbres
+  of consenting Sunbird voice donors. Do not use the generated audio
+  to impersonate identifiable real persons or to produce content that
+  could mislead listeners about who is speaking.
+- **High-stakes decisions.** Generated speech may contain pronunciation
+  errors, prosodic artefacts, or hallucinated phrases — do not deploy
+  in safety-critical contexts (medical, legal, emergency) without
+  human review.
+- **Languages outside the 20 configs.** The model has no signal for
+  languages not present in `Sunbird/tts`; sending German text to any
+  speaker will produce garbled output, not "German with a Luganda
+  accent".
+- **Code-switching.** Each speaker_id was recorded in a single language;
+  the model has not seen mixed-language utterances and will likely
+  produce phonetic artefacts at language boundaries within one prompt.
+- **Cross-language voice transfer.** Sending Acholi text to
+  `salt_lug_0001` (a Luganda speaker_id) is undefined behaviour. The
+  model has no language-conditioning input separate from the speaker
+  tag, so language identity travels via the speaker_id. Use a speaker
+  whose `salt_<lang>_NNNN` prefix matches the language of your text.
+
+---
+
+## Limitations & risks
+
+- **Quality varies by language.** Per-language data volume in
+  `Sunbird/tts` is unbalanced. Languages with fewer hours produce
+  noticeably less natural speech. Run the per-language test-split
+  audit (script below) before committing to a particular voice.
+- **No language conditioning.** There is no `language` token; the
+  model relies entirely on the speaker_id to disambiguate. Mismatching
+  speaker_id and text language is undefined behaviour (see above).
+- **Vocabulary coverage.** Limited to the lexicon present in each
+  config's training subset. Unfamiliar words, code-switching, and
+  out-of-distribution proper nouns may produce artefacts.
+- **Long utterances.** The model was trained on utterances up to ~16 s
+  of audio (`max_seq_length=4096`). Generation may degrade or
+  truncate beyond ~10 s of speech.
+- **Sampling variance.** With `do_sample=True`, identical prompts can
+  produce noticeably different deliveries between runs. Pass `seed=`
+  for reproducibility.
+- **No emotion/style control.** Unlike the upstream `orpheus-3b-0.1-ft`,
+  this fine-tune was not exposed to in-text emotion tags
+  (`<laugh>`, `<sigh>`, …). Such tags will be tokenised as ordinary
+  text and produce no special prosodic effect.
+- **Bias.** Inherits any biases present in the Sunbird/tts corpus and
+  in Llama-3's pretraining; we have not audited these systematically
+  per language.
+
+### Quick per-language audit script
+
+```python
+from datasets import load_dataset, Audio, get_dataset_config_names
+import soundfile as sf
+from pathlib import Path
+
+CONFIGS = get_dataset_config_names("Sunbird/tts")
+out_dir = Path("language_audit"); out_dir.mkdir(exist_ok=True)
+
+for cfg in CONFIGS:
+    ds = load_dataset("Sunbird/tts", cfg, split="test")
+    ds = ds.cast_column("audio", Audio(sampling_rate=24000))
+    row = ds[0]
+    sid, text = row["speaker_id"], row["text"]
+    print(f"{cfg}: {sid} -> {text[:80]}")
+    wav = synthesize(text, speaker_id=sid, seed=0)
+    sf.write(out_dir / f"{cfg}_{sid}.wav", wav, 24000)
+    sf.write(out_dir / f"{cfg}_{sid}_groundtruth.wav",
+             row["audio"]["array"], 24000)
+```
+
+---
+
+## Hardware requirements
+
+| Mode | Min VRAM | Recommended |
+|---|---|---|
+| `transformers` + Unsloth, fp16 | 8 GB (with `load_in_4bit=True`) | 16 GB |
+| `transformers` + Unsloth, bf16 | 14 GB | 24 GB |
+| vLLM, bf16, `max_model_len=4096` | 14 GB | 24 GB |
+
+Audio decoding via SNAC runs on CPU and adds ~50–150 ms per utterance.
+
+---
+
+## License & attribution
+
+This fine-tune is released under **Apache-2.0**, matching the upstream
+[`unsloth/orpheus-3b-0.1-pretrained`](https://huggingface.co/unsloth/orpheus-3b-0.1-pretrained)
+license. It transitively inherits obligations from:
+
+- The [Orpheus-TTS](https://github.com/canopyai/Orpheus-TTS) project (CanopyAI).
+- The [Llama-3](https://llama.meta.com/llama3/) base architecture and weights — Meta Llama 3 Community License.
+- The [SNAC](https://github.com/hubertsiuzdak/snac) audio codec (Hubert Siuzdak, MIT).
+- The [`Sunbird/tts`](https://huggingface.co/datasets/Sunbird/tts) dataset and the SALT voice donors who contributed recordings.
+
+If you redistribute the merged weights, please carry these attributions
+forward.
+
+---
+
+## Citation
+
+If you use this model in your work, please cite both the dataset and the
+fine-tuning project:
+
+```bibtex
+@misc{sunbird_orpheus3b_multilingual_2026,
+  title        = {Orpheus-3B Sunbird Multilingual TTS},
+  author       = {Sunbird AI},
+  year         = {2026},
+  howpublished = {\url{https://huggingface.co/sunbird/orpheus-3b-tts-multilingual}},
+}
+
+@misc{sunbird_tts_dataset,
+  title        = {Sunbird Speech Dataset},
+  author       = {Sunbird AI},
+  howpublished = {\url{https://huggingface.co/datasets/Sunbird/tts}},
+}
+
+@misc{orpheus_tts_2025,
+  title        = {Orpheus-TTS},
+  author       = {Canopy Labs},
+  year         = {2025},
+  howpublished = {\url{https://github.com/canopyai/Orpheus-TTS}},
+}
+```
+
+---
+
+## Single-speaker variant
+
+If you only need one specific voice and want a smaller, more focused
+checkpoint, see
+[`sunbird/orpheus-3b-tts-salt-lug-0001`](https://huggingface.co/sunbird/orpheus-3b-tts-salt-lug-0001)
+— same recipe, scoped to a single Luganda speaker.
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,93 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- if strftime_now is defined %}
+        {%- set date_string = strftime_now("%d %b %Y") %}
+    {%- else %}
+        {%- set date_string = "26 Jul 2024" %}
+    {%- endif %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set system_message = messages[0]['content']|trim %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+{%- endif %}
+
+{#- System message #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if tools is not none %}
+    {{- "Environment: ipython\n" }}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+    {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+    {%- elif 'tool_calls' in message %}
+        {%- if not message.tool_calls|length == 1 %}
+            {{- raise_exception("This model only supports single tool-calls at once!") }}
+        {%- endif %}
+        {%- set tool_call = message.tool_calls[0].function %}
+        {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+        {{- '{"name": "' + tool_call.name + '", ' }}
+        {{- '"parameters": ' }}
+        {{- tool_call.arguments | tojson }}
+        {{- "}" }}
+        {{- "<|eot_id|>" }}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot_id|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,38 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 128000,
+  "torch_dtype": "bfloat16",
+  "eos_token_id": 128009,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 3072,
+  "initializer_range": 0.02,
+  "intermediate_size": 8192,
+  "max_position_embeddings": 131072,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 24,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "pad_token_id": 128004,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": {
+    "factor": 32.0,
+    "high_freq_factor": 4.0,
+    "low_freq_factor": 1.0,
+    "original_max_position_embeddings": 8192,
+    "rope_type": "llama3"
+  },
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": true,
+  "unsloth_fixed": true,
+  "unsloth_optimized": true,
+  "unsloth_version": "2026.4.8",
+  "use_cache": true,
+  "vocab_size": 156939
+}
--- a/model-00001-of-00002.safetensors
+++ b/model-00001-of-00002.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bdb7698468040694fa7577304b5c4d21e7b6229502e8f8b3bdf20a81673373b
+size 4991031824
--- a/model-00002-of-00002.safetensors
+++ b/model-00002-of-00002.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69b09255384b5ea8d5958b4e458b0209e6f37132568b4d1b4d473bb2176385fb
+size 1610725592
--- a/model.safetensors.index.json
+++ b/model.safetensors.index.json
@@ -0,0 +1,261 @@
+{
+  "metadata": {
+    "total_size": 6601728000
+  },
+  "weight_map": {
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,26 @@
+{
+  "additional_special_tokens": [
+    "<|audio|>"
+  ],
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|finetune_right_pad_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc3fecb199b4170636dbfab986d25f628157268d37b861f9cadaca60b1353bce
+size 22849547
--- a/tokenizer_config.json
+++ b/tokenizer_config.json