hypaai/Hypa-Orpheus-3b-TTS-VC
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: hypaai/Hypa-Orpheus-3b-TTS-VC
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-28 10:34:18 +08:00
commit 732a195980
14 changed files with 232815 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

BIN
Hypa_Orpheus.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 87 KiB

412
README.md Normal file
View File

@@ -0,0 +1,412 @@
---
library_name: transformers
pipeline_tag: text-generation
license: apache-2.0
base_model:
- unsloth/orpheus-3b-0.1-ft
datasets:
- hypaai/Hypa-Voices-snac
language:
- en
- fr
- es
- pt
- ar
- sw
- pcm
- yo
- ha
- ig
- ibb
- tiv
- efi
- anw
- idc
- igl
- nup
- ego
- fon
- urh
- tw
- ibr
tags:
- orpheus
- llama
- text-to-speech
- voice-cloning
- speech-synthesis
- snac
- audio-codec
- multilingual
- low-resource
- underrepresented-languages
- translation
- unsloth
- lora
- qlora
- hypa-ai
---
<div align="center">
![Hypa-Orpheus 3B](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC/resolve/main/Hypa_Orpheus.jpg)
**An open multilingual text-to-speech and zero-shot voice-cloning model for underrepresented languages, built on Orpheus 3B and trained on Hypa-Voices.**
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)
[![Base: Orpheus 3B](https://img.shields.io/badge/Base-Orpheus%203B-blue)](https://huggingface.co/unsloth/orpheus-3b-0.1-ft)
[![Collection: Hypa-Orpheus](https://img.shields.io/badge/%F0%9F%A4%97%20Collection-Hypa--Orpheus-yellow)](https://huggingface.co/collections/hypaai/hypa-orpheus)
[![Dataset: Hypa-Voices](https://img.shields.io/badge/Dataset-Hypa--Voices-green)](https://huggingface.co/collections/hypaai/hypa-voices)
[![Trained with Unsloth](https://img.shields.io/badge/Trained%20with-Unsloth-orange)](https://github.com/unslothai/unsloth)
</div>
## Model Description
**Hypa-Orpheus 3B** (`hypaai/Hypa-Orpheus-3b-TTS-VC`) is a LoRA-merged **16-bit bfloat16** checkpoint produced by [Hypa Intelligence](https://hypaintelligence.com). It is Step **3a** of our **Hypa-Whispering-Llama** roadmap: an independent multilingual **voicebox** that turns text (and optional reference speech) into 24 kHz waveform output via discrete **SNAC** audio tokens.
Built by fine-tuning [Orpheus 3B](https://github.com/canopyai/Orpheus-TTS) on the full **Hypa-Voices** corpus, the model unifies four speech capabilities as a single next-token prediction problem:
| Capability | Description |
| --- | --- |
| **Multilingual TTS** | Generate speech from text in a chosen language and named speaker voice. |
| **Translation-conditioned TTS** | Synthesize spoken audio from semantically equivalent text in another language. |
| **Voice cloning (VC)** | Reproduce a reference speaker's voice on new text from a single clip. |
| **Cross-lingual voice cloning** | Preserve speaker identity while generating speech in a different language. |
**Collection:** [hypaai/hypa-orpheus](https://huggingface.co/collections/hypaai/hypa-orpheus)
This release documents the two primary artifacts in the collection:
| Repository | Format | Best for |
| --- | --- | --- |
| [`hypaai/Hypa-Orpheus-3b-TTS-VC`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC) **(this repo)** | Merged 16-bit weights | Inference, vLLM serving, production deployment |
| [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs) | LoRA adapters + TensorBoard | Adapter stacking, continued fine-tuning, training metrics |
| Property | Value |
| --- | --- |
| **Base model** | [`unsloth/orpheus-3b-0.1-ft`](https://huggingface.co/unsloth/orpheus-3b-0.1-ft) |
| **Method** | 4-bit QLoRA (r=512, α=512) via Unsloth, merged to bf16 |
| **Trainable parameters** | 778M / 4.08B (19.07%) |
| **Training data** | 3.73M constructed examples (2.98M TTS + 756K VC) from Hypa-Voices |
| **Corpus scale** | 8,442 hours, 22 languages |
| **Shipped checkpoint** | Step **44,334** (validation-best) |
| **Sequence length (train)** | 2,048 tokens (~25 s of speech at ~82 tokens/sec) |
| **Audio codec** | [SNAC](https://github.com/hubertsiuzdak/snac) @ 24 kHz |
| **Compute** | 1× NVIDIA A100 80 GB PCIe (RunPod) |
| **License** | Apache 2.0 |
Hypa-Orpheus **powers** pronunciation in [Hypa Dictionary](https://dictionary.hypaintelligence.com/) and spoken output in [Hypa Translate](https://translate.hypaintelligence.com/), **supports** experimentation in [Hypa Labs](https://labs.hypaintelligence.com/), and **is being integrated into** Hypa Keyboard read-aloud. It also serves as the cascaded speech fallback in our longer-term plan to fuse hearing and speech inside Hypa-Whispering-Llama:
`User Speech → Hypa-Whisper → Hypa-Llama → Response Text → Hypa-Orpheus → Response Speech`
---
## Quick Start
### Load the merged model (Transformers)
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_id = "hypaai/Hypa-Orpheus-3b-TTS-VC"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
```
Orpheus generation requires **SNAC token decoding** and training-identical **control-token prompts**. For a complete TTS or voice-cloning pipeline (prompt assembly, SNAC encode/decode, frame deduplication, and WAV output), see the reference **`handler.py`** in this repository and the [Orpheus TTS](https://github.com/canopyai/Orpheus-TTS) project.
### vLLM serving
This repository includes a production **vLLM** inference handler (`handler.py`) with backward-compatible API support for legacy Hypa Orpheus clients. Serve with:
```bash
vllm serve hypaai/Hypa-Orpheus-3b-TTS-VC --dtype bfloat16 --max-model-len 4096
```
Use **bfloat16** at inference to match training numerics.
### Load LoRA adapters instead
To continue fine-tuning or stack adapters, use the companion repository:
```python
from peft import PeftModel
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/orpheus-3b-0.1-ft",
max_seq_length=2048,
dtype=None,
load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs")
```
See [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs) for TensorBoard logs and adapter details.
---
## Prompt Format
Hypa-Orpheus treats speech as causal language modeling over interleaved SNAC tokens. Training and inference must share the same prompt structure.
### Text prompts
- **`transcribe` (same-language TTS):** `{speaker}: {text}`
- **`translate` (translation-conditioned TTS):** `{speaker} - {Language}: {text}`
In **`translate`** rows from Hypa-Voices, **speech is in `src_lang` and text is in `tgt_lang`**. The language tag in the prompt refers to the **text side** (`tgt_lang`).
### Control tokens
Orpheus wraps content in role and speech markers. Two rules are fixed at train and serve time:
- **`<BOS>` (`<|begin_of_text|>`) appears once**, at the start of the first human turn.
- **Every text span ends with `<EOT>` (`<|eot_id|>`)**, including the boundary between two texts in one human turn.
**Vanilla TTS sequence:**
```plaintext
[start_of_human] <BOS> {speaker}: {text} <EOT> [end_of_human]
[start_of_ai] [start_of_speech] <SNAC codes> [end_of_speech] [end_of_ai]
```
**Voice cloning (continue-speaking layout):**
```plaintext
[start_of_human] <BOS> {ref_text} <EOT> {target_text} <EOT> [end_of_human]
[start_of_ai] [start_of_speech] <ref SNAC codes> ...continues into target codes... [end_of_speech] [end_of_ai]
```
SNAC codes use Orpheus's **7-token-per-frame interleave** (one coarse, two mid, four fine codes per frame), each offset into a distinct 4,096-token vocabulary slot above the base Llama vocabulary.
---
## Languages Covered
Hypa-Orpheus was trained on **22 languages** spanning AfroVoices studio recordings, African-accented English and French, and adapted public speech (including Common Voice-derived English and Pidgin):
| Language | Language | Language | Language |
| --- | --- | --- | --- |
| Annang | Arabic | Ebira | Efik |
| Eggon | English | Fongbe | French |
| Hausa | Ibibio | Idoma | Igala |
| Igbo | Nupe | Pidgin | Portuguese |
| Spanish | Swahili | Tiv | Twi |
| Urhobo | Yoruba | | |
Quality varies by language. Core AfroVoices languages received the densest supervision; tail languages (Fongbe, Twi, Urhobo, Ebira, and others) have thinner cross-lingual connectivity and should be validated by native listeners before production use.
---
## Training Data
Training data comes from [**Hypa-Voices**](https://huggingface.co/collections/hypaai/hypa-voices), our multilingual audio-text corpus:
| Split | Records | Role |
| --- | --- | --- |
| **TTS source rows** | 2,979,208 | Same-language transcription and cross-lingual translation pairs |
| **VC constructed rows** | 755,760 | Bidirectional same-speaker voice-cloning pairs |
| **Total training rows** | **3,734,968** | Mixed TTS + VC with modality-weighted interleaving |
Every row pairs speech and text as **`transcribe`** (same language) or **`translate`** (cross-lingual). Named speaker IDs enable controllable, voice-aware synthesis rather than anonymous audio.
### Offline SNAC precompute
The full corpus was normalized to fast-decoding **FLAC**, then **SNAC-encoded offline** into integer `codes_list` shards. Training reads only precomputed tokens, keeping the GPU compute-bound rather than blocked on audio decode or codec forward passes. A curated **8,800-record public subset** is released in both FLAC ([Hypa-Voices](https://huggingface.co/datasets/hypaai/Hypa-Voices)) and SNAC token form ([Hypa-Voices-snac](https://huggingface.co/datasets/hypaai/Hypa-Voices-snac)).
For corpus statistics (language tiers, speaker concentration, cross-lingual routes, and sequence-length filtering), see:
**[Hypa-Voices: analysis of the leading multilingual low-resource dataset](https://hypaintelligence.com/updates/hypa-voices-analysis-of-the-leading-multilingual-low-resource-dataset)**
---
## Training Procedure
| Hyperparameter | Value |
| --- | --- |
| LoRA rank (r) | 512 |
| LoRA alpha (α) | 512 |
| LoRA dropout | 0 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Quantization | 4-bit NF4 base, bf16 compute |
| Optimizer | AdamW 8-bit |
| Learning rate | 5e-5 |
| LR schedule | cosine, 3% warmup |
| Weight decay | 0.01 |
| Max grad norm | 1.0 |
| Per-device batch size | 16 |
| Gradient accumulation | 6 |
| **Effective batch size** | **96** |
| Sequence length | 2,048 |
| Prompt masking | Response-only (loss from `[start_of_speech]` onward) |
| Packing | Disabled |
| Checkpoint/eval interval | 2,217 steps |
| **Optimizer-step budget** | **44,334** |
| Precision | bfloat16 |
| Gradient checkpointing | Unsloth variant |
| Hardware | 1× NVIDIA A100 80 GB PCIe (RunPod) |
| Random seed | 3407 |
The step budget targets roughly one effective pass through the TTS distribution under weighted interleaving (~70% TTS / ~30% VC per drawn batch).
Training used [Unsloth](https://github.com/unslothai/unsloth) with 4-bit QLoRA. Labels use **response-only masking**: reference SNAC codes in voice-cloning prompts provide context but are not supervised targets.
---
## Evaluation and Checkpoint Selection
Interactive training logs:
**[TensorBoard on Hugging Face](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs/tensorboard)**
| Metric | Value | Step |
| --- | --- | --- |
| Initial training loss | 4.743 | 1 |
| Best training loss | 3.549 | 26,604 |
| Final training loss | 3.560 | 42,123 |
| Initial validation loss | 3.870 | 2,217 |
| **Best validation loss (shipped)** | **3.513** | **44,334** |
Validation loss improved at **every** evaluation interval (19 intervals total). Training loss reached its minimum earlier (step 26,604); because checkpoint selection follows held-out performance, the released weights use the **validation-best final step 44,334**.
Loss and approximate perplexity (`exp(loss)`) measure uncertainty over the supervised **target SNAC stream**, not text-only SFT perplexity and not speech naturalness directly. Informal listening checks across held-out speakers and languages agreed with the scalar choice; formal WER/CER, speaker-similarity, and MOS benchmarks are planned for a future evaluation release.
**For downstream use, we recommend the merged 16-bit weights in this repository** (step 44,334).
---
## Intended Use
**Direct use cases:**
- Multilingual text-to-speech with named speaker control
- Translation-conditioned speech synthesis (text in `tgt_lang`, speech reflecting `src_lang` semantics)
- Zero-shot voice cloning from a short reference clip
- Cross-lingual voice cloning when the reference and target languages differ
- Cascaded speech output paired with Hypa-Llama or other text models
**Downstream use:**
- Continued fine-tuning from the LoRA adapters in the companion repository
- Adapter stacking for domain- or language-specific heads
- Research on codec-token speech modeling and mixed-task TTS/VC training
## Out-of-Scope and Limitations
- **Not a general chat model.** Despite the Llama backbone, this checkpoint is trained for speech-token generation, not open-ended dialogue.
- **Prompt sensitivity.** Missing `<BOS>`, duplicate BOS tokens, or omitted internal `<EOT>` markers put inference off-distribution and can corrupt audio.
- **Context ceiling.** Training used 2,048 tokens (~25 s of speech). Longer outputs may degrade; voice-cloning rows with two clips hit the limit first.
- **Quality varies by language and speaker.** Tail languages and speakers with few training hours may produce less stable output.
- **No published perceptual benchmark yet.** Declining validation loss and informal listening support a stable fine-tune, but intelligibility, naturalness, and cloning fidelity require formal evaluation before strong product claims.
- **SNAC dependency.** Inference requires the SNAC codec (`hubertsiuzdak/snac_24khz`) for encode/decode around the language model.
## Bias, Risks, and Limitations
This model inherits biases from its base Orpheus weights and from Hypa-Voices, which is weighted toward AfroVoices studio material, English/French pivot translation routes, and adapted public speech. Generated speech should be reviewed for cultural appropriateness in customer-facing applications, especially for underrepresented languages with thinner training coverage.
Do not use this model unsupervised for applications affecting people's rights, health, finances, or wellbeing. Like all generative models, it can produce confident-sounding but incorrect speech.
---
## Released Artifacts
- **Merged 16-bit model (this repo):** [`hypaai/Hypa-Orpheus-3b-TTS-VC`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC)
- **LoRA adapters + TensorBoard:** [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs)
- **Base model:** [`unsloth/orpheus-3b-0.1-ft`](https://huggingface.co/unsloth/orpheus-3b-0.1-ft)
- **Training dataset (public subset):** [Hypa-Voices collection](https://huggingface.co/collections/hypaai/hypa-voices)
- **Dataset analysis:** [Hypa-Voices statistical breakdown](https://hypaintelligence.com/updates/hypa-voices-analysis-of-the-leading-multilingual-low-resource-dataset)
- **Upstream projects:** [Orpheus TTS](https://github.com/canopyai/Orpheus-TTS), [SNAC](https://github.com/hubertsiuzdak/snac), [Unsloth](https://github.com/unslothai/unsloth)
- **Related text model:** [Hypa-Llama collection](https://huggingface.co/collections/hypaai/hypa-llama3)
For notebooks, serving code, or contributions, contact **chris@hypaintelligence.com** or open an issue on Hugging Face.
---
## Citation
If you use Hypa-Orpheus 3B or any of the related work, please cite:
```bibtex
@misc{hypaorpheus2026,
title = {Hypa-Orpheus 3B: Multilingual Text-to-Speech and Voice Cloning for Underrepresented Languages},
author = {{Hypa Intelligence and AfroVoices}},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC}},
note = {Apache 2.0 License. Collection: \url{https://huggingface.co/collections/hypaai/hypa-orpheus}}
}
```
If you use the Hypa-Voices training corpus, please also cite:
```bibtex
@misc{hypavoices2026,
title={Hypa-Voices: Multilingual Low-Resource Audio-Text Dataset},
author={Hypa Intelligence and AfroVoices},
year={2026},
howpublished={Hugging Face collection},
url={https://huggingface.co/collections/hypaai/hypa-voices}
}
```
And the SNAC codec:
```bibtex
@inproceedings{siuzdak2024snac,
title={SNAC: Multi-Scale Neural Audio Codec},
author={Siuzdak, Hubert and Gr{\"o}tschla, Florian and Lanzend{\"o}rfer, Luca A.},
booktitle={Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation},
year={2024}
}
```
---
## License
Released under the **Apache License 2.0**. See the `LICENSE` file in this repository for full details.
---
## Acknowledgments
- **Canopy Labs**, for **Orpheus TTS** and for treating speech generation as language modeling over discrete audio tokens.
- **Meta AI / FAIR**, for the **Llama** family and the open-weights ecosystem Orpheus builds on.
- **Hubert Siuzdak** and collaborators, for **SNAC** and for open-sourcing the codec.
- **Unsloth**, for efficient **4-bit QLoRA** and Orpheus integration.
- **RunPod**, for reliable compute during the production fine-tune.
- **Mozilla** and the **Common Voice** community, for open speech data used in parts of Hypa-Voices.
- **AfroVoices**, for the steadfast curation of high-quality recordings across underrepresented languages.
- **Hypa Intelligence Research (HaIR)**, for our commitment to breaking the language barrier.
---
## Contact and Contributions
For questions, issues, or contributions, please open an issue in this repository or contact **chris@hypaintelligence.com**. Contributions are welcome.
---
## Closing Remarks
By releasing Hypa-Orpheus, we hope to give researchers and developers practical access to multilingual speech synthesis and voice cloning for underrepresented languages, and to make the data path, training recipe, and checkpoint-selection process transparent from the start.
At **Hypa Intelligence**, we believe that for AI to be truly aligned with humanity, it must understand and represent all of us, not just a select few. The first step toward this goal is solving the challenge of multilingualism by **breaking the language barrier**.
Hypa Intelligence remains steadfast in its mission to accelerate the advent of AGI and ASI and to ensure their benefits are globally distributed.
**AfroVoices**, a subsidiary of Hypa AI, is dedicated to amplifying African voices, languages, and cultures in the intelligence age. Focused on bridging the digital representation gap, AfroVoices curates datasets and resources for African languages, promoting inclusivity and cultural appreciation in AI technologies.
---
**Hypa Intelligence** • [Website](https://hypaintelligence.com) • [Hugging Face](https://huggingface.co/hypaai) • [Updates](https://hypaintelligence.com/updates)
*Multilingualism is not a feature. It is a prerequisite for AI that represents all of us.*

93
chat_template.jinja Normal file
View File

@@ -0,0 +1,93 @@
{{- bos_token }}
{%- if custom_tools is defined %}
{%- set tools = custom_tools %}
{%- endif %}
{%- if not tools_in_user_message is defined %}
{%- set tools_in_user_message = true %}
{%- endif %}
{%- if not date_string is defined %}
{%- if strftime_now is defined %}
{%- set date_string = strftime_now("%d %b %Y") %}
{%- else %}
{%- set date_string = "26 Jul 2024" %}
{%- endif %}
{%- endif %}
{%- if not tools is defined %}
{%- set tools = none %}
{%- endif %}
{#- This block extracts the system message, so we can slot it into the right place. #}
{%- if messages[0]['role'] == 'system' %}
{%- set system_message = messages[0]['content']|trim %}
{%- set messages = messages[1:] %}
{%- else %}
{%- set system_message = "" %}
{%- endif %}
{#- System message #}
{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
{%- if tools is not none %}
{{- "Environment: ipython\n" }}
{%- endif %}
{{- "Cutting Knowledge Date: December 2023\n" }}
{{- "Today Date: " + date_string + "\n\n" }}
{%- if tools is not none and not tools_in_user_message %}
{{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
{{- "Do not use variables.\n\n" }}
{%- for t in tools %}
{{- t | tojson(indent=4) }}
{{- "\n\n" }}
{%- endfor %}
{%- endif %}
{{- system_message }}
{{- "<|eot_id|>" }}
{#- Custom tools are passed in a user message with some extra guidance #}
{%- if tools_in_user_message and not tools is none %}
{#- Extract the first user message so we can plug it in here #}
{%- if messages | length != 0 %}
{%- set first_user_message = messages[0]['content']|trim %}
{%- set messages = messages[1:] %}
{%- else %}
{{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
{%- endif %}
{{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
{{- "Given the following functions, please respond with a JSON for a function call " }}
{{- "with its proper arguments that best answers the given prompt.\n\n" }}
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
{{- "Do not use variables.\n\n" }}
{%- for t in tools %}
{{- t | tojson(indent=4) }}
{{- "\n\n" }}
{%- endfor %}
{{- first_user_message + "<|eot_id|>"}}
{%- endif %}
{%- for message in messages %}
{%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
{{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
{%- elif 'tool_calls' in message %}
{%- if not message.tool_calls|length == 1 %}
{{- raise_exception("This model only supports single tool-calls at once!") }}
{%- endif %}
{%- set tool_call = message.tool_calls[0].function %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
{{- '{"name": "' + tool_call.name + '", ' }}
{{- '"parameters": ' }}
{{- tool_call.arguments | tojson }}
{{- "}" }}
{{- "<|eot_id|>" }}
{%- elif message.role == "tool" or message.role == "ipython" %}
{{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
{%- if message.content is mapping or message.content is iterable %}
{{- message.content | tojson }}
{%- else %}
{{- message.content }}
{%- endif %}
{{- "<|eot_id|>" }}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
{%- endif %}

38
config.json Normal file
View File

@@ -0,0 +1,38 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 128000,
"dtype": "bfloat16",
"eos_token_id": 128009,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 3072,
"initializer_range": 0.02,
"intermediate_size": 8192,
"max_position_embeddings": 131072,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 24,
"num_hidden_layers": 28,
"num_key_value_heads": 8,
"pad_token_id": 128004,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": {
"factor": 32.0,
"high_freq_factor": 4.0,
"low_freq_factor": 1.0,
"original_max_position_embeddings": 8192,
"rope_type": "llama3"
},
"rope_theta": 500000.0,
"tie_word_embeddings": true,
"transformers_version": "4.57.3",
"unsloth_fixed": true,
"unsloth_version": "2025.12.9",
"use_cache": true,
"vocab_size": 156940
}

13
generation_config.json Normal file
View File

@@ -0,0 +1,13 @@
{
"_from_model_config": true,
"bos_token_id": 128000,
"do_sample": true,
"eos_token_id": [
128009
],
"max_length": 131072,
"pad_token_id": 128004,
"temperature": 0.6,
"top_p": 0.9,
"transformers_version": "4.57.3"
}

375
handler.py Normal file
View File

@@ -0,0 +1,375 @@
"""
HF Inference Endpoint handler — Hypa Orpheus TTS + Voice Cloning (Step-III merged 16-bit).
BACKWARD COMPATIBLE with the legacy hypaai_orpheus_v5 API: legacy clients work by
changing only the endpoint URL. Legacy schema honored:
data: inputs, clone, clone_on_the_fly, enroll_user, cloning_features, enrollments
parameters: voice (default "Eniola"), temperature, top_p, max_new_tokens, repetition_penalty
Legacy output honored: audio_b64 = base64 WAV/RIFF PCM_16 @24kHz, audio_sample =
raw float32 mono waveform, sample_rate, input_ids_len, gen_ids_len.
NEW capabilities (Step-III model) via parameters:
task ("tts"|"vc"), mode ("vanilla"|"translate"), language, method ("m1"|"m2"),
reference_text + reference_audio (base64), top_k.
Serving notes vs legacy handler (deliberate changes):
- Prompts reach vLLM as token ids (legacy decoded to a string and re-tokenized,
risking a double-BOS and mangled audio tokens in cloning prompts).
- dtype="bfloat16" forced (legacy inherited config torch_dtype; a bf16-trained
model served in fp16 is the leading suspect for endpoint-only audio artifacts).
- Reference codes are frame-deduped to match Step-III training data.
"""
import io
import os
import base64
import tempfile
import traceback
import numpy as np
import torch
import soundfile as sf
import librosa
from transformers import AutoTokenizer
from snac import SNAC
from vllm import LLM, SamplingParams
class EndpointHandler:
TOKENISER_LEN = 128256
START_OF_TEXT = 128000
END_OF_TEXT = 128009
START_OF_SPEECH = TOKENISER_LEN + 1 # 128257
END_OF_SPEECH = TOKENISER_LEN + 2 # 128258
START_OF_HUMAN = TOKENISER_LEN + 3 # 128259
END_OF_HUMAN = TOKENISER_LEN + 4 # 128260
START_OF_AI = TOKENISER_LEN + 5 # 128261
END_OF_AI = TOKENISER_LEN + 6 # 128262
AUDIO_OFFSET = 128266
MAX_MODEL_LEN = 4096
MAX_REF_SECONDS = 30
SNAC_SR = 24000
LANG_DISPLAY = {
"en": "English", "es": "Spanish", "fr": "French", "ha": "Hausa",
"yo": "Yoruba", "sw": "Swahili", "ar": "Arabic", "pt": "Portuguese",
"ann": "Annang", "ebi": "Ebira", "efi": "Efik", "ego": "Eggon",
"urh": "Urhobo", "ibb": "Ibibio", "idm": "Idoma", "igl": "Igala",
"ig": "Igbo", "nup": "Nupe", "tiv": "Tiv", "pg": "Pidgin",
}
# ------------------------------------------------------------------ init
def __init__(self, path=""):
self.device = "cuda" if torch.cuda.is_available() else "cpu"
self.snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").to(self.device).eval()
self.model = LLM(
path,
max_model_len=self.MAX_MODEL_LEN,
gpu_memory_utilization=0.75,
dtype="bfloat16", # match training numerics (see docstring)
)
self.tokenizer = AutoTokenizer.from_pretrained(path)
# ------------------------------------------------------- text encoding
def _lang_display(self, x):
if x is None:
return None
k = str(x).strip().lower()
return self.LANG_DISPLAY.get(k, k.capitalize() if k else None)
def _encode_text_ids(self, text, speaker=None, lang_tag=None):
"""Training-identical text content: '{spk} - {Lang}: {text}' variants.
Returns bare content ids WITHOUT specials (block adds them)."""
text = "" if text is None else str(text).strip()
spk = speaker if (speaker and str(speaker).strip().lower() not in ("", "random", "none")) else None
if spk and lang_tag:
prompt = f"{spk} - {lang_tag}: {text}"
elif spk:
prompt = f"{spk}: {text}"
elif lang_tag:
prompt = f"{lang_tag}: {text}"
else:
prompt = text
return self.tokenizer.encode(prompt, add_special_tokens=False)
def _text_block(self, content_ids, with_bos=True):
"""[SOH] (+BOS) content [EOT] [EOH] — equals legacy format_text_block and
training's [SOH]+encode(add_bos)+[EOT]+[EOH] (BOS == START_OF_TEXT)."""
bos = [self.START_OF_TEXT] if with_bos else []
return [self.START_OF_HUMAN] + bos + list(content_ids) + [self.END_OF_TEXT, self.END_OF_HUMAN]
def _audio_block(self, codes):
return [self.START_OF_AI, self.START_OF_SPEECH] + list(codes) + \
[self.END_OF_SPEECH, self.END_OF_AI]
def _open_speech(self):
return [self.START_OF_AI, self.START_OF_SPEECH]
# ------------------------------------------------------ audio encoding
def _b64_to_wave(self, b64_str):
raw = base64.b64decode(b64_str)
if not raw:
raise ValueError("reference audio is empty.")
try:
arr, sr = sf.read(io.BytesIO(raw), dtype="float32")
except Exception:
tmp = None
try:
with tempfile.NamedTemporaryFile(delete=False, suffix=".audio") as f:
f.write(raw)
tmp = f.name
arr, sr = librosa.load(tmp, sr=None, mono=False)
arr = np.asarray(arr, dtype=np.float32)
if arr.ndim > 1:
arr = arr.T
finally:
if tmp and os.path.exists(tmp):
os.remove(tmp)
if arr.ndim > 1:
arr = arr.mean(axis=1)
if arr.size == 0 or not np.isfinite(arr).all():
raise ValueError("Reference audio is empty or contains invalid samples.")
if sr != self.SNAC_SR:
arr = librosa.resample(arr.astype(np.float32), orig_sr=sr, target_sr=self.SNAC_SR)
dur = len(arr) / self.SNAC_SR
if dur > self.MAX_REF_SECONDS:
raise ValueError(f"Reference audio is {dur:.1f}s; max is {self.MAX_REF_SECONDS}s.")
return arr.astype(np.float32)
@torch.inference_mode()
def _audio_to_codes(self, arr):
wav = torch.from_numpy(arr).to(self.device)[None, None]
codes = self.snac_model.encode(wav)
c0, c1, c2 = codes[0][0].tolist(), codes[1][0].tolist(), codes[2][0].tolist()
n = min(len(c0), len(c1) // 2, len(c2) // 4)
out = []
for i in range(n):
out += [
c0[i] + self.AUDIO_OFFSET,
c1[2 * i] + self.AUDIO_OFFSET + 4096,
c2[4 * i] + self.AUDIO_OFFSET + 2 * 4096,
c2[4 * i + 1] + self.AUDIO_OFFSET + 3 * 4096,
c1[2 * i + 1] + self.AUDIO_OFFSET + 4 * 4096,
c2[4 * i + 2] + self.AUDIO_OFFSET + 5 * 4096,
c2[4 * i + 3] + self.AUDIO_OFFSET + 6 * 4096,
]
return out
@staticmethod
def _dedup_frames(codes):
if not codes:
return codes
codes = list(codes)[: (len(codes) // 7) * 7]
if len(codes) < 7:
return codes
result = codes[:7]
for i in range(7, len(codes), 7):
if codes[i] != result[-7]:
result.extend(codes[i:i + 7])
return result
# ------------------------------------------------------ legacy enrollment
def enroll_user(self, enrollment_pairs):
"""Legacy-format enrollment: torch-serialized {text_ids tensor, audio_codes list}.
Previously issued cloning_features blobs remain loadable."""
enrollment_data = []
for text, base64_audio in enrollment_pairs:
text_ids = self.tokenizer.encode(text, return_tensors="pt",
add_special_tokens=False).cpu()
audio_codes = self._dedup_frames(self._audio_to_codes(self._b64_to_wave(base64_audio)))
enrollment_data.append({"text_ids": text_ids, "audio_codes": audio_codes})
buffer = io.BytesIO()
torch.save(enrollment_data, buffer)
buffer.seek(0)
return base64.b64encode(buffer.read()).decode("utf-8")
# --------------------------------------------------------- generation
def _generate(self, prompt_ids, gp):
sampling = SamplingParams(
temperature = gp["temperature"],
top_p = gp["top_p"],
top_k = gp["top_k"],
max_tokens = gp["max_new_tokens"],
repetition_penalty = gp["repetition_penalty"],
stop_token_ids = [self.END_OF_SPEECH, self.END_OF_AI],
detokenize = False,
)
outputs = self.model.generate({"prompt_token_ids": prompt_ids}, sampling)
return list(outputs[0].outputs[0].token_ids)
# ----------------------------------------------------------- decoding
@torch.inference_mode()
def _codes_to_wave(self, gen_ids):
frames, i, n, resyncs = [], 0, len(gen_ids), 0
while i <= n - 7:
vals, ok = [], True
for k in range(7):
lo = self.AUDIO_OFFSET + k * 4096
t = gen_ids[i + k]
if not (lo <= t < lo + 4096):
ok = False
break
vals.append(t - lo)
if ok:
frames.append(vals)
i += 7
else:
i += 1
resyncs += 1
self._last_resyncs = resyncs
if not frames:
return None
l1 = [f[0] for f in frames]
l2, l3 = [], []
for f in frames:
l2.append(f[1]); l3.append(f[2]); l3.append(f[3])
l2.append(f[4]); l3.append(f[5]); l3.append(f[6])
tensors = [torch.tensor(l1)[None].to(self.device),
torch.tensor(l2)[None].to(self.device),
torch.tensor(l3)[None].to(self.device)]
return self.snac_model.decode(tensors).squeeze().detach().cpu().numpy()
# -------------------------------------------------------------- entry
def __call__(self, data):
try:
# ---- legacy enrollment path (unchanged API) ----
if data.get("enroll_user", False):
pairs = data.get("enrollments", [])
if not pairs:
return {"error": "No enrollment pairs provided"}
return {"cloning_features": self.enroll_user(pairs)}
target_text = data.get("inputs")
if not target_text:
return {"error": "Missing 'inputs' (target text)."}
p = data.get("parameters", {}) or {}
gp = {
"temperature": float(p.get("temperature", 0.6)),
"top_p": float(p.get("top_p", 0.95)),
"top_k": int(p.get("top_k", -1)), # legacy default: no top-k
"max_new_tokens": int(p.get("max_new_tokens", 1200)),
"repetition_penalty": float(p.get("repetition_penalty", 1.1)),
}
if not 0 < gp["top_p"] <= 1:
return {"error": "top_p must be within (0, 1]."}
if not (gp["top_k"] == -1 or gp["top_k"] > 0):
return {"error": "top_k must be -1 (disabled) or a positive integer."}
if not 0 < gp["repetition_penalty"] <= 2:
return {"error": "repetition_penalty must be within (0, 2]."}
if gp["max_new_tokens"] <= 0:
return {"error": "max_new_tokens must be positive."}
task = str(p.get("task", "")).lower()
mode = str(p.get("mode", "vanilla")).lower()
method = str(p.get("method", "m2")).lower()
if mode in ("translation", "trans"):
mode = "translate"
if task and task not in ("tts", "vc"):
return {"error": "parameters.task must be 'tts' or 'vc'."}
if mode not in ("vanilla", "translate"):
return {"error": "parameters.mode must be 'vanilla' or 'translate'."}
if mode == "translate" and not p.get("language"):
return {"error": "parameters.language is required for translate mode."}
lang_tag = self._lang_display(p.get("language")) if mode == "translate" else None
legacy_clone = bool(data.get("clone", False))
resolved_task = "vc" if (legacy_clone or task == "vc") else "tts"
# ---- build prompt ----
if legacy_clone:
# Legacy multi-pair in-context cloning (== M1 generalized)
if data.get("clone_on_the_fly", False):
pairs = data.get("enrollments", [])
if not pairs:
return {"error": "No enrollment pairs provided"}
enrollment = []
for text, b64 in pairs:
enrollment.append({
"text_ids": self.tokenizer.encode(text, add_special_tokens=False),
"audio_codes": self._dedup_frames(
self._audio_to_codes(self._b64_to_wave(b64))),
})
else:
feats = data.get("cloning_features")
if not feats:
return {"error": "No cloning features were provided"}
loaded = torch.load(io.BytesIO(base64.b64decode(feats)))
enrollment = [{
"text_ids": (it["text_ids"].flatten().tolist()
if torch.is_tensor(it["text_ids"]) else list(it["text_ids"])),
"audio_codes": self._dedup_frames(list(it["audio_codes"])),
} for it in loaded]
prompt_ids, method_out = [], "m1"
for it in enrollment:
prompt_ids += self._text_block(it["text_ids"])
prompt_ids += self._audio_block(it["audio_codes"])
prompt_ids += self._text_block(
self._encode_text_ids(target_text, None, lang_tag))
prompt_ids += self._open_speech()
elif resolved_task == "vc":
ref_text = p.get("reference_text")
ref_audio = p.get("reference_audio")
if not ref_text or not ref_audio:
return {"error": "VC requires parameters.reference_text and "
"parameters.reference_audio (base64)."}
if method not in ("m1", "m2"):
return {"error": "parameters.method must be 'm1' or 'm2'."}
ref_codes = self._dedup_frames(self._audio_to_codes(self._b64_to_wave(ref_audio)))
if not ref_codes:
return {"error": "Reference audio produced no SNAC codes."}
tt1 = self._encode_text_ids(ref_text)
tt2 = self._encode_text_ids(target_text, None, lang_tag)
method_out = method
if method == "m1":
prompt_ids = (self._text_block(tt1) + self._audio_block(ref_codes) +
self._text_block(tt2) + self._open_speech())
else: # m2 continue-speaking: both texts one turn, ref codes open the AI turn
prompt_ids = ([self.START_OF_HUMAN, self.START_OF_TEXT] + tt1 +
tt2 + [self.END_OF_TEXT, self.END_OF_HUMAN] +
self._open_speech() + list(ref_codes))
else: # TTS (legacy default voice preserved)
voice = p.get("voice") or p.get("speaker") or "Eniola"
method_out = None
prompt_ids = self._text_block(
self._encode_text_ids(target_text, voice, lang_tag)) + self._open_speech()
budget = self.MAX_MODEL_LEN - gp["max_new_tokens"]
if len(prompt_ids) > budget:
return {"error": f"Prompt is {len(prompt_ids)} tokens; exceeds budget "
f"{budget} (max_model_len - max_new_tokens)."}
gen_ids = self._generate(prompt_ids, gp)
wav = self._codes_to_wave(gen_ids)
if wav is None:
return {"error": "Model generated no audio tokens.",
"input_ids_len": len(prompt_ids),
"gen_ids_len": len(gen_ids)}
# Legacy output format: WAV/RIFF PCM_16 base64 + raw float32 waveform
buffer = io.BytesIO()
sf.write(buffer, wav, samplerate=self.SNAC_SR, format="WAV", subtype="PCM_16")
buffer.seek(0)
audio_b64 = base64.b64encode(buffer.read()).decode("utf-8")
return {
"audio_sample": wav.astype(np.float32).tolist(), # raw waveform (legacy field)
"audio_b64": audio_b64, # base64 WAV PCM_16 (legacy)
"sample_rate": self.SNAC_SR,
"input_ids_len": len(prompt_ids),
"gen_ids_len": len(gen_ids),
"duration_seconds": round(len(wav) / self.SNAC_SR, 3),
"task": resolved_task, "mode": mode, "method": method_out,
"decode_resyncs": getattr(self, "_last_resyncs", 0),
**({"gen_token_ids": gen_ids} if p.get("return_tokens") else {}),
}
except ValueError as e:
return {"error": str(e)}
except Exception as e:
traceback.print_exc()
return {"error": str(e)}

3
model-00001-of-00002.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:24538039b8665ce1e73b83114a4b0b773b24816071296d7a565fa2165b082817
size 4991037968

3
model-00002-of-00002.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e6a128dbc5871a7cdfaf6cbfee6ef1cb5eb115a3122dd5cb928e7de3ae05f2c2
size 1610725592

261
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,261 @@
{
"metadata": {
"total_size": 6601734144
},
"weight_map": {
"model.embed_tokens.weight": "model-00001-of-00002.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.20.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.24.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.25.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.25.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.26.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.input_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
"model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
"model.norm.weight": "model-00002-of-00002.safetensors"
}
}

11
requirements.txt Normal file
View File

@@ -0,0 +1,11 @@
--extra-index-url https://download.pytorch.org/whl/cu121
torch==2.4.0+cu121
torchaudio==2.4.0+cu121
transformers==4.45.2
accelerate==0.34.2
numpy==1.26.4
protobuf==4.25.3
snac==1.2.1
diffusers==0.30.3
vllm==0.6.3.post1
starlette<1.0

26
special_tokens_map.json Normal file
View File

@@ -0,0 +1,26 @@
{
"additional_special_tokens": [
"<|audio|>"
],
"bos_token": {
"content": "<|begin_of_text|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|eot_id|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|finetune_right_pad_id|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fc3fecb199b4170636dbfab986d25f628157268d37b861f9cadaca60b1353bce
size 22849547

231541
tokenizer_config.json Normal file
View File

File diff suppressed because it is too large Load Diff