---
library_name: transformers
pipeline_tag: text-generation
license: apache-2.0
base_model:
- unsloth/orpheus-3b-0.1-ft
datasets:
- hypaai/Hypa-Voices-snac
language:
- en
- fr
- es
- pt
- ar
- sw
- pcm
- yo
- ha
- ig
- ibb
- tiv
- efi
- anw
- idc
- igl
- nup
- ego
- fon
- urh
- tw
- ibr
tags:
- orpheus
- llama
- text-to-speech
- voice-cloning
- speech-synthesis
- snac
- audio-codec
- multilingual
- low-resource
- underrepresented-languages
- translation
- unsloth
- lora
- qlora
- hypa-ai
---
**An open multilingual text-to-speech and zero-shot voice-cloning model for underrepresented languages, built on Orpheus 3B and trained on Hypa-Voices.**
[](https://www.apache.org/licenses/LICENSE-2.0)
[](https://huggingface.co/unsloth/orpheus-3b-0.1-ft)
[](https://huggingface.co/collections/hypaai/hypa-orpheus)
[](https://huggingface.co/collections/hypaai/hypa-voices)
[](https://github.com/unslothai/unsloth)
## Model Description
**Hypa-Orpheus 3B** (`hypaai/Hypa-Orpheus-3b-TTS-VC`) is a LoRA-merged **16-bit bfloat16** checkpoint produced by [Hypa Intelligence](https://hypaintelligence.com). It is Step **3a** of our **Hypa-Whispering-Llama** roadmap: an independent multilingual **voicebox** that turns text (and optional reference speech) into 24 kHz waveform output via discrete **SNAC** audio tokens.
Built by fine-tuning [Orpheus 3B](https://github.com/canopyai/Orpheus-TTS) on the full **Hypa-Voices** corpus, the model unifies four speech capabilities as a single next-token prediction problem:
| Capability | Description |
| --- | --- |
| **Multilingual TTS** | Generate speech from text in a chosen language and named speaker voice. |
| **Translation-conditioned TTS** | Synthesize spoken audio from semantically equivalent text in another language. |
| **Voice cloning (VC)** | Reproduce a reference speaker's voice on new text from a single clip. |
| **Cross-lingual voice cloning** | Preserve speaker identity while generating speech in a different language. |
**Collection:** [hypaai/hypa-orpheus](https://huggingface.co/collections/hypaai/hypa-orpheus)
This release documents the two primary artifacts in the collection:
| Repository | Format | Best for |
| --- | --- | --- |
| [`hypaai/Hypa-Orpheus-3b-TTS-VC`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC) **(this repo)** | Merged 16-bit weights | Inference, vLLM serving, production deployment |
| [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs) | LoRA adapters + TensorBoard | Adapter stacking, continued fine-tuning, training metrics |
| Property | Value |
| --- | --- |
| **Base model** | [`unsloth/orpheus-3b-0.1-ft`](https://huggingface.co/unsloth/orpheus-3b-0.1-ft) |
| **Method** | 4-bit QLoRA (r=512, α=512) via Unsloth, merged to bf16 |
| **Trainable parameters** | 778M / 4.08B (19.07%) |
| **Training data** | 3.73M constructed examples (2.98M TTS + 756K VC) from Hypa-Voices |
| **Corpus scale** | 8,442 hours, 22 languages |
| **Shipped checkpoint** | Step **44,334** (validation-best) |
| **Sequence length (train)** | 2,048 tokens (~25 s of speech at ~82 tokens/sec) |
| **Audio codec** | [SNAC](https://github.com/hubertsiuzdak/snac) @ 24 kHz |
| **Compute** | 1× NVIDIA A100 80 GB PCIe (RunPod) |
| **License** | Apache 2.0 |
Hypa-Orpheus **powers** pronunciation in [Hypa Dictionary](https://dictionary.hypaintelligence.com/) and spoken output in [Hypa Translate](https://translate.hypaintelligence.com/), **supports** experimentation in [Hypa Labs](https://labs.hypaintelligence.com/), and **is being integrated into** Hypa Keyboard read-aloud. It also serves as the cascaded speech fallback in our longer-term plan to fuse hearing and speech inside Hypa-Whispering-Llama:
`User Speech → Hypa-Whisper → Hypa-Llama → Response Text → Hypa-Orpheus → Response Speech`
---
## Quick Start
### Load the merged model (Transformers)
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_id = "hypaai/Hypa-Orpheus-3b-TTS-VC"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
```
Orpheus generation requires **SNAC token decoding** and training-identical **control-token prompts**. For a complete TTS or voice-cloning pipeline (prompt assembly, SNAC encode/decode, frame deduplication, and WAV output), see the reference **`handler.py`** in this repository and the [Orpheus TTS](https://github.com/canopyai/Orpheus-TTS) project.
### vLLM serving
This repository includes a production **vLLM** inference handler (`handler.py`) with backward-compatible API support for legacy Hypa Orpheus clients. Serve with:
```bash
vllm serve hypaai/Hypa-Orpheus-3b-TTS-VC --dtype bfloat16 --max-model-len 4096
```
Use **bfloat16** at inference to match training numerics.
### Load LoRA adapters instead
To continue fine-tuning or stack adapters, use the companion repository:
```python
from peft import PeftModel
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/orpheus-3b-0.1-ft",
max_seq_length=2048,
dtype=None,
load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs")
```
See [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs) for TensorBoard logs and adapter details.
---
## Prompt Format
Hypa-Orpheus treats speech as causal language modeling over interleaved SNAC tokens. Training and inference must share the same prompt structure.
### Text prompts
- **`transcribe` (same-language TTS):** `{speaker}: {text}`
- **`translate` (translation-conditioned TTS):** `{speaker} - {Language}: {text}`
In **`translate`** rows from Hypa-Voices, **speech is in `src_lang` and text is in `tgt_lang`**. The language tag in the prompt refers to the **text side** (`tgt_lang`).
### Control tokens
Orpheus wraps content in role and speech markers. Two rules are fixed at train and serve time:
- **`` (`<|begin_of_text|>`) appears once**, at the start of the first human turn.
- **Every text span ends with `` (`<|eot_id|>`)**, including the boundary between two texts in one human turn.
**Vanilla TTS sequence:**
```plaintext
[start_of_human] {speaker}: {text} [end_of_human]
[start_of_ai] [start_of_speech] [end_of_speech] [end_of_ai]
```
**Voice cloning (continue-speaking layout):**
```plaintext
[start_of_human] {ref_text} {target_text} [end_of_human]
[start_of_ai] [start_of_speech] [ ...continues into target codes... [end_of_speech] [end_of_ai]
```
SNAC codes use Orpheus's **7-token-per-frame interleave** (one coarse, two mid, four fine codes per frame), each offset into a distinct 4,096-token vocabulary slot above the base Llama vocabulary.
---
## Languages Covered
Hypa-Orpheus was trained on **22 languages** spanning AfroVoices studio recordings, African-accented English and French, and adapted public speech (including Common Voice-derived English and Pidgin):
| Language | Language | Language | Language |
| --- | --- | --- | --- |
| Annang | Arabic | Ebira | Efik |
| Eggon | English | Fongbe | French |
| Hausa | Ibibio | Idoma | Igala |
| Igbo | Nupe | Pidgin | Portuguese |
| Spanish | Swahili | Tiv | Twi |
| Urhobo | Yoruba | | |
Quality varies by language. Core AfroVoices languages received the densest supervision; tail languages (Fongbe, Twi, Urhobo, Ebira, and others) have thinner cross-lingual connectivity and should be validated by native listeners before production use.
---
## Training Data
Training data comes from [**Hypa-Voices**](https://huggingface.co/collections/hypaai/hypa-voices), our multilingual audio-text corpus:
| Split | Records | Role |
| --- | --- | --- |
| **TTS source rows** | 2,979,208 | Same-language transcription and cross-lingual translation pairs |
| **VC constructed rows** | 755,760 | Bidirectional same-speaker voice-cloning pairs |
| **Total training rows** | **3,734,968** | Mixed TTS + VC with modality-weighted interleaving |
Every row pairs speech and text as **`transcribe`** (same language) or **`translate`** (cross-lingual). Named speaker IDs enable controllable, voice-aware synthesis rather than anonymous audio.
### Offline SNAC precompute
The full corpus was normalized to fast-decoding **FLAC**, then **SNAC-encoded offline** into integer `codes_list` shards. Training reads only precomputed tokens, keeping the GPU compute-bound rather than blocked on audio decode or codec forward passes. A curated **8,800-record public subset** is released in both FLAC ([Hypa-Voices](https://huggingface.co/datasets/hypaai/Hypa-Voices)) and SNAC token form ([Hypa-Voices-snac](https://huggingface.co/datasets/hypaai/Hypa-Voices-snac)).
For corpus statistics (language tiers, speaker concentration, cross-lingual routes, and sequence-length filtering), see:
**[Hypa-Voices: analysis of the leading multilingual low-resource dataset](https://hypaintelligence.com/updates/hypa-voices-analysis-of-the-leading-multilingual-low-resource-dataset)**
---
## Training Procedure
| Hyperparameter | Value |
| --- | --- |
| LoRA rank (r) | 512 |
| LoRA alpha (α) | 512 |
| LoRA dropout | 0 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Quantization | 4-bit NF4 base, bf16 compute |
| Optimizer | AdamW 8-bit |
| Learning rate | 5e-5 |
| LR schedule | cosine, 3% warmup |
| Weight decay | 0.01 |
| Max grad norm | 1.0 |
| Per-device batch size | 16 |
| Gradient accumulation | 6 |
| **Effective batch size** | **96** |
| Sequence length | 2,048 |
| Prompt masking | Response-only (loss from `[start_of_speech]` onward) |
| Packing | Disabled |
| Checkpoint/eval interval | 2,217 steps |
| **Optimizer-step budget** | **44,334** |
| Precision | bfloat16 |
| Gradient checkpointing | Unsloth variant |
| Hardware | 1× NVIDIA A100 80 GB PCIe (RunPod) |
| Random seed | 3407 |
The step budget targets roughly one effective pass through the TTS distribution under weighted interleaving (~70% TTS / ~30% VC per drawn batch).
Training used [Unsloth](https://github.com/unslothai/unsloth) with 4-bit QLoRA. Labels use **response-only masking**: reference SNAC codes in voice-cloning prompts provide context but are not supervised targets.
---
## Evaluation and Checkpoint Selection
Interactive training logs:
**[TensorBoard on Hugging Face](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs/tensorboard)**
| Metric | Value | Step |
| --- | --- | --- |
| Initial training loss | 4.743 | 1 |
| Best training loss | 3.549 | 26,604 |
| Final training loss | 3.560 | 42,123 |
| Initial validation loss | 3.870 | 2,217 |
| **Best validation loss (shipped)** | **3.513** | **44,334** |
Validation loss improved at **every** evaluation interval (19 intervals total). Training loss reached its minimum earlier (step 26,604); because checkpoint selection follows held-out performance, the released weights use the **validation-best final step 44,334**.
Loss and approximate perplexity (`exp(loss)`) measure uncertainty over the supervised **target SNAC stream**, not text-only SFT perplexity and not speech naturalness directly. Informal listening checks across held-out speakers and languages agreed with the scalar choice; formal WER/CER, speaker-similarity, and MOS benchmarks are planned for a future evaluation release.
**For downstream use, we recommend the merged 16-bit weights in this repository** (step 44,334).
---
## Intended Use
**Direct use cases:**
- Multilingual text-to-speech with named speaker control
- Translation-conditioned speech synthesis (text in `tgt_lang`, speech reflecting `src_lang` semantics)
- Zero-shot voice cloning from a short reference clip
- Cross-lingual voice cloning when the reference and target languages differ
- Cascaded speech output paired with Hypa-Llama or other text models
**Downstream use:**
- Continued fine-tuning from the LoRA adapters in the companion repository
- Adapter stacking for domain- or language-specific heads
- Research on codec-token speech modeling and mixed-task TTS/VC training
## Out-of-Scope and Limitations
- **Not a general chat model.** Despite the Llama backbone, this checkpoint is trained for speech-token generation, not open-ended dialogue.
- **Prompt sensitivity.** Missing ``, duplicate BOS tokens, or omitted internal `` markers put inference off-distribution and can corrupt audio.
- **Context ceiling.** Training used 2,048 tokens (~25 s of speech). Longer outputs may degrade; voice-cloning rows with two clips hit the limit first.
- **Quality varies by language and speaker.** Tail languages and speakers with few training hours may produce less stable output.
- **No published perceptual benchmark yet.** Declining validation loss and informal listening support a stable fine-tune, but intelligibility, naturalness, and cloning fidelity require formal evaluation before strong product claims.
- **SNAC dependency.** Inference requires the SNAC codec (`hubertsiuzdak/snac_24khz`) for encode/decode around the language model.
## Bias, Risks, and Limitations
This model inherits biases from its base Orpheus weights and from Hypa-Voices, which is weighted toward AfroVoices studio material, English/French pivot translation routes, and adapted public speech. Generated speech should be reviewed for cultural appropriateness in customer-facing applications, especially for underrepresented languages with thinner training coverage.
Do not use this model unsupervised for applications affecting people's rights, health, finances, or wellbeing. Like all generative models, it can produce confident-sounding but incorrect speech.
---
## Released Artifacts
- **Merged 16-bit model (this repo):** [`hypaai/Hypa-Orpheus-3b-TTS-VC`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC)
- **LoRA adapters + TensorBoard:** [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs)
- **Base model:** [`unsloth/orpheus-3b-0.1-ft`](https://huggingface.co/unsloth/orpheus-3b-0.1-ft)
- **Training dataset (public subset):** [Hypa-Voices collection](https://huggingface.co/collections/hypaai/hypa-voices)
- **Dataset analysis:** [Hypa-Voices statistical breakdown](https://hypaintelligence.com/updates/hypa-voices-analysis-of-the-leading-multilingual-low-resource-dataset)
- **Upstream projects:** [Orpheus TTS](https://github.com/canopyai/Orpheus-TTS), [SNAC](https://github.com/hubertsiuzdak/snac), [Unsloth](https://github.com/unslothai/unsloth)
- **Related text model:** [Hypa-Llama collection](https://huggingface.co/collections/hypaai/hypa-llama3)
For notebooks, serving code, or contributions, contact **chris@hypaintelligence.com** or open an issue on Hugging Face.
---
## Citation
If you use Hypa-Orpheus 3B or any of the related work, please cite:
```bibtex
@misc{hypaorpheus2026,
title = {Hypa-Orpheus 3B: Multilingual Text-to-Speech and Voice Cloning for Underrepresented Languages},
author = {{Hypa Intelligence and AfroVoices}},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC}},
note = {Apache 2.0 License. Collection: \url{https://huggingface.co/collections/hypaai/hypa-orpheus}}
}
```
If you use the Hypa-Voices training corpus, please also cite:
```bibtex
@misc{hypavoices2026,
title={Hypa-Voices: Multilingual Low-Resource Audio-Text Dataset},
author={Hypa Intelligence and AfroVoices},
year={2026},
howpublished={Hugging Face collection},
url={https://huggingface.co/collections/hypaai/hypa-voices}
}
```
And the SNAC codec:
```bibtex
@inproceedings{siuzdak2024snac,
title={SNAC: Multi-Scale Neural Audio Codec},
author={Siuzdak, Hubert and Gr{\"o}tschla, Florian and Lanzend{\"o}rfer, Luca A.},
booktitle={Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation},
year={2024}
}
```
---
## License
Released under the **Apache License 2.0**. See the `LICENSE` file in this repository for full details.
---
## Acknowledgments
- **Canopy Labs**, for **Orpheus TTS** and for treating speech generation as language modeling over discrete audio tokens.
- **Meta AI / FAIR**, for the **Llama** family and the open-weights ecosystem Orpheus builds on.
- **Hubert Siuzdak** and collaborators, for **SNAC** and for open-sourcing the codec.
- **Unsloth**, for efficient **4-bit QLoRA** and Orpheus integration.
- **RunPod**, for reliable compute during the production fine-tune.
- **Mozilla** and the **Common Voice** community, for open speech data used in parts of Hypa-Voices.
- **AfroVoices**, for the steadfast curation of high-quality recordings across underrepresented languages.
- **Hypa Intelligence Research (HaIR)**, for our commitment to breaking the language barrier.
---
## Contact and Contributions
For questions, issues, or contributions, please open an issue in this repository or contact **chris@hypaintelligence.com**. Contributions are welcome.
---
## Closing Remarks
By releasing Hypa-Orpheus, we hope to give researchers and developers practical access to multilingual speech synthesis and voice cloning for underrepresented languages, and to make the data path, training recipe, and checkpoint-selection process transparent from the start.
At **Hypa Intelligence**, we believe that for AI to be truly aligned with humanity, it must understand and represent all of us, not just a select few. The first step toward this goal is solving the challenge of multilingualism by **breaking the language barrier**.
Hypa Intelligence remains steadfast in its mission to accelerate the advent of AGI and ASI and to ensure their benefits are globally distributed.
**AfroVoices**, a subsidiary of Hypa AI, is dedicated to amplifying African voices, languages, and cultures in the intelligence age. Focused on bridging the digital representation gap, AfroVoices curates datasets and resources for African languages, promoting inclusivity and cultural appreciation in AI technologies.
---
**Hypa Intelligence** • [Website](https://hypaintelligence.com) • [Hugging Face](https://huggingface.co/hypaai) • [Updates](https://hypaintelligence.com/updates)
*Multilingualism is not a feature. It is a prerequisite for AI that represents all of us.*
]