115 lines
4.3 KiB
Markdown
115 lines
4.3 KiB
Markdown
---
|
|
library_name: transformers
|
|
datasets:
|
|
- malaysia-ai/Multilingual-TTS
|
|
- malaysia-ai/Emilia-YODAS-Voice-Conversion
|
|
- mesolitica/Malaysian-Emilia-v2
|
|
base_model:
|
|
- Qwen/Qwen3-1.7B-Base
|
|
new_version: Scicom-intl/Multilingual-TTS-1.7B-Base
|
|
---
|
|
|
|
# Qwen3-1.7B-Multilingual-TTS
|
|
|
|
Continue pretraining [Qwen/Qwen3-1.7B-Base](https://huggingface.co/Qwen/Qwen3-1.7B-Base) on Multilingual Voice Conversion and TTS.
|
|
|
|
1. Use [neucodec](https://huggingface.co/neuphonic/neucodec) as speech detokenizer, 50 TPS, output in 24k sample rate.
|
|
2. Multi-speaker multilingual Voice Conversion **up to 25.5B tokens**.
|
|
3. Multi-speaker multilingual TTS up to **5B tokens**.
|
|
4. Flash Attention 3 10k context length multipacking.
|
|
5. Liger Kernel for `swiglu`, `rms_norm` and `fused_linear_cross_entropy`.
|
|
|
|
WanDB at https://wandb.ai/huseinzol05/Qwen-Qwen3-1.7B-Base-multilingual-tts-neucodec
|
|
|
|
**We released better version at [Scicom-intl/Multilingual-TTS-1.7B-Base](https://huggingface.co/Scicom-intl/Multilingual-TTS-1.7B-Base)**.
|
|
|
|
## How to
|
|
|
|
```python
|
|
import soundfile as sf
|
|
import torch
|
|
import torchaudio
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
from neucodec import NeuCodec
|
|
import re
|
|
|
|
model_name = "malaysia-ai/Qwen3-1.7B-Multilingual-TTS"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto").to('cuda')
|
|
codec = NeuCodec.from_pretrained("neuphonic/neucodec")
|
|
_ = codec.eval().to('cuda')
|
|
```
|
|
|
|
## TTS
|
|
|
|
```python
|
|
text = "Hello! how come I help you? 你好！有什么可以帮你的吗？வணக்கம்! நான் உங்களுக்கு எப்படி உதவுவது? Bonjour! Comment puis-je vous aider ? Xin chào! Tôi có thể giúp gì cho bạn? こんにちは！どうしてお手伝いしましょうか？안녕하세요! 어떻게 도와드릴까요?"
|
|
prompt = f"<|im_start|>jenny_tts_dataset_audio_jenny: {text}<|speech_start|>"
|
|
|
|
inputs = tokenizer(prompt,return_tensors="pt", add_special_tokens=True).to('cuda')
|
|
|
|
with torch.no_grad():
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=2048,
|
|
do_sample=True,
|
|
temperature=0.6,
|
|
repetition_penalty=1.15,
|
|
)
|
|
|
|
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=False)
|
|
audio_tokens = re.findall(r'<\|s_(\d+)\|>', generated_text.split('<|speech_start|>')[1])
|
|
audio_tokens = [int(token) for token in audio_tokens]
|
|
audio_codes = torch.tensor(audio_tokens)[None, None]
|
|
|
|
with torch.no_grad():
|
|
audio_waveform = codec.decode_code(audio_codes.cuda())
|
|
|
|
sf.write('7-languages.mp3', audio_waveform[0, 0].cpu(), 24000)
|
|
```
|
|
|
|
You can check the audio [7-languages.mp3](7-languages.mp3).
|
|
|
|
- You can pick any speaker name from [malaysia-ai/Multilingual-TTS](malaysia-ai/Multilingual-TTS).
|
|
- **Not bad from 0.35 epoch model**.
|
|
|
|
### Voice Conversion
|
|
|
|
```python
|
|
import librosa
|
|
|
|
y, sr = librosa.load('jenny.wav', sr = 16000)
|
|
with torch.no_grad():
|
|
codes = codec.encode_code(torch.tensor(y)[None, None])
|
|
tokens = ''.join([f'<|s_{i}|>' for i in codes[0, 0]])
|
|
prompt = f"<|im_start|>I wonder if I shall ever be happy enough to have real lace on my clothes and bows on my caps.<|speech_start|>{tokens}<|im_end|><|im_start|>Hello, how come I help you, 你好, 有什么可以帮你的吗, வணக்கம், நான் உங்களுக்கு எப்படி உதவுவது, bonjour, comment puis-je vous aider.<|speech_start|>"
|
|
|
|
inputs = tokenizer(prompt,return_tensors="pt", add_special_tokens=True).to('cuda')
|
|
|
|
with torch.no_grad():
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens=2048,
|
|
do_sample=True,
|
|
temperature=0.6,
|
|
repetition_penalty=1.15,
|
|
)
|
|
|
|
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=False)
|
|
audio_tokens = re.findall(r'<\|s_(\d+)\|>', generated_text.split('<|speech_start|>')[-1])
|
|
audio_tokens = [int(token) for token in audio_tokens]
|
|
audio_codes = torch.tensor(audio_tokens)[None, None]
|
|
|
|
with torch.no_grad():
|
|
audio_waveform = codec.decode_code(audio_codes.cuda())
|
|
|
|
sf.write('jenny-4-languages.mp3', audio_waveform[0, 0].cpu(), 24000)
|
|
```
|
|
|
|
You can check the audio [jenny-4-languages.mp3](jenny-4-languages.mp3).
|
|
|
|
- **Not too great, we need to trim the silents first before convert to audio tokens, the model tends to generate long silents**.
|
|
|
|
## Source code
|
|
|
|
Source code at https://github.com/malaysia-ai/cooking/tree/main/qwen-tts |