初始化项目，由ModelHub XC社区提供模型
Model: ryota-komatsu/SylReg-LM-7B-Instruct Source: Original Platform
This commit is contained in:
ModelHub XC
143
README.md
Normal file
143
README.md
Normal file
@@ -0,0 +1,143 @@
|
||||
---
|
||||
library_name: transformers
|
||||
license: cc-by-nc-sa-4.0
|
||||
datasets:
|
||||
- ryota-komatsu/SylReg
|
||||
language:
|
||||
- en
|
||||
base_model:
|
||||
- ryota-komatsu/SylReg-LM-7B
|
||||
---
|
||||
|
||||
# SylReg-LM 7B Instruct
|
||||
|
||||
<!-- Provide a quick summary of what the model is/does. -->
|
||||
|
||||
|
||||
|
||||
## Model Details
|
||||
|
||||
### Model Description
|
||||
|
||||
<!-- Provide a longer summary of what this model is. -->
|
||||
|
||||
- **Model type:** Qwen2ForCausalLM
|
||||
- **Language(s) (NLP):** English
|
||||
- **License:** CC BY-NC-SA 4.0
|
||||
- **Finetuned from model:** [ryota-komatsu/SylReg-LM-7B](https://huggingface.co/ryota-komatsu/SylReg-LM-7B)
|
||||
|
||||
### Model Sources
|
||||
|
||||
<!-- Provide the basic links for the model. -->
|
||||
|
||||
- **Repository:** [Code](https://github.com/ryota-komatsu/speaker_disentangled_hubert)
|
||||
- **Paper:** [arXiv:2607.04064](https://arxiv.org/abs/2607.04064)
|
||||
- **Demo:** [Project page](https://ryota-komatsu.github.io/speaker_disentangled_hubert)
|
||||
|
||||
## How to Get Started with the Model
|
||||
|
||||
Use the code below to get started with the model.
|
||||
|
||||
```sh
|
||||
git clone https://github.com/ryota-komatsu/speaker_disentangled_hubert.git
|
||||
cd speaker_disentangled_hubert
|
||||
|
||||
sudo apt install git-lfs # for UTMOS
|
||||
|
||||
conda create -y -n py310 -c pytorch -c nvidia -c conda-forge python=3.10 pip=24.0 setuptools=81.0.0 faiss-gpu=1.13.2
|
||||
conda activate py310
|
||||
pip install -r requirements/requirements.txt
|
||||
|
||||
sh scripts/setup.sh
|
||||
```
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
import torch
|
||||
import torchaudio
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
from src.flow_matching import FlowMatchingWithBigVGan
|
||||
from src.s5hubert import SylRegForSyllableDiscovery
|
||||
|
||||
wav_path = "/path/to/wav"
|
||||
|
||||
# download pretrained models from hugging face hub
|
||||
encoder = SylRegForSyllableDiscovery.from_pretrained("ryota-komatsu/SylReg-Distill", device_map="cuda", dtype="auto")
|
||||
decoder = FlowMatchingWithBigVGan.from_pretrained("ryota-komatsu/SylReg-Decoder", device_map="cuda", dtype="auto")
|
||||
speechlm = AutoModelForCausalLM.from_pretrained("ryota-komatsu/SylReg-LM-7B-Instruct", device_map="cuda", dtype="auto")
|
||||
tokenizer = AutoTokenizer.from_pretrained("ryota-komatsu/SylReg-LM-7B-Instruct")
|
||||
|
||||
# load a waveform
|
||||
waveform, sr = torchaudio.load(wav_path)
|
||||
waveform = torchaudio.functional.resample(waveform, sr, 16000)
|
||||
|
||||
# encode a waveform into syllabic units
|
||||
outputs = encoder(waveform.to(encoder.device))
|
||||
units = outputs[0]["units"] # [3950, 67, ..., 503]
|
||||
|
||||
# speech language modeling
|
||||
messages = [
|
||||
{"role": "user", "content": "".join(f"<{unit}>" for unit in units)},
|
||||
]
|
||||
|
||||
input_ids = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
return_tensors="pt",
|
||||
).input_ids.to(speechlm.device)
|
||||
|
||||
generated_ids = speechlm.generate(input_ids=input_ids, do_sample=True, temperature=0.8)[0]
|
||||
|
||||
units = tokenizer.decode(generated_ids)
|
||||
units = torch.tensor([int(unit) for unit in re.findall(r"<(\d+)>", units)], device=decoder.device)
|
||||
|
||||
# unit-to-speech synthesis
|
||||
generated_speech = decoder(units.unsqueeze(0)).waveform.cpu()
|
||||
```
|
||||
|
||||
## Training Details
|
||||
|
||||
### Training Data
|
||||
|
||||
| | Hours | License | Provider |
|
||||
| --- | --- | --- | --- |
|
||||
| [LibriSpeech](https://huggingface.co/datasets/openslr/librispeech_asr) | 960 | CC BY 4.0 | V. Panayotov *et al.* |
|
||||
| [Libriheavy](https://huggingface.co/datasets/pkufool/libriheavy) | 50,978 | public domain | W. Kang *et al.* |
|
||||
| [Emilia-Large](https://huggingface.co/datasets/amphion/Emilia-Dataset) | 4,447 | CC BY 4.0, CC BY-NC 4.0 | H. He *et al.* |
|
||||
| [People's Speech (clean, clean_sa)](https://huggingface.co/datasets/MLCommons/peoples_speech) | 5,640 | CC-BY, CC-BY-SA | D. Galvez *et al.* |
|
||||
| [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) | 543 | CC0-1.0 | C. Wang *et al.* |
|
||||
| [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) | 27,810 | cdla-sharing-1.0 | R. Eldan *et al.* |
|
||||
| [Cosmopedia-v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | 38,986 | odc-by | L. B. Allal *et al.* |
|
||||
| Total | 129,364 | | |
|
||||
|
||||
### Training Hyperparameters
|
||||
|
||||
- **Training regime:** bf16 mixed precision
|
||||
- **Training steps:** 15k
|
||||
- **Batch size:** 2,097,152 (=2<sup>21</sup>) tokens
|
||||
- **Optimizer:** AdamW(lr=0.0003, betas=(0.9, 0.95), weight_decay=0.01)
|
||||
- **Scheduler:** warmup_stable_decay(num_warmup_steps=100, num_decay_steps=5000, min_lr_ratio=0.1)
|
||||
|
||||
## Hardware
|
||||
|
||||
32 NVIDIA H100 GPUs
|
||||
|
||||
## Citation
|
||||
|
||||
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
|
||||
|
||||
**BibTeX:**
|
||||
|
||||
```bibtex
|
||||
@article{Komatsu_SylReg_2026,
|
||||
author = {Komatsu, Ryota and Kawakita, Kota and Okamoto, Takuma and Shinozaki, Takahiro},
|
||||
title = {Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization},
|
||||
year = {2026},
|
||||
volume = {7},
|
||||
journal = {IEEE Open Journal of Signal Processing},
|
||||
pages = {},
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user