初始化项目，由ModelHub XC社区提供模型
Model: voidful/llm-codec Source: Original Platform
This commit is contained in:
ModelHub XC
292
README.md
Normal file
292
README.md
Normal file
@@ -0,0 +1,292 @@
|
||||
---
|
||||
base_model:
|
||||
- Qwen/Qwen3-4B-Instruct-2507
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- audio
|
||||
- speech
|
||||
- audio-codec
|
||||
- neural-audio-codec
|
||||
- spoken-language-modeling
|
||||
- codec-superb
|
||||
- qwen3
|
||||
datasets:
|
||||
- librispeech_asr
|
||||
metrics:
|
||||
- perplexity
|
||||
- pesq
|
||||
- stoi
|
||||
---
|
||||
|
||||
# LLM-Codec
|
||||
|
||||
LLM-Codec is a neural audio codec checkpoint trained to produce discrete audio
|
||||
tokens that are both reconstructable and easier for autoregressive language
|
||||
models to predict.
|
||||
|
||||
Model: https://huggingface.co/voidful/llm-codec
|
||||
|
||||
Code: https://github.com/voidful/llm-codec
|
||||
|
||||
Usage reference: https://github.com/voidful/Codec-SUPERB
|
||||
|
||||
## Model Description
|
||||
|
||||
Most neural audio codecs are trained for waveform reconstruction. Spoken
|
||||
language models, however, consume codec tokens with a next-token prediction
|
||||
objective. This mismatch can make acoustically valid variation appear as token
|
||||
uncertainty to the language model.
|
||||
|
||||
LLM-Codec adapts a codec with language-model-facing objectives while keeping the
|
||||
deployed codec interface unchanged. The model is trained with:
|
||||
|
||||
- Future Token Prediction (FTP): Medusa-style heads predict future audio tokens
|
||||
from frozen-LLM hidden states.
|
||||
- Semantic Alignment (SA): audio-induced hidden states are aligned with paired
|
||||
text hidden states inside a frozen LLM.
|
||||
- Differentiable Gumbel bridge: hard Gumbel-Softmax keeps discrete forward
|
||||
tokens while enabling gradients to flow to the codec encoder.
|
||||
- Reconstruction losses: mel, multi-scale mel, multi-resolution STFT, complex
|
||||
STFT, VQ, GAN, and feature matching losses.
|
||||
|
||||
The deployed codec does not require the auxiliary FTP heads.
|
||||
|
||||
## Intended Use
|
||||
|
||||
This model is intended for research and development in:
|
||||
|
||||
- audio tokenization for spoken language modeling
|
||||
- codec reconstruction experiments
|
||||
- token-level speech LM training
|
||||
- Codec-SUPERB style codec evaluation
|
||||
- speech token analysis and ablation studies
|
||||
|
||||
It is not a full text-to-speech system by itself. For speech generation, use the
|
||||
codec as the tokenizer/decoder inside a separate speech language modeling
|
||||
pipeline.
|
||||
|
||||
## Out-of-Scope Use
|
||||
|
||||
Do not use this model for:
|
||||
|
||||
- impersonation or unauthorized voice cloning
|
||||
- surveillance or speaker tracking without consent
|
||||
- high-stakes speaker, language, or identity decisions
|
||||
- generating deceptive audio content
|
||||
|
||||
## Installation
|
||||
|
||||
The easiest inference path is through the Codec-SUPERB `SoundCodec` interface.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/voidful/Codec-SUPERB.git
|
||||
cd Codec-SUPERB
|
||||
pip install -r requirements.txt
|
||||
export PYTHONPATH=$PWD:$PYTHONPATH
|
||||
```
|
||||
|
||||
If your environment supports editable installs, this is also convenient:
|
||||
|
||||
```bash
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
Load LLM-Codec through the Codec-SUPERB codec registry:
|
||||
|
||||
```python
|
||||
from SoundCodec import codec
|
||||
|
||||
print(codec.list_codec())
|
||||
model = codec.load_codec("llmcodec")
|
||||
```
|
||||
|
||||
Encode and reconstruct one audio file:
|
||||
|
||||
```python
|
||||
from SoundCodec import codec
|
||||
import torchaudio
|
||||
import soundfile as sf
|
||||
|
||||
model = codec.load_codec("llmcodec")
|
||||
|
||||
waveform, sample_rate = torchaudio.load("sample_audio.wav")
|
||||
data_item = {
|
||||
"audio": {
|
||||
"array": waveform.numpy()[0],
|
||||
"sampling_rate": sample_rate,
|
||||
}
|
||||
}
|
||||
|
||||
units = model.extract_unit(data_item).unit
|
||||
print("Unit shape:", units.shape)
|
||||
|
||||
result = model.synth(data_item, local_save=False)
|
||||
reconstructed = result["audio"]["array"]
|
||||
reconstructed_sr = result["audio"].get("sampling_rate", sample_rate)
|
||||
|
||||
sf.write("reconstructed.wav", reconstructed, reconstructed_sr)
|
||||
```
|
||||
|
||||
## Batch Usage
|
||||
|
||||
Codec-SUPERB also provides batch APIs:
|
||||
|
||||
```python
|
||||
from SoundCodec import codec
|
||||
import torchaudio
|
||||
|
||||
model = codec.load_codec("llmcodec")
|
||||
|
||||
audio_files = ["audio1.wav", "audio2.wav", "audio3.wav"]
|
||||
data_list = []
|
||||
|
||||
for path in audio_files:
|
||||
waveform, sample_rate = torchaudio.load(path)
|
||||
data_list.append({
|
||||
"id": path,
|
||||
"audio": {
|
||||
"array": waveform.numpy()[0],
|
||||
"sampling_rate": sample_rate,
|
||||
},
|
||||
})
|
||||
|
||||
batch_units = model.batch_extract_unit(data_list)
|
||||
batch_audio = model.batch_decode_unit(batch_units)
|
||||
|
||||
results = model.batch_synth(data_list, local_save=False)
|
||||
for item in results:
|
||||
print(item["unit"].shape, item["audio"]["array"].shape)
|
||||
```
|
||||
|
||||
For better throughput, group audio samples with similar lengths before batching.
|
||||
|
||||
## Codec-SUPERB Evaluation
|
||||
|
||||
To evaluate LLM-Codec with Codec-SUPERB-tiny:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. python3 scripts/dataset_creator.py \
|
||||
--dataset voidful/codec-superb-tiny
|
||||
|
||||
PYTHONPATH=. python3 scripts/benchmarking.py \
|
||||
--dataset datasets/voidful/codec-superb-tiny_synth \
|
||||
--models llmcodec
|
||||
```
|
||||
|
||||
## Model Files
|
||||
|
||||
The model repository provides:
|
||||
|
||||
- codec weights as `llm-codec.pt`
|
||||
- a tokenizer extended with `<CODEC_*>` audio tokens
|
||||
- Qwen-compatible model artifacts containing trained audio-token embeddings
|
||||
|
||||
The codec uses 20,480 audio tokens with the canonical token format:
|
||||
|
||||
```text
|
||||
<CODEC_0>, <CODEC_1>, ..., <CODEC_20479>
|
||||
```
|
||||
|
||||
## Training Data
|
||||
|
||||
The codec was trained on LibriSpeech `train-clean-100` with paired transcripts.
|
||||
The validation split used during training is LibriSpeech `validation`.
|
||||
|
||||
Because training is speech-centric and transcript-supervised, performance may be
|
||||
weaker on non-English speech, conversational speech, music, environmental audio,
|
||||
or audio with strong noise and overlap.
|
||||
|
||||
## Training Procedure
|
||||
|
||||
Base components:
|
||||
|
||||
- Base codec: AUV
|
||||
- Frozen LLM backbone: Qwen3-4B-Instruct
|
||||
- Token rate: 50 Hz
|
||||
- Audio vocabulary size: 20,480
|
||||
- Segment length: 4 seconds
|
||||
|
||||
Losses:
|
||||
|
||||
- reconstruction mel loss
|
||||
- multi-scale mel loss
|
||||
- multi-resolution STFT loss
|
||||
- complex STFT loss with phase term
|
||||
- VQ commitment loss
|
||||
- Gumbel bridge cross entropy
|
||||
- Future Token Prediction loss
|
||||
- Semantic Alignment cosine loss
|
||||
- Semantic Alignment contrastive loss with memory bank
|
||||
- MPD/MSD GAN and feature matching losses
|
||||
|
||||
## Evaluation Results
|
||||
|
||||
### Token Learnability
|
||||
|
||||
SALMon speech coherence accuracy after token-level LM training:
|
||||
|
||||
| Tokenizer | Overall accuracy |
|
||||
| --- | ---: |
|
||||
| WavTok-L | 48.3 |
|
||||
| BigCodec | 49.4 |
|
||||
| UniCodec | 50.1 |
|
||||
| AUV | 49.4 |
|
||||
| LLM-Codec | 61.6 |
|
||||
|
||||
Token-level perplexity on LibriSpeech after 3 epochs of LM training:
|
||||
|
||||
| Tokenizer | Eval loss | Perplexity |
|
||||
| --- | ---: | ---: |
|
||||
| WavTok-L | 11.91 | 148,122 |
|
||||
| UniCodec | 11.92 | 150,197 |
|
||||
| BigCodec | 11.96 | 156,448 |
|
||||
| AUV | 11.98 | 159,768 |
|
||||
| LLM-Codec | 8.44 | 4,617 |
|
||||
|
||||
### Reconstruction Quality
|
||||
|
||||
Codec-SUPERB-tiny speech reconstruction:
|
||||
|
||||
| Model | Mel lower is better | STFT lower is better | PESQ higher is better | STOI higher is better |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| AUV base | 0.762 | 1.648 | 2.094 | 0.850 |
|
||||
| LLM-Codec | 0.724 | 1.599 | 2.102 | 0.859 |
|
||||
|
||||
## Limitations
|
||||
|
||||
- The semantic alignment objective depends on paired speech and text.
|
||||
- The model is primarily validated on read speech.
|
||||
- Downstream generation quality depends on the separate speech language model.
|
||||
- The model may preserve speaker identity information present in the input.
|
||||
- The Hugging Face `transformers` artifacts are not a standalone text chatbot;
|
||||
they accompany the codec/tokenizer workflow.
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@article{chung2026llm,
|
||||
title={LLM-Codec: Neural Audio Codec Meets Language Model Objectives},
|
||||
author={Chung, Ho-Lam and Chen, Yiming and Lee, Hung-yi},
|
||||
journal={arXiv preprint arXiv:2604.17852},
|
||||
note = {Model and code available at https://github.com/voidful/llm-codec},
|
||||
year={2026}
|
||||
}
|
||||
```
|
||||
|
||||
If you use the Codec-SUPERB interface or benchmark, please also cite
|
||||
Codec-SUPERB:
|
||||
|
||||
```bibtex
|
||||
@inproceedings{wu-etal-2024-codec,
|
||||
title = {Codec-SUPERB: An In-Depth Analysis of Sound Codec Models},
|
||||
author = {Wu, Haibin and Chung, Ho-Lam and Lin, Yi-Cheng and Wu, Yuan-Kuei and Chen, Xuanjun and Pai, Yu-Chi and Wang, Hsiu-Hsuan and Chang, Kai-Wei and Liu, Alexander and Lee, Hung-yi},
|
||||
booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
|
||||
year = {2024},
|
||||
url = {https://aclanthology.org/2024.findings-acl.616},
|
||||
doi = {10.18653/v1/2024.findings-acl.616},
|
||||
pages = {10330--10348}
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user