222 lines
6.9 KiB
Markdown
222 lines
6.9 KiB
Markdown
---
|
|
datasets:
|
|
- homebrewltd/instruction-speech-whispervq-v2
|
|
language:
|
|
- en
|
|
license: apache-2.0
|
|
tags:
|
|
- sound language model
|
|
pipeline_tag: audio-text-to-text
|
|
---
|
|
|
|
[](https://github.com/homebrewltd/ichigo/stargazers)
|
|
|
|
## Model Details
|
|
|
|
We have developed and released the family [Ichigo-llama3s](https://huggingface.co/collections/homebrew-research/llama3-s-669df2139f0576abc6eb7405). This family is natively understanding audio and text input.
|
|
|
|
We expand the Semantic tokens experiment with WhisperVQ as a tokenizer for audio files from [homebrewltd/mini-Ichigo-llama3.2-3B-s-base](https://huggingface.co/homebrewltd/mini-Ichigo-llama3.2-3B-s-base) with nearly 1B tokens from [Instruction Speech WhisperVQ v3](homebrewltd/mixed-instruction-speech-whispervq-v3-full) dataset.
|
|
|
|
**Model developers** Homebrew Research.
|
|
|
|
**Input** Text and sound.
|
|
|
|
**Output** Text.
|
|
|
|
**Model Architecture** Llama-3.
|
|
|
|
**Language(s):** English.
|
|
|
|
## Intended Use
|
|
|
|
**Intended Use Cases** This family is primarily intended for research applications. This version aims to further improve the LLM on sound understanding capabilities.
|
|
|
|
**Out-of-scope** The use of llama3-s in any manner that violates applicable laws or regulations is strictly prohibited.
|
|
|
|
## How to Get Started with the Model
|
|
|
|
Try this model using [Google Colab Notebook](https://colab.research.google.com/drive/18IiwN0AzBZaox5o0iidXqWD1xKq11XbZ?usp=sharing).
|
|
|
|
First, we need to convert the audio file to sound tokens
|
|
|
|
```python
|
|
device = "cuda" if torch.cuda.is_available() else "cpu"
|
|
if not os.path.exists("whisper-vq-stoks-medium-en+pl-fixed.model"):
|
|
hf_hub_download(
|
|
repo_id="jan-hq/WhisperVQ",
|
|
filename="whisper-vq-stoks-medium-en+pl-fixed.model",
|
|
local_dir=".",
|
|
)
|
|
vq_model = RQBottleneckTransformer.load_model(
|
|
"whisper-vq-stoks-medium-en+pl-fixed.model"
|
|
).to(device)
|
|
vq_model.ensure_whisper(device)
|
|
def audio_to_sound_tokens(audio_path, target_bandwidth=1.5, device=device):
|
|
|
|
wav, sr = torchaudio.load(audio_path)
|
|
if sr != 16000:
|
|
wav = torchaudio.functional.resample(wav, sr, 16000)
|
|
with torch.no_grad():
|
|
codes = vq_model.encode_audio(wav.to(device))
|
|
codes = codes[0].cpu().tolist()
|
|
|
|
result = ''.join(f'<|sound_{num:04d}|>' for num in codes)
|
|
return f'<|sound_start|>{result}<|sound_end|>'
|
|
```
|
|
|
|
Then, we can inference the model the same as any other LLM.
|
|
|
|
```python
|
|
def setup_pipeline(model_path, use_4bit=False, use_8bit=False):
|
|
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
|
|
|
model_kwargs = {"device_map": "auto"}
|
|
|
|
if use_4bit:
|
|
model_kwargs["quantization_config"] = BitsAndBytesConfig(
|
|
load_in_4bit=True,
|
|
bnb_4bit_compute_dtype=torch.bfloat16,
|
|
bnb_4bit_use_double_quant=True,
|
|
bnb_4bit_quant_type="nf4",
|
|
)
|
|
elif use_8bit:
|
|
model_kwargs["quantization_config"] = BitsAndBytesConfig(
|
|
load_in_8bit=True,
|
|
bnb_8bit_compute_dtype=torch.bfloat16,
|
|
bnb_8bit_use_double_quant=True,
|
|
)
|
|
else:
|
|
model_kwargs["torch_dtype"] = torch.bfloat16
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(model_path, **model_kwargs)
|
|
|
|
return pipeline("text-generation", model=model, tokenizer=tokenizer)
|
|
|
|
def generate_text(pipe, messages, max_new_tokens=64, temperature=0.0, do_sample=False):
|
|
generation_args = {
|
|
"max_new_tokens": max_new_tokens,
|
|
"return_full_text": False,
|
|
"temperature": temperature,
|
|
"do_sample": do_sample,
|
|
}
|
|
|
|
output = pipe(messages, **generation_args)
|
|
return output[0]['generated_text']
|
|
|
|
# Usage
|
|
llm_path = "homebrewltd/llama3.1-s-instruct-v0.2"
|
|
pipe = setup_pipeline(llm_path, use_8bit=True)
|
|
```
|
|
|
|
## Training process
|
|
**Training Metrics Image**: Below is a snapshot of the training loss curve visualized.
|
|
|
|
|
|
|
|
**[MMLU](https://huggingface.co/datasets/cais/mmlu)**:
|
|
|
|
| Model | MMLU Score |
|
|
| --- | --- |
|
|
| llama3.1-instruct-8b | 69.40 |
|
|
| ichigo-llama3.1-s-v0.3: phase 3 | 63.79 |
|
|
| ichigo-llama3.1-s-v0.3: phase 2 | 63.08 |
|
|
| ichigo-llama3.1-s-base-v0.3 | 42.11 |
|
|
| mini-ichigo-llama3.2-3B-s-instruct | **58.60** |
|
|
| mini-ichigo-llama3.2-3B-s-base | 59.61 |
|
|
| llama3.1-s-instruct-v0.2 | 50.27 |
|
|
|
|
|
|
**[AudioBench](https://arxiv.org/abs/2406.16020) Eval**:
|
|
|
|
| Model Bench | [Open-hermes Instruction Audio](https://huggingface.co/datasets/AudioLLMs/openhermes_instruction_test) (GPT-4-O judge 0:5) | [Alpaca Instruction Audio](https://huggingface.co/datasets/AudioLLMs/alpaca_audio_test) (GPT-4-O judge 0:5) |
|
|
| --- | --- | --- |
|
|
| [Llama3.1-s-v2](https://huggingface.co/homebrewltd/llama3-s-instruct-v0.2) | 3.45 | 3.53 |
|
|
| [Ichigo-llama3.1-s v0.3-phase2 -cp7000](https://huggingface.co/homebrewltd/Ichigo-llama3.1-s-instruct-v0.3-phase-2) | 3.42 | 3.62 |
|
|
| [Ichigo-llama3.1-s v0.3-phase2-cplast](https://huggingface.co/jan-hq/llama3-s-instruct-v0.3-checkpoint-last) | 3.31 | 3.6 |
|
|
| [Ichigo-llama3.1-s v0.3-phase3](https://huggingface.co/homebrewltd/Ichigo-llama3.1-s-instruct-v0.3-phase-3) | 3.64 | 3.68 |
|
|
| [mini-Ichigo-llama3.2-3B-s-instruct](https://huggingface.co/homebrewltd/mini-Ichigo-llama3.2-3B-s-instruct) | **2.58** | **2.07** |
|
|
| [Qwen2-audio-7B](https://huggingface.co/Qwen/Qwen2-Audio-7B) | 2.63 | 2.24 |
|
|
|
|
### Hardware
|
|
|
|
**GPU Configuration**: Cluster of 10x NVIDIA A6000-48GB.
|
|
|
|
**GPU Usage**:
|
|
- **Fine-tuning**: 12 hours.
|
|
|
|
### Training Arguments
|
|
|
|
We utilize [torchtune](https://github.com/pytorch/torchtune) library for the latest FSDP2 training code implementation.
|
|
|
|
| Parameter | Instruction Fine-tuning |
|
|
|----------------------------|-------------------------|
|
|
| **Epoch** | 1 |
|
|
| **Global batch size** | 360 |
|
|
| **Learning Rate** | 7e-5 |
|
|
| **Learning Scheduler** | LambdaLR with warmup |
|
|
| **Optimizer** | Adam torch fused |
|
|
| **Warmup Ratio** | 0.01 |
|
|
| **Weight Decay** | 0.005 |
|
|
| **Max Sequence Length** | 4096 |
|
|
|
|
|
|
## Examples
|
|
|
|
1. Good example:
|
|
|
|
<details>
|
|
<summary>Click to toggle Example 1</summary>
|
|
|
|
```
|
|
|
|
```
|
|
</details>
|
|
|
|
<details>
|
|
<summary>Click to toggle Example 2</summary>
|
|
|
|
```
|
|
|
|
```
|
|
</details>
|
|
|
|
|
|
2. Misunderstanding example:
|
|
|
|
<details>
|
|
<summary>Click to toggle Example 3</summary>
|
|
|
|
```
|
|
|
|
```
|
|
</details>
|
|
|
|
3. Off-tracked example:
|
|
|
|
<details>
|
|
<summary>Click to toggle Example 4</summary>
|
|
|
|
```
|
|
|
|
```
|
|
</details>
|
|
|
|
|
|
## Citation Information
|
|
|
|
**BibTeX:**
|
|
|
|
```
|
|
@article{Llama3-S: Sound Instruction Language Model 2024,
|
|
title={Llama3-S},
|
|
author={Homebrew Research},
|
|
year=2024,
|
|
month=August},
|
|
url={https://huggingface.co/homebrewltd/llama3.1-s-2024-08-20}
|
|
```
|
|
|
|
## Acknowledgement
|
|
|
|
- **[WhisperSpeech](https://github.com/collabora/WhisperSpeech)**
|
|
|
|
- **[Meta-Llama-3.1-8B-Instruct ](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)** |