初始化项目，由ModelHub XC社区提供模型

Model: nineninesix/kani-tts-400m-es Source: Original Platform
2026-07-04 23:55:12 +08:00
commit fb4b8bee7e
10 changed files with 413 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,51 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*.tfevents* filter=lfs diff=lfs merge=lfs -text
+*.db* filter=lfs diff=lfs merge=lfs -text
+*.ark* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
+ 
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.gguf* filter=lfs diff=lfs merge=lfs -text
+*.ggml filter=lfs diff=lfs merge=lfs -text
+*.llamafile* filter=lfs diff=lfs merge=lfs -text
+*.pt2 filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text
+tokenizer_config.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,220 @@
+---
+license: other
+license_name: lfm1.0
+license_link: https://www.liquid.ai/lfm-license
+language:
+- es
+pipeline_tag: text-to-speech
+library_name: transformers
+base_model:
+- nineninesix/kani-tts-400m-0.3-pt
+---
+<p>
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/64fab67bd268b2f1ad8a826b/Ki5aExt7SmQwHYLuGyLdc.png" alt="Logo" width="200" height="200">
+</p>
+
+# KaniTTS Spanish
+
+[![](https://dcbadge.limes.pink/api/server/https://discord.gg/NzP3rjB4SB?style=flat)](https://discord.gg/NzP3rjB4SB) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+
+
+
+A high-speed, high-fidelity Text-to-Speech model optimized for real-time conversational AI applications.
+
+## Overview
+
+KaniTTS uses a two-stage pipeline combining a large language model with an efficient audio codec for exceptional speed and audio quality. The architecture generates compressed token representations through a backbone LLM, then rapidly synthesizes waveforms via neural audio codec, achieving extremely low latency.
+
+**Key Specifications:**
+- **Model Size:** 400M parameters
+- **Sample Rate:** 22kHz
+- **Language:** Spanish
+- **License:** Apache 2.0
+
+## Performance
+
+**On [NovitaAI](https://novita.ai/) RTX 5090 using vLLM:**
+- **RTF:** ~0.2 (5 times faster than realtime)
+- **Memory:** 16GB GPU VRAM used
+- **Source Code:** https://github.com/nineninesix-ai/kanitts-vllm
+
+#### GPU Benchmark Results
+
+| GPU Model | VRAM | Cost ($/hr) | RTF |
+|-----------|------|-------------|-----|
+| RTX 5090 | 32GB | $0.423 | 0.190 |
+| RTX 4080 | 16GB | $0.220 | 0.200 |
+| RTX 5060 Ti | 16GB | $0.138 | 0.529 |
+| RTX 4060 Ti | 16GB | $0.122 | 0.537 |
+| RTX 3060 | 12GB | $0.093 | 0.600 |
+
+*Lower RTF is better (< 1.0 means faster than real-time). Benchmarks conducted on [Vast AI](https://vast.ai/).*
+
+## Quickstart: Install from PyPI & Run Inference
+
+It’s a lightweight so you can install, load a model, and speak in minutes.
+Designed for quick starts and simple workflows—no heavy setup, just pip install and run.
+[More detailes...](https://pypi.org/project/kani-tts/)
+
+### Install
+
+```bash
+pip install kani-tts
+pip install -U "transformers==4.57.1" # for LFM2 !!!
+```
+### Quick Start
+
+```python
+from kani_tts import KaniTTS
+
+model = KaniTTS('nineninesix/kani-tts-400m-es')
+
+# Generate audio from text
+audio, text = model("¡Hola a todos! Qué bueno que estés aquí.")
+
+# Save to file (requires soundfile)
+model.save_audio(audio, "output.wav")
+```
+
+### Working with Multi-Speaker Models
+
+This model support multiple speakers. You can check if your model supports speakers and select a specific voice:
+
+```python
+from kani_tts import KaniTTS
+
+model = KaniTTS('nineninesix/kani-tts-400m-es')
+
+# Check if model supports multiple speakers
+print(f"Model type: {model.status}")  # 'singlspeaker' or 'multispeaker'
+
+# Display available speakers (pretty formatted)
+model.show_speakers()
+
+# Or access the speaker list directly
+print(model.speaker_list)
+
+# Generate audio with a specific speaker
+audio, text = model("¡Hola a todos! Qué bueno que estés aquí.", speaker_id="ash")
+```
+
+### Custom Configuration
+
+```python
+from kani_tts import KaniTTS
+
+model = KaniTTS(
+    'nineninesix/kani-tts-400m-es',
+    temperature=0.7,           # Control randomness (default: 1.0)
+    top_p=0.9,                 # Nucleus sampling (default: 0.95)
+    max_new_tokens=2000,       # Max audio length (default: 1200)
+    repetition_penalty=1.2,    # Prevent repetition (default: 1.1)
+    suppress_logs=True,        # Suppress library logs (default: True)
+    show_info=True,            # Show model info on init (default: True)
+)
+
+audio, text = model("Your text here")
+```
+
+### Playing Audio in Jupyter Notebooks
+
+You can listen to generated audio directly in Jupyter notebooks or IPython:
+
+```python
+from kani_tts import KaniTTS
+from IPython.display import Audio as aplay
+
+model = KaniTTS('nineninesix/kani-tts-400m-es')
+audio, text = model("¡Hola a todos! Qué bueno que estés aquí.")
+
+# Play audio in notebook
+aplay(audio, rate=model.sample_rate)
+```
+
+---
+
+## Datasets
+- https://huggingface.co/datasets/laion/Emolia
+- https://huggingface.co/datasets/sirekist98/spanish_Audios
+
+## Voices:
+- `nova`
+- `ballad`
+- `ash`
+
+## Audio Examples
+| Text | Audio |
+|---|---|
+| El viento sopla suave, como si también quisiera decir algo. | <audio controls src="https://cdn-uploads.huggingface.co/production/uploads/64fab67bd268b2f1ad8a826b/KPO9ZsTx2T8K2l4fP7Sqg.wav"></audio> |
+| A veces, una simple mirada dice más que mil palabras. | <audio controls src="https://cdn-uploads.huggingface.co/production/uploads/64fab67bd268b2f1ad8a826b/KwUGhaXeX-3Oyp0-TkmXG.wav"></audio> |
+| ¿Será que todavía me recuerdas como antes? | <audio controls src="https://cdn-uploads.huggingface.co/production/uploads/64fab67bd268b2f1ad8a826b/rKbdJ6-Ouexpijb6R8LDJ.wav"></audio> |
+| ¡Qué alegría volver a verte después de tanto tiempo! | <audio controls src="https://cdn-uploads.huggingface.co/production/uploads/64fab67bd268b2f1ad8a826b/Jj9uJ_msHARXdkq38OHya.wav"></audio> |
+
+## Use Cases
+- **Conversational AI:** Real-time speech for chatbots and virtual assistants
+- **Edge/Server Deployment:** Resource-efficient inference on affordable hardware
+- **Accessibility:** Screen readers and language learning applications
+- **Research:** Fine-tuning for specific voices, accents, or emotions
+  
+## Limitations
+- Performance degrades with inputs exceeding 15 seconds (need to use sliding window chunking)
+- Limited expressivity without fine-tuning for specific emotions
+- May inherit biases from training data in prosody or pronunciation
+- Optimized primarily for English; other languages may require additional training
+  
+## Optimization Tips
+- **Multilingual Performance:** Continually pretrain on target language datasets and fine-tune NanoCodec
+- **Batch Processing:** Use batches of 8-16 for high-throughput scenarios
+- **Hardware:** Optimized for NVIDIA Blackwell architecture GPUs
+  
+## Resources
+
+**Models:**
+- **Pretrained Model:** https://huggingface.co/nineninesix/kani-tts-500m-0.3-pt
+- **Space:** https://huggingface.co/spaces/nineninesix/KaniTTS
+  
+**Examples:**
+- **OpenAI compatible API Example**: https://github.com/nineninesix-ai/kanitts-vllm
+- **Finetuning code pipeline:** https://github.com/nineninesix-ai/KaniTTS-Finetune-pipeline
+- **Dataset preparation pipeline:** https://github.com/nineninesix-ai/nano-codec-dataset-pipeline
+- **Example Dataset:** https://huggingface.co/datasets/nineninesix/expresso-conversational-en-nano-codec-dataset
+- **GitHub Repository:** https://github.com/nineninesix-ai/kani-tts
+- **ComfyUI node:** https://github.com/wildminder/ComfyUI-KaniTTS by [WildAi](https://github.com/wildminder)
+- **NextJS basic app:** https://github.com/nineninesix-ai/open-audio. It uses the OpenAI npm package to connect to the OpenAI-compatible server API provided by [kanitts-vllm](https://github.com/nineninesix-ai/kanitts-vllm).
+
+**Links:**
+- **Website:** https://www.nineninesix.ai
+- **Contact Form:** https://airtable.com/appX2G2TpoRk4M5Bf/pagO2xbIOjiwulPcP/form
+
+  
+## Acknowledgments
+Built on top of [LiquidAI LFM2 350M](https://huggingface.co/LiquidAI/LFM2-350M) as the backbone and [Nvidia NanoCodec](https://huggingface.co/nvidia/nemo-nano-codec-22khz-0.6kbps-12.5fps) for audio processing.
+## Responsible Use
+**Prohibited activities include:**
+- Illegal content or harmful, threatening, defamatory, or obscene material
+- Hate speech, harassment, or incitement of violence
+- Generating false or misleading information
+- Impersonating individuals without consent
+- Malicious activities such as spamming, phishing, or fraud
+By using this model, you agree to comply with these restrictions and all applicable laws.
+
+## Contact
+Have a question, feedback, or need support? Please fill out our [contact form](https://airtable.com/appX2G2TpoRk4M5Bf/pagO2xbIOjiwulPcP/form) and we'll get back to you as soon as possible.
+
+## Citation
+```
+@inproceedings{emilialarge,
+  author={He, Haorui and Shang, Zengqiang and Wang, Chaoren and Li, Xuyuan and Gu, Yicheng and Hua, Hua and Liu, Liwei and Yang, Chen and Li, Jiaqi and Shi, Peiyang and Wang, Yuancheng and Chen, Kai and Zhang, Pengyuan and Wu, Zhizheng},
+  title={Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation},
+  booktitle={arXiv:2501.15907},
+  year={2025}
+}
+```
+```
+@article{emonet_voice_2025,
+  author={Schuhmann, Christoph and Kaczmarczyk, Robert and Rabby, Gollam and Friedrich, Felix and Kraus, Maurice and Nadi, Kourosh and Nguyen, Huu and Kersting, Kristian and Auer, Sören},
+  title={EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection},
+  journal={arXiv preprint arXiv:2506.09827},
+  year={2025}
+}
+```
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,37 @@
+{{- bos_token -}}
+{%- set system_prompt = "" -%}
+{%- set ns = namespace(system_prompt="") -%}
+{%- if messages[0]["role"] == "system" -%}
+	{%- set ns.system_prompt = messages[0]["content"] -%}
+	{%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+	{%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: <|tool_list_start|>[" -%}
+	{%- for tool in tools -%}
+		{%- if tool is not string -%}
+            {%- set tool = tool | tojson -%}
+		{%- endif -%}
+		{%- set ns.system_prompt = ns.system_prompt + tool -%}
+        {%- if not loop.last -%}
+            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
+        {%- endif -%}
+	{%- endfor -%}
+	{%- set ns.system_prompt = ns.system_prompt + "]<|tool_list_end|>" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+	{{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- for message in messages -%}
+	{{- "<|im_start|>" + message["role"] + "\n" -}}
+	{%- set content = message["content"] -%}
+	{%- if content is not string -%}
+		{%- set content = content | tojson -%}
+	{%- endif -%}
+	{%- if message["role"] == "tool" -%}
+		{%- set content = "<|tool_response_start|>" + content + "<|tool_response_end|>" -%}
+	{%- endif -%}
+	{{- content + "<|im_end|>\n" -}}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+	{{- "<|im_start|>assistant\n" -}}
+{%- endif -%}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,65 @@
+{
+  "architectures": [
+    "Lfm2ForCausalLM"
+  ],
+  "block_auto_adjust_ff_dim": true,
+  "block_dim": 1024,
+  "block_ff_dim": 6656,
+  "block_ffn_dim_multiplier": 1.0,
+  "block_mlp_init_scale": 1.0,
+  "block_multiple_of": 256,
+  "block_norm_eps": 1e-05,
+  "block_out_init_scale": 1.0,
+  "block_use_swiglu": true,
+  "block_use_xavier_init": true,
+  "bos_token_id": 1,
+  "conv_L_cache": 3,
+  "conv_bias": false,
+  "conv_dim": 1024,
+  "conv_dim_out": 1024,
+  "conv_use_xavier_init": true,
+  "dtype": "bfloat16",
+  "eos_token_id": 7,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 6656,
+  "layer_types": [
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv"
+  ],
+  "max_position_embeddings": 128000,
+  "model_type": "lfm2",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_heads": 16,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "rope_theta": 1000000.0,
+  "transformers_version": "4.56.0",
+  "use_cache": true,
+  "use_pos_enc": true,
+  "vocab_size": 80539,
+  "speaker_settings": {
+                        "status": "multispeaker",
+                        "speaker_list": ["nova", "ballad", "ash"]
+                      }
+}
--- a/configuration.json
+++ b/configuration.json
@@ -0,0 +1 @@
+{"framework": "pytorch", "task": "text-to-speech", "allow_remote": true}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 7,
+  "pad_token_id": 0,
+  "transformers_version": "4.56.0"
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a957951eabb884e6706ca8f32202d690d0a253f4f854b643a6cccfb5b55ab18a
+size 739710608
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aad29b357e116edb5ea32a47735500464292014f76bb598718d9be3417c4b3a8
+size 7884560
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:543d6c05a02aa4eb03d25defba59e72658e5704033b91d1d60459ffa1a620eac
+size 3082321
				`@@ -0,0 +1 @@`
				`{"framework": "pytorch", "task": "text-to-speech", "allow_remote": true}`