44 lines
1.4 KiB
Markdown
44 lines
1.4 KiB
Markdown
---
|
|
base_model:
|
|
- dungnvt/Tini-8B-A1B
|
|
language:
|
|
- en
|
|
library_name: transformers
|
|
tags:
|
|
- gguf
|
|
- lfm2_moe
|
|
- reasoning
|
|
- agent
|
|
- function-calling
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# Tini-8B-A1B-GGUF
|
|
|
|
This repository contains GGUF format versions of the fine-tuned model **Tini-8B-A1B**, which is based on the **LiquidAI/LFM2.5-8B-A1B** architecture and optimized for agent reasoning and function-calling.
|
|
|
|
## Files Available
|
|
|
|
* **Tini-8B-A1B-BF16.gguf** (15.78 GB): Unquantized Brain Float 16 base GGUF file.
|
|
* **Tini-8B-A1B-Q8_0.gguf** (8.39 GB): 8-bit standard quantization. High accuracy, recommended for general inference.
|
|
* **Tini-8B-A1B-Q6_K.gguf** (6.48 GB): 6-bit quantization. Good balance between size and perplexity.
|
|
* **Tini-8B-A1B-Q4_K_M.gguf** (4.80 GB): 4-bit Medium K-quantized model. Highly efficient resource usage.
|
|
|
|
## Running the Model
|
|
|
|
Since the `lfm2_moe` architecture is relatively new, make sure to use a recent version of `llama.cpp` or downstream tools (LM Studio, Ollama, etc.) that support this model type.
|
|
|
|
### Using llama-cli
|
|
|
|
You can run the model directly using `llama-cli`:
|
|
|
|
```bash
|
|
llama-cli -m Tini-8B-A1B-Q8_0.gguf -p "<|im_start|>user\nHello, how can you help me today?<|im_end|>\n<|im_start|>assistant\n"
|
|
```
|
|
|
|
## Model Architecture Details
|
|
|
|
* **Architecture**: `Lfm2MoeForCausalLM`
|
|
* **Experts**: 32 experts (MoE)
|
|
* **Experts per Token**: 4 active experts
|
|
* **Context Window**: Up to 128k tokens |