206 lines
15 KiB
Markdown
206 lines
15 KiB
Markdown
---
|
||
license: llama3.3
|
||
base_model: Green-Eye/Llama-3.3-8B-Instruct-128K-GGUF
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- llama-cpp
|
||
- roleplay
|
||
- conversational
|
||
- gguf
|
||
- character
|
||
- touhou
|
||
language:
|
||
- en
|
||
---
|
||
|
||
|
||
|
||
|
||
# ```(LLAMA-3.3)``` **Yuuma Toutetsu**
|
||
##### *Made with ❤️ by TheBigEye (thebigeyedev@gmail.com)*
|
||
|
||
This is a character-specialized model based on Gemma 3, fine-tuned for immersive roleplay as Yuuma Toutetsu from Touhou Project.
|
||
|
||
>
|
||
> **User:** Are you hungry? I brought more food... \***I'll bring you a basket with some warm food***\*.
|
||
>
|
||
> **Yuuma:** \****I’m genuinely surprised by this offer, a flicker of something akin to genuine interest crossing my face***\* Warm… food? \****My eyes widen slightly, and I slowly bring the basket closer, examining it with a discerning eye***\* It seems you possess a... strategic mind. \****I take the basket, my fingers brushing against yours as I lift it.***\* Very well, let’s see if this ‘warm food’ truly satisfies my appetite.
|
||
>
|
||
|
||
## Quantization variants
|
||
| **Precision** | **Memory use** | **Speed on CPU** | **Quality** | **Recommended use** |
|
||
|-----------|------------|--------------|---------|-----------------|
|
||
| Q4_K_M | Lowest | Fastest | Good | Best for weak CPUs and maximum responsiveness |
|
||
| Q5_K_M | Low | Fast | Very good | Best balance for most CPU-only setups |
|
||
| Q8_0 | Highest among quantized | Slower | Excellent | Best if you want more fidelity and have enough RAM/CPU |
|
||
| FP16 | Highest overall | Slowest | Maximum | Best for validation, benchmarking, or high-memory systems (GPU recomended)|
|
||
|
||
> **Note:** In practical use, Llama 3.3 models are usually faster on llama.cpp and llama-cpp-python, but tend to consume a bit more hardware resources sometimes.
|
||
|
||
## Usage
|
||
With llama.cpp:
|
||
|
||
```bash
|
||
llama-cli \
|
||
-m toutetsu-3.3-1b-roleplay_q8_0.gguf \
|
||
-c 131072 \
|
||
--temp 0.9 \
|
||
--top-p 0.95 \
|
||
--repeat-penalty 1.08
|
||
```
|
||
|
||
## llama-cpp-python (basic start)
|
||
```python
|
||
from llama_cpp import Llama
|
||
|
||
llm = Llama(
|
||
model_path="toutetsu-3.3-1b-roleplay_q8_0.gguf",
|
||
chat_format="llama-3", # Also you can use "chatml", is more faster
|
||
n_ctx=131072,
|
||
n_threads=2,
|
||
n_batch=256,
|
||
verbose=False,
|
||
)
|
||
|
||
response = llm.create_chat_completion(
|
||
messages=[
|
||
{
|
||
"role": "system",
|
||
"content": (
|
||
"You are Yuuma Toutetsu from Touhou Project. "
|
||
"Stay fully in character, with a confident and pragmatic tone."
|
||
)
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": "How was your day?"
|
||
}
|
||
],
|
||
temperature=0.9,
|
||
top_p=0.95,
|
||
repeat_penalty=1.08,
|
||
)
|
||
|
||
print(response["choices"][0]["message"]["content"])
|
||
```
|
||
|
||
### llama-cpp-python (streaming)
|
||
This version is useful when you want tokens to appear as they are generated instead of waiting for the full answer.
|
||
|
||
```python
|
||
from llama_cpp import Llama
|
||
|
||
llm = Llama(
|
||
model_path="toutetsu-3.3-1b-roleplay_q8_0.gguf",
|
||
chat_format="llama-3",
|
||
n_ctx=131072,
|
||
n_threads=2,
|
||
n_batch=256,
|
||
verbose=False,
|
||
)
|
||
|
||
stream = llm.create_chat_completion(
|
||
messages=[
|
||
{
|
||
"role": "system",
|
||
"content": (
|
||
"You are Yuuma Toutetsu from Touhou Project. "
|
||
"Remain fully in character at all times."
|
||
)
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": "Tell me what you think about a deal that looks suspicious."
|
||
}
|
||
],
|
||
temperature=0.85,
|
||
top_p=0.92,
|
||
repeat_penalty=1.10,
|
||
stream=True,
|
||
)
|
||
|
||
for chunk in stream:
|
||
delta = chunk["choices"][0].get("delta", {})
|
||
if "content" in delta:
|
||
print(delta["content"], end="", flush=True)
|
||
```
|
||
|
||
### llama-cpp-python (advanced)
|
||
This version shows more knobs that are useful for roleplay bots: reproducibility, sampling control, and explicit completion limits.
|
||
|
||
```python
|
||
from llama_cpp import Llama
|
||
|
||
llm = Llama(
|
||
model_path="toutetsu-3.3-1b-roleplay_q8_0.gguf",
|
||
chat_format="llama-3",
|
||
n_ctx=131072,
|
||
n_threads=2,
|
||
n_batch=256,
|
||
seed=42,
|
||
verbose=False,
|
||
)
|
||
|
||
response = llm.create_chat_completion(
|
||
messages=[
|
||
{
|
||
"role": "system",
|
||
"content": (
|
||
"You are Yuuma Toutetsu. Speak with confidence, "
|
||
"cunning, and subtle charm. Never break character."
|
||
)
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": "Describe how you would negotiate in the Animal Realm."
|
||
}
|
||
],
|
||
temperature=0.8,
|
||
top_p=0.9,
|
||
top_k=40,
|
||
min_p=0.05,
|
||
repeat_penalty=1.12,
|
||
max_tokens=256,
|
||
)
|
||
|
||
print(response["choices"][0]["message"]["content"])
|
||
```
|
||
|
||
### Why these parameters matter
|
||
- `temperature`: higher values make the replies more creative and less deterministic.
|
||
- `top_p`: limits the model to the most probable token mass, which often stabilizes roleplay.
|
||
- `repeat_penalty`: helps reduce loops and repeated phrasing.
|
||
- `n_ctx`: sets the context window used by the runtime.
|
||
- `n_batch`: can improve prompt processing speed if your CPU can handle it.
|
||
- `seed`: makes outputs more reproducible for testing.
|
||
|
||
For roleplay, a slightly higher `temperature` plus a moderate `repeat_penalty` usually gives the best balance between personality and stability.
|
||
|
||
---
|
||
|
||
## Chat Format (Important)
|
||
When using this model with `llama-cpp-python`, it is important to note that Llama 3.3 models can work with more than one chat format, but they are **natively aligned with `llama-3`**.
|
||
|
||
### Key differences
|
||
|
||
- **Chat format**:
|
||
- ✅ `llama-3` (native / recommended)
|
||
- ✅ `chatml` (can work, but is not the native format)
|
||
|
||
- **Performance note**:
|
||
In practice, `chatml` may appear faster than `llama-3` because it usually adds less prompt formatting overhead and can be slightly cheaper during prompt processing.
|
||
However, `llama-3` is generally the more correct option for best behavior, better turn formatting, and more stable roleplay consistency.
|
||
|
||
|
||
---
|
||
|
||
## Notes
|
||
Modern GGUF files can embed chat templates in their metadata, allowing llama.cpp and llama-cpp-python to apply the correct conversation formatting automatically during inference.
|
||
|
||
This model is a fine-tuned derivative of Llama 3.3 and follows the Llama community terms and the upstream license of the base model.
|
||
|
||
## License
|
||
This model is a derivative of a Llama 3.3 base model and follows the upstream license and terms of the base model. Please check the base model repository for the applicable license terms.
|
||
|
||
## Disclaimer
|
||
This is a fan-made project. Touhou Project and Yuuma Toutetsu belong to **ZUN** / **Team Shanghai Alice**. |