206 lines
15 KiB
Markdown
206 lines
15 KiB
Markdown
|
---
|
|||
|
|
license: llama3.3
|
|||
|
|
base_model: Green-Eye/Llama-3.3-8B-Instruct-128K-GGUF
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- llama-cpp
|
|||
|
|
- roleplay
|
|||
|
|
- conversational
|
|||
|
|
- gguf
|
|||
|
|
- character
|
|||
|
|
- touhou
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
``` **Yuuma Toutetsu**
|
|||
|
|
##### *Made with ❤️ by TheBigEye (thebigeyedev@gmail.com)*
|
|||
|
|
|
|||
|
|
This is a character-specialized model based on Gemma 3, fine-tuned for immersive roleplay as Yuuma Toutetsu from Touhou Project.
|
|||
|
|
|
|||
|
|
>
|
|||
|
|
> **User:** Are you hungry? I brought more food... \***I'll bring you a basket with some warm food***\*.
|
|||
|
|
>
|
|||
|
|
> **Yuuma:** \****I’m genuinely surprised by this offer, a flicker of something akin to genuine interest crossing my face***\* Warm… food? \****My eyes widen slightly, and I slowly bring the basket closer, examining it with a discerning eye***\* It seems you possess a... strategic mind. \****I take the basket, my fingers brushing against yours as I lift it.***\* Very well, let’s see if this ‘warm food’ truly satisfies my appetite.
|
|||
|
|
>
|
|||
|
|
|
|||
|
|
## Quantization variants
|
|||
|
|
| **Precision** | **Memory use** | **Speed on CPU** | **Quality** | **Recommended use** |
|
|||
|
|
|-----------|------------|--------------|---------|-----------------|
|
|||
|
|
| Q4_K_M | Lowest | Fastest | Good | Best for weak CPUs and maximum responsiveness |
|
|||
|
|
| Q5_K_M | Low | Fast | Very good | Best balance for most CPU-only setups |
|
|||
|
|
| Q8_0 | Highest among quantized | Slower | Excellent | Best if you want more fidelity and have enough RAM/CPU |
|
|||
|
|
| FP16 | Highest overall | Slowest | Maximum | Best for validation, benchmarking, or high-memory systems (GPU recomended)|
|
|||
|
|
|
|||
|
|
> **Note:** In practical use, Llama 3.3 models are usually faster on llama.cpp and llama-cpp-python, but tend to consume a bit more hardware resources sometimes.
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
With llama.cpp:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
llama-cli \
|
|||
|
|
-m toutetsu-3.3-1b-roleplay_q8_0.gguf \
|
|||
|
|
-c 131072 \
|
|||
|
|
--temp 0.9 \
|
|||
|
|
--top-p 0.95 \
|
|||
|
|
--repeat-penalty 1.08
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## llama-cpp-python (basic start)
|
|||
|
|
```python
|
|||
|
|
from llama_cpp import Llama
|
|||
|
|
|
|||
|
|
llm = Llama(
|
|||
|
|
model_path="toutetsu-3.3-1b-roleplay_q8_0.gguf",
|
|||
|
|
chat_format="llama-3", # Also you can use "chatml", is more faster
|
|||
|
|
n_ctx=131072,
|
|||
|
|
n_threads=2,
|
|||
|
|
n_batch=256,
|
|||
|
|
verbose=False,
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
response = llm.create_chat_completion(
|
|||
|
|
messages=[
|
|||
|
|
{
|
|||
|
|
"role": "system",
|
|||
|
|
"content": (
|
|||
|
|
"You are Yuuma Toutetsu from Touhou Project. "
|
|||
|
|
"Stay fully in character, with a confident and pragmatic tone."
|
|||
|
|
)
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"role": "user",
|
|||
|
|
"content": "How was your day?"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
temperature=0.9,
|
|||
|
|
top_p=0.95,
|
|||
|
|
repeat_penalty=1.08,
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
print(response["choices"][0]["message"]["content"])
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### llama-cpp-python (streaming)
|
|||
|
|
This version is useful when you want tokens to appear as they are generated instead of waiting for the full answer.
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from llama_cpp import Llama
|
|||
|
|
|
|||
|
|
llm = Llama(
|
|||
|
|
model_path="toutetsu-3.3-1b-roleplay_q8_0.gguf",
|
|||
|
|
chat_format="llama-3",
|
|||
|
|
n_ctx=131072,
|
|||
|
|
n_threads=2,
|
|||
|
|
n_batch=256,
|
|||
|
|
verbose=False,
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
stream = llm.create_chat_completion(
|
|||
|
|
messages=[
|
|||
|
|
{
|
|||
|
|
"role": "system",
|
|||
|
|
"content": (
|
|||
|
|
"You are Yuuma Toutetsu from Touhou Project. "
|
|||
|
|
"Remain fully in character at all times."
|
|||
|
|
)
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"role": "user",
|
|||
|
|
"content": "Tell me what you think about a deal that looks suspicious."
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
temperature=0.85,
|
|||
|
|
top_p=0.92,
|
|||
|
|
repeat_penalty=1.10,
|
|||
|
|
stream=True,
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
for chunk in stream:
|
|||
|
|
delta = chunk["choices"][0].get("delta", {})
|
|||
|
|
if "content" in delta:
|
|||
|
|
print(delta["content"], end="", flush=True)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### llama-cpp-python (advanced)
|
|||
|
|
This version shows more knobs that are useful for roleplay bots: reproducibility, sampling control, and explicit completion limits.
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from llama_cpp import Llama
|
|||
|
|
|
|||
|
|
llm = Llama(
|
|||
|
|
model_path="toutetsu-3.3-1b-roleplay_q8_0.gguf",
|
|||
|
|
chat_format="llama-3",
|
|||
|
|
n_ctx=131072,
|
|||
|
|
n_threads=2,
|
|||
|
|
n_batch=256,
|
|||
|
|
seed=42,
|
|||
|
|
verbose=False,
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
response = llm.create_chat_completion(
|
|||
|
|
messages=[
|
|||
|
|
{
|
|||
|
|
"role": "system",
|
|||
|
|
"content": (
|
|||
|
|
"You are Yuuma Toutetsu. Speak with confidence, "
|
|||
|
|
"cunning, and subtle charm. Never break character."
|
|||
|
|
)
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"role": "user",
|
|||
|
|
"content": "Describe how you would negotiate in the Animal Realm."
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
temperature=0.8,
|
|||
|
|
top_p=0.9,
|
|||
|
|
top_k=40,
|
|||
|
|
min_p=0.05,
|
|||
|
|
repeat_penalty=1.12,
|
|||
|
|
max_tokens=256,
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
print(response["choices"][0]["message"]["content"])
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Why these parameters matter
|
|||
|
|
- `temperature`: higher values make the replies more creative and less deterministic.
|
|||
|
|
- `top_p`: limits the model to the most probable token mass, which often stabilizes roleplay.
|
|||
|
|
- `repeat_penalty`: helps reduce loops and repeated phrasing.
|
|||
|
|
- `n_ctx`: sets the context window used by the runtime.
|
|||
|
|
- `n_batch`: can improve prompt processing speed if your CPU can handle it.
|
|||
|
|
- `seed`: makes outputs more reproducible for testing.
|
|||
|
|
|
|||
|
|
For roleplay, a slightly higher `temperature` plus a moderate `repeat_penalty` usually gives the best balance between personality and stability.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Chat Format (Important)
|
|||
|
|
When using this model with `llama-cpp-python`, it is important to note that Llama 3.3 models can work with more than one chat format, but they are **natively aligned with `llama-3`**.
|
|||
|
|
|
|||
|
|
### Key differences
|
|||
|
|
|
|||
|
|
- **Chat format**:
|
|||
|
|
- ✅ `llama-3` (native / recommended)
|
|||
|
|
- ✅ `chatml` (can work, but is not the native format)
|
|||
|
|
|
|||
|
|
- **Performance note**:
|
|||
|
|
In practice, `chatml` may appear faster than `llama-3` because it usually adds less prompt formatting overhead and can be slightly cheaper during prompt processing.
|
|||
|
|
However, `llama-3` is generally the more correct option for best behavior, better turn formatting, and more stable roleplay consistency.
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Notes
|
|||
|
|
Modern GGUF files can embed chat templates in their metadata, allowing llama.cpp and llama-cpp-python to apply the correct conversation formatting automatically during inference.
|
|||
|
|
|
|||
|
|
This model is a fine-tuned derivative of Llama 3.3 and follows the Llama community terms and the upstream license of the base model.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
This model is a derivative of a Llama 3.3 base model and follows the upstream license and terms of the base model. Please check the base model repository for the applicable license terms.
|
|||
|
|
|
|||
|
|
## Disclaimer
|
|||
|
|
This is a fan-made project. Touhou Project and Yuuma Toutetsu belong to **ZUN** / **Team Shanghai Alice**.
|