133 lines
3.3 KiB
Markdown
133 lines
3.3 KiB
Markdown
---
|
||
base_model: mistralai/Mistral-7B-v0.1
|
||
library_name: gguf
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- gguf
|
||
- mistral
|
||
- macedonian
|
||
- llama-cpp
|
||
license: mit
|
||
language:
|
||
- mk
|
||
---
|
||
|
||
# JARVIS-Mistral-Phase1b-GGUF
|
||
|
||
**Model ID:** `Miki-T/JARVIS-Mistral-Phase1b-GGUF`
|
||
|
||
GGUF-format versions of the JARVIS Mistral 7B Phase 1b model for use with llama.cpp and llama-cpp-python. This is the inference-ready deployment format of `Miki-T/JARVIS-Mistral-Phase1b`.
|
||
|
||
---
|
||
|
||
## Files
|
||
|
||
| File | Size | Description |
|
||
|------|------|-------------|
|
||
| `mistral-7b-jarvis-phase1b.Q4_K_M.gguf` | 4.1GB | 4-bit quantized — recommended for daily use |
|
||
| `mistral-7b-jarvis-phase1b.f16.gguf` | 14.5GB | Full float16 — master file for requantization |
|
||
|
||
---
|
||
|
||
## Model Details
|
||
|
||
This model is the result of merging two LoRA adapters into base Mistral 7B:
|
||
|
||
- **Phase 1a adapter** (`Miki-T/JARVIS-Mistral-Phase1a`) — Macedonian language foundation trained on 500k rows of Macedonian web text
|
||
- **Phase 1b adapter** (`Miki-T/JARVIS-Mistral-Phase1b`) — Macedonian instruction following trained on 134k instruction-response pairs
|
||
|
||
The adapters were merged sequentially using `merge_and_unload()`, converted to GGUF via `convert_hf_to_gguf.py`, and quantized to Q4_K_M via `llama-quantize`.
|
||
|
||
| Property | Value |
|
||
|----------|-------|
|
||
| Base model | mistralai/Mistral-7B-v0.1 |
|
||
| Language | Macedonian (mk), English (en) |
|
||
| Format | GGUF |
|
||
| Quantization | Q4_K_M (recommended), f16 (master) |
|
||
| Context length | 32768 tokens |
|
||
|
||
---
|
||
|
||
## Usage
|
||
|
||
### With llama-cpp-python
|
||
|
||
```python
|
||
from llama_cpp import Llama
|
||
|
||
model = Llama(
|
||
model_path="mistral-7b-jarvis-phase1b.Q4_K_M.gguf",
|
||
n_ctx=2048,
|
||
n_gpu_layers=40,
|
||
verbose=False,
|
||
)
|
||
|
||
prompt = "[INST] Кој е главен град на Македонија? [/INST]"
|
||
response = model(prompt, max_tokens=256, temperature=0.7)
|
||
print(response["choices"][0]["text"])
|
||
# Output: Скопје е главниот град на Македонија.
|
||
```
|
||
|
||
### Prompt format
|
||
|
||
Uses Mistral instruct format:
|
||
[INST] <<SYS>>
|
||
Your system prompt here
|
||
<</SYS>>
|
||
User message [/INST] Assistant response
|
||
|
||
---
|
||
|
||
## Hardware
|
||
|
||
- **Trained on:** NVIDIA RTX 5070 (12GB VRAM)
|
||
- **Q4_K_M inference:** ~4.1GB VRAM, runs on any GPU with 6GB+
|
||
- **f16 inference:** ~14.5GB VRAM
|
||
|
||
---
|
||
|
||
## Merge and Conversion
|
||
|
||
```python
|
||
# Step 1 — Merge adapters
|
||
from transformers import AutoModelForCausalLM
|
||
from peft import PeftModel
|
||
import torch
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
"mistralai/Mistral-7B-v0.1",
|
||
torch_dtype=torch.float16,
|
||
device_map="cpu",
|
||
)
|
||
model = PeftModel.from_pretrained(model, "path/to/phase1a/adapter")
|
||
model = model.merge_and_unload()
|
||
model = PeftModel.from_pretrained(model, "path/to/phase1b/adapter")
|
||
model = model.merge_and_unload()
|
||
model.save_pretrained("merged_model/")
|
||
|
||
# Step 2 — Convert to GGUF (requires llama.cpp)
|
||
# python convert_hf_to_gguf.py merged_model/ --outfile model.f16.gguf --outtype f16
|
||
|
||
# Step 3 — Quantize
|
||
# llama-quantize.exe model.f16.gguf model.Q4_K_M.gguf Q4_K_M
|
||
```
|
||
|
||
---
|
||
|
||
## Related Repositories
|
||
|
||
- **Phase 1a adapter:** `Miki-T/JARVIS-Mistral-Phase1a`
|
||
- **Phase 1b adapter:** `Miki-T/JARVIS-Mistral-Phase1b`
|
||
- **Project:** https://github.com/MikiTrajkovski/JARVIS
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
MIT License
|
||
|
||
---
|
||
|
||
**Author:** Miki Trajkovski | [GitHub](https://github.com/MikiTrajkovski/JARVIS) | [HuggingFace](https://huggingface.co/Miki-T)
|
||
|
||
**Last Updated:** July 3, 2026 |