JARVIS-Mistral-Phase1b-GGUF/README.md

---
base_model: mistralai/Mistral-7B-v0.1
library_name: gguf
pipeline_tag: text-generation
tags:
  - gguf
  - mistral
  - macedonian
  - llama-cpp
license: mit
language:
  - mk
---

# JARVIS-Mistral-Phase1b-GGUF

**Model ID:** `Miki-T/JARVIS-Mistral-Phase1b-GGUF`

GGUF-format versions of the JARVIS Mistral 7B Phase 1b model for use with llama.cpp and llama-cpp-python. This is the inference-ready deployment format of `Miki-T/JARVIS-Mistral-Phase1b`.

---

## Files

| File | Size | Description |
|------|------|-------------|
| `mistral-7b-jarvis-phase1b.Q4_K_M.gguf` | 4.1GB | 4-bit quantized — recommended for daily use |
| `mistral-7b-jarvis-phase1b.f16.gguf` | 14.5GB | Full float16 — master file for requantization |

---

## Model Details

This model is the result of merging two LoRA adapters into base Mistral 7B:

- **Phase 1a adapter** (`Miki-T/JARVIS-Mistral-Phase1a`) — Macedonian language foundation trained on 500k rows of Macedonian web text
- **Phase 1b adapter** (`Miki-T/JARVIS-Mistral-Phase1b`) — Macedonian instruction following trained on 134k instruction-response pairs

The adapters were merged sequentially using `merge_and_unload()`, converted to GGUF via `convert_hf_to_gguf.py`, and quantized to Q4_K_M via `llama-quantize`.

| Property | Value |
|----------|-------|
| Base model | mistralai/Mistral-7B-v0.1 |
| Language | Macedonian (mk), English (en) |
| Format | GGUF |
| Quantization | Q4_K_M (recommended), f16 (master) |
| Context length | 32768 tokens |

---

## Usage

### With llama-cpp-python

```python
from llama_cpp import Llama

model = Llama(
    model_path="mistral-7b-jarvis-phase1b.Q4_K_M.gguf",
    n_ctx=2048,
    n_gpu_layers=40,
    verbose=False,
)

prompt = "[INST] Кој е главен град на Македонија? [/INST]"
response = model(prompt, max_tokens=256, temperature=0.7)
print(response["choices"][0]["text"])
# Output: Скопје е главниот град на Македонија.
```

### Prompt format

Uses Mistral instruct format:
[INST] <<SYS>>
Your system prompt here
<</SYS>>
User message [/INST] Assistant response

---

## Hardware

- **Trained on:** NVIDIA RTX 5070 (12GB VRAM)
- **Q4_K_M inference:** ~4.1GB VRAM, runs on any GPU with 6GB+
- **f16 inference:** ~14.5GB VRAM

---

## Merge and Conversion

```python
# Step 1 — Merge adapters
from transformers import AutoModelForCausalLM
from peft import PeftModel
import torch

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    torch_dtype=torch.float16,
    device_map="cpu",
)
model = PeftModel.from_pretrained(model, "path/to/phase1a/adapter")
model = model.merge_and_unload()
model = PeftModel.from_pretrained(model, "path/to/phase1b/adapter")
model = model.merge_and_unload()
model.save_pretrained("merged_model/")

# Step 2 — Convert to GGUF (requires llama.cpp)
# python convert_hf_to_gguf.py merged_model/ --outfile model.f16.gguf --outtype f16

# Step 3 — Quantize
# llama-quantize.exe model.f16.gguf model.Q4_K_M.gguf Q4_K_M
```

---

## Related Repositories

- **Phase 1a adapter:** `Miki-T/JARVIS-Mistral-Phase1a`
- **Phase 1b adapter:** `Miki-T/JARVIS-Mistral-Phase1b`
- **Project:** https://github.com/MikiTrajkovski/JARVIS

---

## License

MIT License

---

**Author:** Miki Trajkovski | [GitHub](https://github.com/MikiTrajkovski/JARVIS) | [HuggingFace](https://huggingface.co/Miki-T)

**Last Updated:** July 3, 2026