121 lines
4.5 KiB
Markdown
121 lines
4.5 KiB
Markdown
---
|
|
license: mit
|
|
language:
|
|
- en
|
|
- ru
|
|
library_name: gguf
|
|
pipeline_tag: text-generation
|
|
base_model:
|
|
- occ-ai/OCC-RAG-0.6B
|
|
tags:
|
|
- rag
|
|
- faithful-qa
|
|
- occ
|
|
- gguf
|
|
- llama.cpp
|
|
- quantized
|
|
---
|
|
|
|
# OCC-RAG-0.6B-GGUF
|
|
|
|
<p align="center">
|
|
<img src="figures/occ.png" alt="OCC-RAG" width="320"/>
|
|
</p>
|
|
|
|
<p align="center">
|
|
<a href="https://github.com/optimal-cognitive-core/OCC-RAG"><img src="figures/github-mark.png" alt=""/> <b>GitHub</b></a> |
|
|
<a href="https://arxiv.org/abs/2606.00683"><b>Technical Report</b></a> |
|
|
<a href="https://cloud.ru/products/evolution-ml-inference"><b>Cloud</b></a> |
|
|
<a href="https://huggingface.co/occ-ai/OCC-RAG-0.6B"><b>Base model</b></a>
|
|
</p>
|
|
|
|
**GGUF quantizations of [`occ-ai/OCC-RAG-0.6B`](https://huggingface.co/occ-ai/OCC-RAG-0.6B)**
|
|
for native inference with [llama.cpp](https://github.com/ggml-org/llama.cpp),
|
|
[Ollama](https://ollama.com/), [LM Studio](https://lmstudio.ai/), and other
|
|
GGUF-compatible runtimes.
|
|
|
|
OCC-RAG-0.6B is a 0.6B-parameter small language model specialized for **faithful,
|
|
context-grounded question answering**: given a question and a set of sources, it
|
|
produces a structured reasoning trace with explicit source citations, decides whether
|
|
the context supports an answer, and either answers from the context or abstains. See the
|
|
[base model card](https://huggingface.co/occ-ai/OCC-RAG-0.6B) for training details and
|
|
benchmarks.
|
|
|
|
## Files
|
|
|
|
The underlying architecture is Qwen3 (0.6B). The chat template is embedded in the GGUF,
|
|
so `llama.cpp`/Ollama apply it automatically.
|
|
|
|
| File | Quant | Size | Notes |
|
|
|---|---|---|---|
|
|
| `OCC-RAG-0.6B-Q4_0.gguf` | Q4_0 | 0.38 GB | 4-bit, legacy — smallest |
|
|
| `OCC-RAG-0.6B-Q4_K_M.gguf` | Q4_K_M | 0.40 GB | 4-bit K-quant — **recommended balance** |
|
|
| `OCC-RAG-0.6B-Q5_K_M.gguf` | Q5_K_M | 0.44 GB | 5-bit K-quant |
|
|
| `OCC-RAG-0.6B-Q6_K.gguf` | Q6_K | 0.50 GB | 6-bit K-quant |
|
|
| `OCC-RAG-0.6B-Q8_0.gguf` | Q8_0 | 0.64 GB | 8-bit — near-lossless |
|
|
| `OCC-RAG-0.6B-F16.gguf` | F16 | 1.20 GB | 16-bit float (full precision) |
|
|
| `OCC-RAG-0.6B-BF16.gguf` | BF16 | 1.20 GB | 16-bit bfloat (lossless base) |
|
|
|
|
For most uses pick **Q4_K_M** (smallest good quality) or **Q8_0** (highest quality at
|
|
under a GB). All quants are derived from the BF16 base.
|
|
|
|
## Usage — llama.cpp
|
|
|
|
```bash
|
|
# Run directly from the Hub (downloads the chosen quant)
|
|
llama-cli -hf occ-ai/OCC-RAG-0.6B-GGUF:Q4_K_M -p "Hello" -no-cnv
|
|
|
|
# Or download a file and run it
|
|
llama-cli -m OCC-RAG-0.6B-Q4_K_M.gguf -p "Hello" -no-cnv
|
|
# (newer llama.cpp: use `llama-completion` for non-interactive runs)
|
|
```
|
|
|
|
## Usage — Ollama
|
|
|
|
```bash
|
|
ollama run hf.co/occ-ai/OCC-RAG-0.6B-GGUF:Q4_K_M
|
|
```
|
|
|
|
## Input / output format
|
|
|
|
OCC-RAG uses a **structured RAG prompt with special tokens**: the question is wrapped in
|
|
`<|query_start|> … <|query_end|>` and each source in
|
|
`<|source_start|><|source_id|>N … <|source_end|>`. The response has five sections —
|
|
**query analysis → source analysis → reasoning → status (`ANSWERABLE` / `UNANSWERABLE`)
|
|
→ answer** — and the final answer is in `<|answer_start|> … <|answer_end|>`.
|
|
|
|
The embedded chat template (apply with llama.cpp's `--jinja`) builds the query/source
|
|
tokens for you when sources are supplied as documents; alternatively assemble the tokens
|
|
manually. See the [base model card](https://huggingface.co/occ-ai/OCC-RAG-0.6B) for the
|
|
full format and a runnable example.
|
|
|
|
> We recommend **greedy decoding** (`--temp 0`), the training/evaluation default.
|
|
|
|
## Limitations
|
|
|
|
- **Context-grounded only.** Trained to answer from the supplied sources and to ignore
|
|
parametric knowledge — not a general-purpose chat or knowledge model.
|
|
- **Reasoning depth.** Training/evaluation are capped at three-hop reasoning; longer
|
|
chains are out of distribution.
|
|
- **Quantization.** Lower-bit quants (Q4) trade some quality for size; prefer Q6_K/Q8_0
|
|
when accuracy matters most.
|
|
|
|
## License
|
|
|
|
Released under the [MIT License](https://huggingface.co/occ-ai/OCC-RAG-0.6B), inherited
|
|
from the base model.
|
|
|
|
## Citation
|
|
|
|
```bibtex
|
|
@misc{savkin2026occragoptimalcognitivecore,
|
|
title = {OCC-RAG: Optimal Cognitive Core for Faithful Question Answering},
|
|
author = {Maksim Savkin and Mikhail Goncharov and Alexander Gambashidze and Alla Chepurova and Dmitrii Tarasov and Nikita Andriianov and Daria Pugacheva and Vasily Konovalov and Andrey Galichin and Ivan Oseledets},
|
|
year = {2026},
|
|
eprint = {2606.00683},
|
|
archivePrefix = {arXiv},
|
|
primaryClass = {cs.CL},
|
|
url = {https://arxiv.org/abs/2606.00683}
|
|
}
|
|
```
|