Pollux-4B-Judge-GGUF/README.md

---
license: mit

language:
- ru

base_model: ai-forever/Pollux-4B-Judge

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

tags:

- gguf

- llama.cpp

- lmstudio

- qwen3

- pollux

- russian

- llm-as-a-judge

- quantized
quantized_by: ledgergap
---

# Pollux-4B-Judge GGUF

This repository contains GGUF versions of [`ai-forever/Pollux-4B-Judge`](https://huggingface.co/ai-forever/Pollux-4B-Judge) for local inference with llama.cpp, LM Studio, and other GGUF-compatible runtimes.

Pollux-4B-Judge is a Russian-oriented LLM-as-a-judge model based on Qwen3-4B. It is intended for evaluating model answers against a specific criterion and scoring rubric.

## Files

| File | Type | Quantized | Notes |
|---|---:|---:|---|
| `Pollux-4B-Judge.BF16.gguf` | BF16 GGUF conversion | No | High-precision reference version |
| `Pollux-4B-Judge.Q8_0.gguf` | Q8_0 GGUF quantization | Yes | High-quality quantized version |

## Which file should I use?

Use `Pollux-4B-Judge.BF16.gguf` if you want the highest-quality reference version.

Use `Pollux-4B-Judge.Q8_0.gguf` if you want a practical local version with lower memory usage and minimal expected quality loss.

## Recommended inference settings

For judge-style usage, the original model card uses:

| Setting | Value |
|---|---:|
| Temperature | `0.0` |
| Max tokens | `512` |

For local GGUF inference, choose a context length large enough to fit the full evaluation prompt: instruction, reference answer, evaluated answer, criterion, and rubric. A practical starting point is `8192`, but this is a local runtime recommendation rather than an official value from the original model card.

The model is intended to evaluate one criterion per request.

## Prompt format

Recommended prompt structure:

```text
### Задание для оценки:
{instruction}

### Эталонный ответ:
{reference_answer}

### Ответ для оценки:
{answer}

### Критерий оценки:
{criterion}

### Шкала оценивания по критерию:
{rubric}
```

## Use with llama.cpp

BF16:

```bash
llama-server -hf ledgergap/Pollux-4B-Judge-GGUF:BF16 -c 8192 -ngl 99
```

Q8_0:

```bash
llama-server -hf ledgergap/Pollux-4B-Judge-GGUF:Q8_0 -c 8192 -ngl 99
```

## Use with LM Studio

Open LM Studio and paste this repository URL into the model search/download field:

```text
https://huggingface.co/ledgergap/Pollux-4B-Judge-GGUF
```

Then select either the BF16 or Q8_0 GGUF file.

## Original model

Original model: [`ai-forever/Pollux-4B-Judge`](https://huggingface.co/ai-forever/Pollux-4B-Judge)