KnowRL-Nemotron-1.5B/README.md

---
pretty_name: KnowRL-Nemotron-1.5B
language:
  - en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - knowrl
  - rlvr
  - reasoning
  - math
  - knowledge-points
  - reinforcement-learning
base_model: nvidia/OpenMath-Nemotron-1.5B
model-index:
  - name: KnowRL-Nemotron-1.5B
    results:
      - task:
          type: mathematical-reasoning
        dataset:
          name: AIME 2024
          type: aime24
        metrics:
          - name: Accuracy (w/o KP)
            type: accuracy
            value: 69.79
          - name: Accuracy (CSS)
            type: accuracy
            value: 74.58
      - task:
          type: mathematical-reasoning
        dataset:
          name: AIME 2025
          type: aime25
        metrics:
          - name: Accuracy (w/o KP)
            type: accuracy
            value: 64.69
          - name: Accuracy (CSS)
            type: accuracy
            value: 65.21
      - task:
          type: mathematical-reasoning
        dataset:
          name: MATH-500
          type: math-500
        metrics:
          - name: Accuracy (w/o KP)
            type: accuracy
            value: 95.70
          - name: Accuracy (CSS)
            type: accuracy
            value: 96.20
      - task:
          type: mathematical-reasoning
        dataset:
          name: Olympiad Bench
          type: olympiad-bench
        metrics:
          - name: Accuracy (w/o KP)
            type: accuracy
            value: 80.23
          - name: Accuracy (CSS)
            type: accuracy
            value: 82.44
---

# KnowRL-Nemotron-1.5B

> **KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance**

[![arXiv](https://img.shields.io/badge/2604.12627-b31b1b.svg)](https://arxiv.org/abs/2604.12627)
[![GitHub](https://img.shields.io/badge/💻%20GitHub-KnowRL-black)](https://github.com/HasuerYu/KnowRL)
[![Collection](https://img.shields.io/badge/🤗%20HuggingFace-Collection-yellow)](https://huggingface.co/collections/HasuerYu/knowrl)
[![Training Data](https://img.shields.io/badge/🤗%20HuggingFace-Training%20Data-yellow)](https://huggingface.co/datasets/HasuerYu/KnowRL-Train-Data)
[![KP Annotations](https://img.shields.io/badge/🤗%20HuggingFace-KP%20Annotations-yellow)](https://huggingface.co/datasets/HasuerYu/KnowRL-KP-Annotations)

## Model Summary

**KnowRL-Nemotron-1.5B** is a 1.5B-parameter math reasoning model trained with reinforcement learning (DAPO/GRPO) under **minimal-sufficient knowledge point (KP) guidance**. It is fine-tuned from [nvidia/OpenMath-Nemotron-1.5B](https://huggingface.co/nvidia/OpenMath-Nemotron-1.5B) and achieves state-of-the-art results among 1.5B-scale models on competition-level math benchmarks.

Instead of injecting long solution hints or full reasoning templates, KnowRL decomposes guidance into atomic **knowledge points (KPs)** and identifies the **minimal subset** required to unlock reward learning — achieving more with less.

## Key Highlights

- **74.16** average accuracy (CSS) across 8 competition-level math benchmarks — new SOTA at 1.5B scale
- **70.08** average accuracy even **without** KP hints at inference, demonstrating genuine policy improvement (+9.63 over baseline)
- Trained with **~38% fewer KPs** than full-KP injection via the CSS (Constrained Subset Search) selection strategy
- Reward sparsity reduced from **41.21%** zero-correct to **13.00%** during training

## Results

| Benchmark | w/o KP | CBRS | CSS |
|:----------|:------:|:----:|:---:|
| AIME 2024 | 69.79 | 75.52 | 74.58 |
| AIME 2025 | 64.69 | 65.00 | 65.21 |
| BRUMO 2025 | 69.48 | 78.33 | 78.12 |
| HMMT 2025 | 41.04 | 45.00 | 48.75 |
| AMC 2023 | 95.55 | 95.78 | 95.70 |
| CMIMC 2025 | 44.14 | 49.22 | 52.19 |
| MATH-500 | 95.70 | 96.45 | 96.20 |
| Olympiad Bench | 80.23 | 82.34 | 82.44 |
| **Average** | **70.08** | **73.46** | **74.16** |

> **w/o KP**: No knowledge point hints at inference.
> **CBRS / CSS**: KP hints selected by the respective strategy are prepended to the prompt at inference.

## Usage

### Basic Inference (without KP hints)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "HasuerYu/KnowRL-Nemotron-1.5B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")

problem = "Find the sum of all positive integers n such that n^2 - 19n + 99 is a perfect square."
prompt = f"{problem}\nPlease reason step by step, and put your final answer within \\boxed{{}}."

messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=32768, temperature=0.6, top_p=0.95)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)
```

### Inference with KP Hints

For best performance, prepend selected knowledge points as a hint section in the prompt:

```python
knowledge_points = [
    "If n^2 - 19n + 99 = m^2, then (2n - 19)^2 - 4m^2 = -15.",
]
hint = "## Hint\n" + "\n".join(f"- {kp}" for kp in knowledge_points)
prompt = f"{problem}\n{hint}\nPlease reason step by step, and put your final answer within \\boxed{{}}."
```

### vLLM Serving

```bash
vllm serve HasuerYu/KnowRL-Nemotron-1.5B \
    --tensor-parallel-size 1 \
    --max-model-len 32768 \
    --trust-remote-code
```

## Training Details

| Parameter | Value |
|-----------|-------|
| Base model | `nvidia/OpenMath-Nemotron-1.5B` |
| Algorithm | DAPO / GRPO |
| Framework | [verl](https://github.com/volcengine/verl) + Ray |
| Learning rate | 1e-6 |
| Batch size | 256 |
| Max prompt length | 8,192 |
| Max response length | 32,768 |
| Samples per prompt | 8 |
| Total training steps | 2,960 |
| Hardware | 8× NVIDIA H100 nodes (64 GPUs) |

An **entropy annealing** strategy is applied: after step 2,590, the clip upper bound is reduced from 0.28 to 0.26 to encourage the policy to shift from exploration to exploitation, contributing +0.74 average accuracy.

## How KnowRL Works

1. **KP Extraction**: Decompose solution guidance into atomic knowledge points (KPs)
2. **KP Selection**: Apply selection strategies (CSS, CBRS) to identify the minimal-sufficient subset of KPs per problem
3. **RL Training**: Train with DAPO/GRPO, injecting selected KPs as hints in the prompt during rollout
4. **Inference**: The trained model can be used with or without KP hints — even without hints, it significantly outperforms the baseline

## Related Resources

| Resource | Link |
|----------|------|
| KnowRL Collection | [HasuerYu/knowrl](https://huggingface.co/collections/HasuerYu/knowrl) |
| Training Data | [HasuerYu/KnowRL-Train-Data](https://huggingface.co/datasets/HasuerYu/KnowRL-Train-Data) |
| KP Annotations | [HasuerYu/KnowRL-KP-Annotations](https://huggingface.co/datasets/HasuerYu/KnowRL-KP-Annotations) |

## Limitations

- Optimized for competition-level math reasoning; performance on other domains is not evaluated
- KP hint quality at inference depends on upstream KP extraction and selection pipelines
- The model inherits limitations from the base model (`nvidia/OpenMath-Nemotron-1.5B`)

## Citation

If you find this model helpful, please cite:

```bibtex
@misc{yu2026knowrlboostingllmreasoning,
      title={KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance}, 
      author={Linhao Yu and Tianmeng Yang and Siyu Ding and Renren Jin and Naibin Gu and Xiangzhao Hao and Shuaiyi Nie and Deyi Xiong and Weichong Yin and Yu Sun and Hua Wu},
      year={2026},
      eprint={2604.12627},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2604.12627}, 
}
```

## License

This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).