---
license: apache-2.0
base_model: Qwen/Qwen3-8B
tags:
- reinforcement-learning
- grpo
- infrastructure-management
- sre
- kubernetes
- lora
- unsloth
- trl
language:
- en
pipeline_tag: text-generation
---
# Qwen3-8B — GRPO Fine-tuned on DIME
**Qwen3-8B fine-tuned via Group Relative Policy Optimization (GRPO) to act as an autonomous Site-Reliability Engineer in a simulated 8-node Kubernetes cluster.**
---
## What is this model?
This is a **merged BF16 checkpoint** of [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) after 300 steps of GRPO fine-tuning on the DIME (Distributed Infrastructure Management Environment) benchmark — trained in **44 minutes on a single A100-SXM4-80GB**.
The model observes per-node CPU, memory, queue depths, and tail-latency telemetry, then outputs a single `kubectl` command to maintain cluster health. It was trained from scratch with a completely redesigned reward signal after the original reward function was found to produce zero-variance advantages that blocked all gradient flow.
---
## Benchmark Results
**+44.2% relative improvement over zero-shot Qwen3-8B** on the 14-task DIME benchmark.
| Metric | Zero-shot | Fine-tuned |
|---|---|---|
| Overall avg score | 0.394 | **0.569** (best episode) |
| connection_pool_deadlock | 0.630 | **0.976** |
| memory_leak_slow_burn | 0.990 | **0.990** |
| node_failure | 0.220 | **0.920** |
| retry_storm | 0.377 | **0.587** |
| thundering_herd | 0.393 | **0.606** |
| traffic_spike | 0.024 | **0.399** |
---
## The Reward Engineering Challenge
The central technical contribution is replacing a **gradient-blocking reward cliff** with a differentiable seven-component signal.
The original reward returned `r = −1000` whenever the database node failed — which happened within the first 3 steps of most episodes. With all rewards identical, GRPO advantages collapsed to zero and no gradient flowed.
The fixed reward:
$$R_{env}(s,a) = \text{clip}(\, r_{topo} + r_{shed} + r_{mem} + r_{fric} + r_{lat} + r_{up} + r_{eff},\; -5,\; +5 \,)$$
with bounded components ensuring non-zero gradient everywhere.
---
## DIME Environment
- **8-node cluster**: node-0 is a stateful PostgreSQL DB (SPOF), nodes 1–7 are stateless workers
- **Partial observability**: telemetry dropout when `cpu_i = −1`
- **6 action types**: restart, reroute, scale_up, throttle, query_logs, no_op
- **14 failure scenarios**: traffic spikes, node failures, memory leaks, retry storms, split-brain, and more
- **Error budget**: throttle burns irreplaceable budget; the agent must be economical
---
## Training Dynamics
- `reward_format` reached 3.0 (perfect) from step 1 — the model learned XML scaffold immediately
- `reward_validity` stabilised at 1.9+ — no invalid commands after step ~10
- `reward_env` improved steadily — environment physics signal dominated learning
- `clipped_ratio` stayed near 0 throughout — healthy PPO clip utilisation
- Total wall-clock: **44 minutes** at 8.4 s/step
---
## Triage Oracle and the Priority Inversion Bug
The model is guided during training by a 10-rule deterministic triage tree:
A critical discovery: the original oracle evaluated the **Black Swan rule** (`|F(s)| ≥ 2 → throttle(0.3)`) *before* the **DB Recovery rule** (`0 ∈ F(s) → restart_node(0)`). When multiple nodes including the database were failed, the oracle prescribed `throttle` instead of `restart_node(0)`. The model faithfully learned this suboptimal policy. Fixing the priority ordering accounted for **+0.044 benchmark score** improvement.
---
## Failure Modes Documented
Five training configurations failed before convergence — each diagnosable from standard TRL metrics:
| Run | Failure | Signal |
|---|---|---|
| vLLM on A100-40GB | SM 8.0 segfault (`compilation_config` not set) | Crash at init |
| batch=4, gen=8 | CPU-bound rewards; 126 s/step, GPU idle | `samples/sec = 0.06` |
| max_comp=256 | `` blocks truncated before `` | `frac_reward_zero_std = 1.0` |
| reward_env×2 | 10:1 env-to-triage ratio recreated zero-variance | `zero_std → 1.0` at step 119 |
| oracle inverted | Learned `throttle` in DB-failure states | Low `triage/mean` |
---
## Reward Components
```
R(a,s) = R_fmt [-3,+3] + R_val [-2,+2] + R_env [-5,+5] + R_tri [-0.5,+1]
```
- **R_fmt**: XML scaffold compliance (`` + `` tags)
- **R_val**: Syntactic kubectl parse success
- **R_env**: 7-component physics reward (topology, latency, memory, uptime, budget)
- **R_tri**: Oracle triage alignment (gentle guidance, not primary teacher)
---
## Quick Start
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"Naseer-010/Qwen3-8B-Finetuned-DIME",
torch_dtype="bfloat16",
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Naseer-010/Qwen3-8B-Finetuned-DIME")
system_prompt = """You are an autonomous SRE agent managing an 8-node Kubernetes cluster.
Node-0 is the PostgreSQL database (SPOF). Nodes 1-7 are stateless workers.
TRIAGE PRIORITY (check in order):
1. OOM: if any node mem > 0.92 → kubectl delete pod node-
2. DB RECOVERY: if node-0 in failed_nodes → kubectl delete pod node-0
3. SPLIT-BRAIN: if io_wait > 0.80 → kubectl throttle ingress --rate=0.5
4. HOT-SHARD: if one worker cpu > 0.90, others low → reroute traffic
5. RETRY STORM: if p99 > 100ms and rr > 150 → kubectl throttle ingress --rate=0.4
6. ZOMBIE NODE: if worker cpu near 0 → reroute away from it
7. BLACK SWAN: if 2+ nodes failed (DB alive) → kubectl throttle ingress --rate=0.3
8. DB STRESS: if node-0 cpu > 0.80 → kubectl throttle ingress --rate=0.7
9. SAFE SCALE: if avg worker cpu > 0.75 and budget > 20 → scale up
10. HEALTHY → no_op
Output format:
One sentence identifying which rule applies.
{"command": "kubectl ..."}"""
obs = {
"cpu_loads": [0.45, 0.82, 0.79, 0.88, 0.75, 0.81, 0.77, 0.73],
"mem_utilizations": [0.41, 0.68, 0.71, 0.65, 0.62, 0.70, 0.66, 0.64],
"queue_lengths": [12, 45, 41, 53, 38, 44, 40, 37],
"failed_nodes": [],
"latency_ms": 187.3,
"p99_latency": 312.5,
"request_rate": 1840.0,
"io_wait": 0.12,
"error_budget": 85,
"step": 4,
"task_hint": "System is under heavy traffic load."
}
import json
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": f"Current system state:\n{json.dumps(obs, indent=2)}\nWhat action should be taken?"}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=1024,
temperature=0.6,
top_p=0.95,
do_sample=True,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
Expected output:
```
P99 latency is 312ms with request rate 1840 rps and no failed nodes.
Rule 5 (retry storm): p99 > 100ms and rr > 150 → throttle at 0.4.
Rule 5 applies: p99 latency 312ms exceeds threshold with high request rate 1840 rps — throttle ingress to shed load.
{"command": "kubectl throttle ingress --rate=0.4"}
```
---
## Training Details
| Parameter | Value |
|---|---|
| Base model | `Qwen/Qwen3-8B` (BF16) |
| Method | GRPO (TRL 0.24.0 + Unsloth + vLLM 0.6.3) |
| LoRA rank | 32, alpha=64, all projection layers |
| Trainable params | 1.05% (349 MB adapter) |
| Training steps | 300 |
| Batch size | 1 × 4 generations = 4 completions/step |
| Learning rate | 5e-6, cosine schedule |
| Max completion length | 1024 tokens |
| GPU | A100-SXM4-80GB |
| Wall-clock time | 44 minutes |
---
## Citation
```bibtex
@misc{dime2026,
title = {Fine-Tuning Language Models as Autonomous SREs via GRPO: The DIME Benchmark},
author = {Nithish Sriram and Naseer},
year = {2026},
url = {https://huggingface.co/Naseer-010/Qwen3-8B-Finetuned-DIME}
}
```
---
*Trained at SRM AP · Hackathon 2026*