73 lines
3.2 KiB
Markdown
73 lines
3.2 KiB
Markdown
---
|
||
library_name: mlx
|
||
license: apache-2.0
|
||
license_link: https://huggingface.co/Qwen/Qwen3-1.7B/blob/main/LICENSE
|
||
pipeline_tag: text-generation
|
||
base_model: Qwen/Qwen3-1.7B
|
||
language:
|
||
- en
|
||
tags:
|
||
- mlx
|
||
- sigma
|
||
- detection-engineering
|
||
- cybersecurity
|
||
- blue-team
|
||
- security
|
||
---
|
||
|
||
# Qwen3-1.7B-SigmaRL (v0.1 · prototype)
|
||
|
||
Threat description → **Sigma detection rule**. A small, **defensive** detection-engineering model: a LoRA fine-tune of [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) that writes valid, backend-compilable [Sigma](https://github.com/SigmaHQ/sigma) rules from natural-language threat descriptions.
|
||
|
||
> **Defensive use only.** This model generates blue-team detection rules. It is not designed or intended for offensive security.
|
||
|
||
## Results (held-out, n=24)
|
||
|
||
Same 24 held-out prompts, scored by a **graded verifiable reward**: parse YAML → valid Sigma schema (pySigma) → compiles to a Splunk query (pySigma-backend-splunk).
|
||
|
||
| metric | base Qwen3-1.7B | **this model (SFT)** |
|
||
|---|---|---|
|
||
| valid + compilable rule rate | 0.0% | **45.8%** |
|
||
| mean reward (0–0.6) | 0.10 | **0.36** |
|
||
| tier breakdown (0.0 / 0.2 / 0.6) | 12 / 12 / 0 | 3 / 10 / 11 |
|
||
|
||
The base model **never once** produced a valid, compilable rule; after fine-tuning, nearly half of outputs are valid and Splunk-compilable.
|
||
|
||
## How it was made
|
||
|
||
- **Base:** `Qwen/Qwen3-1.7B` (Apache-2.0)
|
||
- **Data:** 3,116 `(description → rule)` pairs derived from [SigmaHQ](https://github.com/SigmaHQ/sigma) (MIT)
|
||
- **Method:** LoRA SFT via [MLX-LM](https://github.com/ml-explore/mlx-lm) — 8 layers, 600 iters, batch 1, seq 512, trained locally on an Apple M4 (peak 4.8 GB)
|
||
- **Eval/reward:** graded verifiable reward using pySigma + the Splunk backend as the correctness oracle
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model_id = "e12ex2/Qwen3-1.7B-SigmaRL" # <-- set to your repo
|
||
tok = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||
|
||
SYSTEM = ("You are a detection engineer. Given a threat description, output a single "
|
||
"valid Sigma detection rule in YAML. Output only the YAML, no prose, no code fences.")
|
||
messages = [
|
||
{"role": "system", "content": SYSTEM},
|
||
{"role": "user", "content": "Write a Sigma rule that detects: powershell launched with a base64 -enc command"},
|
||
]
|
||
text = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
|
||
out = model.generate(**tok(text, return_tensors="pt"), max_new_tokens=512)
|
||
print(tok.decode(out[0], skip_special_tokens=True))
|
||
```
|
||
|
||
## Limitations
|
||
|
||
- **Prototype.** ~54% of outputs are still not fully valid Sigma (often valid YAML with an incorrect schema).
|
||
- Trained with **512-token truncation**, so long rules were cut — a v0.2 with longer context is planned.
|
||
- Validity is measured by **backend compilation**, not by running the rules against real logs. True-positive / false-positive evaluation against labeled telemetry is future work.
|
||
- **Single-turn.** Generated rules should be reviewed by a detection engineer before deployment.
|
||
|
||
## Reward function & training code
|
||
|
||
The graded verifiable reward, data pipeline, and training scripts are open. Rules are scored, not hand-labeled — the same reward drives both eval and (planned) reinforcement learning.
|