73 lines
3.2 KiB
Markdown
73 lines
3.2 KiB
Markdown
|
---
|
|||
|
|
library_name: mlx
|
|||
|
|
license: apache-2.0
|
|||
|
|
license_link: https://huggingface.co/Qwen/Qwen3-1.7B/blob/main/LICENSE
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
base_model: Qwen/Qwen3-1.7B
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
tags:
|
|||
|
|
- mlx
|
|||
|
|
- sigma
|
|||
|
|
- detection-engineering
|
|||
|
|
- cybersecurity
|
|||
|
|
- blue-team
|
|||
|
|
- security
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen3-1.7B-SigmaRL (v0.1 · prototype)
|
|||
|
|
|
|||
|
|
Threat description → **Sigma detection rule**. A small, **defensive** detection-engineering model: a LoRA fine-tune of [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) that writes valid, backend-compilable [Sigma](https://github.com/SigmaHQ/sigma) rules from natural-language threat descriptions.
|
|||
|
|
|
|||
|
|
> **Defensive use only.** This model generates blue-team detection rules. It is not designed or intended for offensive security.
|
|||
|
|
|
|||
|
|
## Results (held-out, n=24)
|
|||
|
|
|
|||
|
|
Same 24 held-out prompts, scored by a **graded verifiable reward**: parse YAML → valid Sigma schema (pySigma) → compiles to a Splunk query (pySigma-backend-splunk).
|
|||
|
|
|
|||
|
|
| metric | base Qwen3-1.7B | **this model (SFT)** |
|
|||
|
|
|---|---|---|
|
|||
|
|
| valid + compilable rule rate | 0.0% | **45.8%** |
|
|||
|
|
| mean reward (0–0.6) | 0.10 | **0.36** |
|
|||
|
|
| tier breakdown (0.0 / 0.2 / 0.6) | 12 / 12 / 0 | 3 / 10 / 11 |
|
|||
|
|
|
|||
|
|
The base model **never once** produced a valid, compilable rule; after fine-tuning, nearly half of outputs are valid and Splunk-compilable.
|
|||
|
|
|
|||
|
|
## How it was made
|
|||
|
|
|
|||
|
|
- **Base:** `Qwen/Qwen3-1.7B` (Apache-2.0)
|
|||
|
|
- **Data:** 3,116 `(description → rule)` pairs derived from [SigmaHQ](https://github.com/SigmaHQ/sigma) (MIT)
|
|||
|
|
- **Method:** LoRA SFT via [MLX-LM](https://github.com/ml-explore/mlx-lm) — 8 layers, 600 iters, batch 1, seq 512, trained locally on an Apple M4 (peak 4.8 GB)
|
|||
|
|
- **Eval/reward:** graded verifiable reward using pySigma + the Splunk backend as the correctness oracle
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
model_id = "e12ex2/Qwen3-1.7B-SigmaRL" # <-- set to your repo
|
|||
|
|
tok = AutoTokenizer.from_pretrained(model_id)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(model_id)
|
|||
|
|
|
|||
|
|
SYSTEM = ("You are a detection engineer. Given a threat description, output a single "
|
|||
|
|
"valid Sigma detection rule in YAML. Output only the YAML, no prose, no code fences.")
|
|||
|
|
messages = [
|
|||
|
|
{"role": "system", "content": SYSTEM},
|
|||
|
|
{"role": "user", "content": "Write a Sigma rule that detects: powershell launched with a base64 -enc command"},
|
|||
|
|
]
|
|||
|
|
text = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
|
|||
|
|
out = model.generate(**tok(text, return_tensors="pt"), max_new_tokens=512)
|
|||
|
|
print(tok.decode(out[0], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- **Prototype.** ~54% of outputs are still not fully valid Sigma (often valid YAML with an incorrect schema).
|
|||
|
|
- Trained with **512-token truncation**, so long rules were cut — a v0.2 with longer context is planned.
|
|||
|
|
- Validity is measured by **backend compilation**, not by running the rules against real logs. True-positive / false-positive evaluation against labeled telemetry is future work.
|
|||
|
|
- **Single-turn.** Generated rules should be reviewed by a detection engineer before deployment.
|
|||
|
|
|
|||
|
|
## Reward function & training code
|
|||
|
|
|
|||
|
|
The graded verifiable reward, data pipeline, and training scripts are open. Rules are scored, not hand-labeled — the same reward drives both eval and (planned) reinforcement learning.
|