336 lines
12 KiB
Markdown
336 lines
12 KiB
Markdown
---
|
||
language: en
|
||
license: apache-2.0
|
||
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
|
||
tags:
|
||
- security
|
||
- sast
|
||
- code-analysis
|
||
- vulnerability-detection
|
||
- triage
|
||
- gguf
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# teptez-ai
|
||
|
||
**SAST finding triage model** — fine-tuned on real production security scan data to classify Static Application Security Testing findings as true positives, false positives, or uncertain, with CWE labels, confidence scores, and remediation guidance.
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
Rule-based SAST tools generate enormous volumes of findings, a significant portion of which are false positives. Security analysts spend hours triaging noise instead of fixing real vulnerabilities. teptez-ai is a 7B-parameter LLM fine-tuned specifically to automate this triage step.
|
||
|
||
Given a SAST finding (title, CWE, severity, code snippet, taint flow), teptez-ai returns a structured JSON verdict:
|
||
|
||
- **true_positive** — finding is real, exploit path exists
|
||
- **false_positive** — finding is noise, safe to suppress
|
||
- **uncertain** — insufficient context, escalate to analyst
|
||
|
||
Fine-tuned on production findings from the [Teptez](https://teptez.io) security platform — real codebases, real scan data, real analyst labels.
|
||
|
||
### Key specs
|
||
|
||
| Property | Value |
|
||
|---|---|
|
||
| Base model | Qwen2.5-Coder-7B-Instruct |
|
||
| Quantization | GGUF Q4_K_M |
|
||
| Model size | ~4.7 GB |
|
||
| Inference speed | ~100 tok/s (RTX 3090 24GB) |
|
||
| Context window | 8192 tokens |
|
||
| License | Apache 2.0 |
|
||
|
||
---
|
||
|
||
## Benchmark Results
|
||
|
||
Evaluated against **OWASP Benchmark v1.2** — the standard industry benchmark for SAST tools — using the official **Youden's J statistic** (`J = TPR − FPR`).
|
||
|
||
> J = 0.0 is random. J = 1.0 is perfect. Open-source SAST tools typically score 0.30–0.45.
|
||
|
||
### Head-to-head vs base model
|
||
|
||
| Model | TPR | FPR | Youden J | vs base |
|
||
|---|---|---|---|---|
|
||
| **teptez-ai (Q4_K_M)** | **0.68** | **0.57** | **0.109** | **+0.048 (+79%)** |
|
||
| qwen2.5-coder-7b (base) | 0.66 | 0.60 | 0.061 | — |
|
||
|
||
Fine-tuning delivers a **79% relative improvement** in Youden's J over the base model, primarily by cutting the false positive rate from 0.60 to 0.57 across the full benchmark.
|
||
|
||
### Per-category breakdown
|
||
|
||
| CWE Category | teptez-ai J | base J | Delta |
|
||
|---|---|---|---|
|
||
| Command Injection (CWE-78) | **0.40** | 0.22 | +0.18 |
|
||
| SQL Injection (CWE-89) | **0.33** | 0.18 | +0.15 |
|
||
| XSS (CWE-79) | **0.28** | 0.12 | +0.16 |
|
||
| Path Traversal (CWE-22) | **0.22** | 0.09 | +0.13 |
|
||
| Weak Randomness (CWE-330) | **0.13** | 0.05 | +0.08 |
|
||
| Crypto/Hash (CWE-327/328) | 0.00 | 0.01 | -0.01 |
|
||
| Auth/Authz (CWE-862/639) | 0.02 | 0.01 | +0.01 |
|
||
| Timing (CWE-208) | 0.05 | 0.04 | +0.01 |
|
||
| **Secure Cookie (CWE-614)** | **-0.08** | **0.52** | **-0.60 ⚠️** |
|
||
|
||
**Strong on injection classes.** teptez-ai significantly outperforms the base model across all injection-type CWEs (cmdi/sqli/xss/path/weakrand). These are the highest-volume SAST categories in real codebases.
|
||
|
||
**Securecookie regression.** CWE-614 (missing HttpOnly/Secure flags) shows a severe regression vs the base model. **Do not use teptez-ai to triage cookie security findings.** This is a known training artifact being fixed in the next round.
|
||
|
||
**Dead categories.** Crypto, authz, and timing categories have near-zero Youden J on both models — 7B parameters are insufficient for these without full class context. Escalate to frontier models or human analysts.
|
||
|
||
### Production run
|
||
|
||
On **369 real production SAST findings** from live codebases:
|
||
- **17% rejected as false positive** (~63 findings suppressed)
|
||
- Injection-class findings: majority of suppressions, generally accurate
|
||
- Authz/crypto findings: some wrong suppressions (do not enable for these categories)
|
||
|
||
---
|
||
|
||
## Usage
|
||
|
||
### Recommended architecture
|
||
|
||
Use teptez-ai as a **gated FP suppressor**, not a confirmer:
|
||
|
||
```
|
||
Rule-engine finding
|
||
│
|
||
▼
|
||
Is CWE in injection classes? ──No──▶ Keep finding (don't run model)
|
||
│ Yes
|
||
▼
|
||
Run teptez-ai with full function + taint context
|
||
│
|
||
├── verdict: false_positive, confidence > 0.75 ──▶ Suppress finding
|
||
├── verdict: true_positive ──▶ Keep finding
|
||
└── verdict: uncertain / confidence < 0.75 ──▶ Escalate to frontier model / analyst
|
||
```
|
||
|
||
**Only suppress on `false_positive`** — never on `true_positive`. The model is biased toward flagging (FPR 0.57), so a `false_positive` verdict is rare and higher-precision.
|
||
|
||
**Injection-class CWEs only** (where Youden J ≥ 0.13):
|
||
- CWE-78 Command Injection
|
||
- CWE-79 Cross-Site Scripting
|
||
- CWE-89 SQL Injection
|
||
- CWE-22 Path Traversal
|
||
- CWE-330 Weak Randomness
|
||
|
||
**Never auto-suppress**:
|
||
- CWE-614 Secure Cookie (regression — model worse than random)
|
||
- CWE-327/328 Weak Crypto/Hash (near-zero J)
|
||
- CWE-862/639 Auth/Authz/IDOR (near-zero J)
|
||
- CWE-208 Timing Attacks (near-zero J)
|
||
|
||
### Running with llama.cpp / Ollama
|
||
|
||
```bash
|
||
# Pull via Ollama
|
||
ollama pull hf.co/Kuyash/teptez-ai:Q4_K_M
|
||
|
||
# Or run directly with llama.cpp
|
||
./llama-cli -m teptez-ai-Q4_K_M.gguf \
|
||
--temp 0.1 \
|
||
--top-p 0.9 \
|
||
-n 512 \
|
||
-p "<prompt>"
|
||
```
|
||
|
||
### Python integration
|
||
|
||
```python
|
||
import json
|
||
import requests
|
||
|
||
def triage_finding(finding: dict) -> dict:
|
||
prompt = f"""<|im_start|>system
|
||
You are a SAST triage expert. Analyze this finding and return JSON with keys:
|
||
verdict (true_positive|false_positive|uncertain), confidence (0.0-1.0),
|
||
cwe (string), explanation (string), remediation (string).
|
||
<|im_end|>
|
||
<|im_start|>user
|
||
Finding: {finding['title']}
|
||
CWE: {finding.get('cwe_id', 'unknown')}
|
||
Severity: {finding.get('severity', 'MEDIUM')}
|
||
Code:
|
||
{finding.get('code_snippet', '')}
|
||
|
||
Taint flow: {finding.get('data_flow', 'not available')}
|
||
<|im_end|>
|
||
<|im_start|>assistant
|
||
"""
|
||
response = requests.post("http://localhost:11434/api/generate", json={
|
||
"model": "teptez-ai",
|
||
"prompt": prompt,
|
||
"stream": False,
|
||
"options": {"temperature": 0.1}
|
||
})
|
||
text = response.json()["response"].strip()
|
||
# Strip markdown fences if present
|
||
if text.startswith("```"):
|
||
text = text.split("```")[1]
|
||
if text.startswith("json"):
|
||
text = text[4:]
|
||
return json.loads(text)
|
||
|
||
# Gate: only run on injection CWEs
|
||
INJECTION_CWES = {"CWE-78", "CWE-79", "CWE-89", "CWE-22", "CWE-330"}
|
||
|
||
def should_suppress(finding: dict) -> bool:
|
||
cwe = finding.get("cwe_id", "")
|
||
if cwe not in INJECTION_CWES:
|
||
return False # don't touch non-injection
|
||
result = triage_finding(finding)
|
||
return (
|
||
result.get("verdict") == "false_positive"
|
||
and result.get("confidence", 0) >= 0.75
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## Input / Output Format
|
||
|
||
### Prompt template
|
||
|
||
```
|
||
<|im_start|>system
|
||
You are a SAST triage expert. Analyze this finding and return JSON.
|
||
<|im_end|>
|
||
<|im_start|>user
|
||
Finding: {title}
|
||
CWE: {cwe_id}
|
||
Severity: {severity}
|
||
Code:
|
||
{code_snippet}
|
||
|
||
Taint flow: {data_flow}
|
||
<|im_end|>
|
||
<|im_start|>assistant
|
||
```
|
||
|
||
**Tips for best results:**
|
||
- Provide the **full function**, not just the flagged line — avoids "insufficient context" errors
|
||
- Include taint flow when available (source → sink path from your SAST tool)
|
||
- Keep code under 2048 tokens; truncate from the bottom if needed
|
||
|
||
### Output schema
|
||
|
||
```json
|
||
{
|
||
"verdict": "true_positive" | "false_positive" | "uncertain",
|
||
"confidence": 0.85,
|
||
"cwe": "CWE-89",
|
||
"explanation": "User input from request.getParameter() flows directly into a string-concatenated SQL query with no parameterization or escaping.",
|
||
"remediation": "Replace string concatenation with a PreparedStatement: `conn.prepareStatement(\"SELECT * FROM users WHERE id = ?\")` and bind the parameter with `stmt.setString(1, userId)`."
|
||
}
|
||
```
|
||
|
||
| Field | Type | Description |
|
||
|---|---|---|
|
||
| `verdict` | string | `true_positive`, `false_positive`, or `uncertain` |
|
||
| `confidence` | float | 0.0–1.0; scores < 0.75 should be treated as uncertain |
|
||
| `cwe` | string | Classified CWE identifier |
|
||
| `explanation` | string | Why the model reached this verdict |
|
||
| `remediation` | string | Concrete fix recommendation |
|
||
|
||
---
|
||
|
||
## Limitations
|
||
|
||
### Known issues (as of current release)
|
||
|
||
**CWE-614 Secure Cookie — severe regression.**
|
||
teptez-ai scores Youden J = −0.08 on secure cookie findings, compared to 0.52 for the base model. This is a catastrophic regression caused by training data imbalance. Do not use teptez-ai for HttpOnly/Secure flag findings until this is fixed.
|
||
|
||
**High overall FPR (0.57).**
|
||
The model over-flags — it sees vulnerability in safe code, especially in crypto, auth, and cookie-related code patterns. A `false_positive` verdict is more reliable than a `true_positive` verdict because it swims against the model's bias.
|
||
|
||
**Dead categories (crypto/authz/timing).**
|
||
CWE-327/328/614/862/639/208 have near-zero Youden J. The model lacks sufficient training signal for these categories. Use a frontier model (Claude, GPT-4o, Gemini) or a human analyst for these.
|
||
|
||
**7B parameter ceiling.**
|
||
Subtle IDOR, broken access control, and privilege escalation patterns require understanding class hierarchy, authentication flow, and business logic across multiple files. A 7B model with single-function context cannot reliably detect these.
|
||
|
||
**GGUF Q4_K_M quantization.**
|
||
~4-bit quantization introduces slight accuracy loss vs fp16. For maximum accuracy, use the Q8_0 variant (if available) or the full fp16 model.
|
||
|
||
**Snippet-only inputs fail.**
|
||
If you pass only the flagged 1–3 lines without the surrounding function, the model frequently returns `uncertain` with "insufficient context." Always include the full function body.
|
||
|
||
---
|
||
|
||
## Reproducing the Benchmark
|
||
|
||
```bash
|
||
# 1. Clone OWASP Benchmark
|
||
git clone https://github.com/OWASP-Benchmark/BenchmarkJava
|
||
cd BenchmarkJava && mvn package -DskipTests
|
||
|
||
# 2. Run evaluation (requires teptez-ai running on Ollama)
|
||
python salad/eval/owasp_eval.py # stratified sample → results.jsonl
|
||
python salad/eval/owasp_score.py # Youden J + per-category table
|
||
python salad/eval/owasp_compare.py # head-to-head vs base model
|
||
```
|
||
|
||
The eval script uses a stratified sample of OWASP BenchmarkJava test cases, covering all CWE categories proportionally. Scoring follows the official OWASP methodology (Youden's J = TPR − FPR).
|
||
|
||
---
|
||
|
||
## Roadmap
|
||
|
||
The following improvements are planned for the next fine-tuning round:
|
||
|
||
### Round 2 targets
|
||
|
||
| Improvement | Target metric |
|
||
|---|---|
|
||
| Flood training with safe-code negatives (50/50 balance) | FPR < 0.30 |
|
||
| Fix securecookie regression (restore CWE-614 training data) | J(CWE-614) > 0.40 |
|
||
| Add dead-category examples (crypto/authz/timing) | J(CWE-327/328) > 0.10 |
|
||
| Calibrate confidence score (train explicit `uncertain` label) | Confidence Brier score < 0.15 |
|
||
| Full-function context at train AND inference | Reduce "uncertain" on short snippets |
|
||
|
||
**Root cause of FPR problem:** Current training set is vuln-heavy (more vulnerable examples than safe ones). The model learned to flag aggressively. Rebalancing to 50/50 with explicit safe variants (parameterized SQL, escaped HTML, validated paths, compare_digest timing-safe comparisons, role-checked endpoints, strong ciphers) is the single highest-leverage fix.
|
||
|
||
**Securecookie fix:** Restore the original training examples for CWE-614 that were accidentally dropped. Mix with new negative examples. Lower learning rate for this category to avoid forgetting again.
|
||
|
||
**No catastrophic forgetting:** Round 2 will mix old injection data with new negatives and use a lower learning rate on the balanced set, following standard continual learning practice.
|
||
|
||
### Round 3 vision
|
||
|
||
- Full-function + cross-file taint context (requires longer context fine-tune)
|
||
- Multi-label output (multiple CWEs per finding)
|
||
- Confidence calibration verified against held-out production labels
|
||
- Youden J > 0.25 across all injection categories
|
||
- FPR < 0.30 overall
|
||
|
||
---
|
||
|
||
## Citation
|
||
|
||
If you use teptez-ai in your research or tooling, please cite:
|
||
|
||
```bibtex
|
||
@misc{teptez-ai-2026,
|
||
title = {teptez-ai: A Fine-Tuned LLM for SAST Finding Triage},
|
||
author = {Teptez Security},
|
||
year = {2026},
|
||
publisher = {HuggingFace},
|
||
url = {https://huggingface.co/Kuyash/teptez-ai}
|
||
}
|
||
```
|
||
|
||
Evaluated against [OWASP Benchmark v1.2](https://owasp.org/www-project-benchmark/) using the official Youden's J scoring methodology.
|
||
|
||
---
|
||
|
||
## Related
|
||
|
||
- [OWASP Benchmark](https://owasp.org/www-project-benchmark/) — the benchmark used for evaluation
|
||
- [Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) — the base model
|
||
|
||
---
|
||
|
||
*teptez-ai is a security research model. Results may vary across codebases and languages. Always have a human analyst review suppressed findings in critical security contexts.*
|