---
language: en
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
tags:
  - security
  - sast
  - code-analysis
  - vulnerability-detection
  - triage
  - gguf
pipeline_tag: text-generation
---

# teptez-ai

**SAST finding triage model** — fine-tuned on real production security scan data to classify Static Application Security Testing findings as true positives, false positives, or uncertain, with CWE labels, confidence scores, and remediation guidance.

---

## Overview

Rule-based SAST tools generate enormous volumes of findings, a significant portion of which are false positives. Security analysts spend hours triaging noise instead of fixing real vulnerabilities. teptez-ai is a 7B-parameter LLM fine-tuned specifically to automate this triage step.

Given a SAST finding (title, CWE, severity, code snippet, taint flow), teptez-ai returns a structured JSON verdict:

- **true_positive** — finding is real, exploit path exists
- **false_positive** — finding is noise, safe to suppress
- **uncertain** — insufficient context, escalate to analyst

Fine-tuned on production findings from the [Teptez](https://teptez.io) security platform — real codebases, real scan data, real analyst labels.

### Key specs

| Property | Value |
|---|---|
| Base model | Qwen2.5-Coder-7B-Instruct |
| Quantization | GGUF Q4_K_M |
| Model size | ~4.7 GB |
| Inference speed | ~100 tok/s (RTX 3090 24GB) |
| Context window | 8192 tokens |
| License | Apache 2.0 |

---

## Benchmark Results

Evaluated against **OWASP Benchmark v1.2** — the standard industry benchmark for SAST tools — using the official **Youden's J statistic** (`J = TPR − FPR`).

> J = 0.0 is random. J = 1.0 is perfect. Open-source SAST tools typically score 0.30–0.45.

### Head-to-head vs base model

| Model | TPR | FPR | Youden J | vs base |
|---|---|---|---|---|
| **teptez-ai (Q4_K_M)** | **0.68** | **0.57** | **0.109** | **+0.048 (+79%)** |
| qwen2.5-coder-7b (base) | 0.66 | 0.60 | 0.061 | — |

Fine-tuning delivers a **79% relative improvement** in Youden's J over the base model, primarily by cutting the false positive rate from 0.60 to 0.57 across the full benchmark.

### Per-category breakdown

| CWE Category | teptez-ai J | base J | Delta |
|---|---|---|---|
| Command Injection (CWE-78) | **0.40** | 0.22 | +0.18 |
| SQL Injection (CWE-89) | **0.33** | 0.18 | +0.15 |
| XSS (CWE-79) | **0.28** | 0.12 | +0.16 |
| Path Traversal (CWE-22) | **0.22** | 0.09 | +0.13 |
| Weak Randomness (CWE-330) | **0.13** | 0.05 | +0.08 |
| Crypto/Hash (CWE-327/328) | 0.00 | 0.01 | -0.01 |
| Auth/Authz (CWE-862/639) | 0.02 | 0.01 | +0.01 |
| Timing (CWE-208) | 0.05 | 0.04 | +0.01 |
| **Secure Cookie (CWE-614)** | **-0.08** | **0.52** | **-0.60 ⚠️** |

**Strong on injection classes.** teptez-ai significantly outperforms the base model across all injection-type CWEs (cmdi/sqli/xss/path/weakrand). These are the highest-volume SAST categories in real codebases.

**Securecookie regression.** CWE-614 (missing HttpOnly/Secure flags) shows a severe regression vs the base model. **Do not use teptez-ai to triage cookie security findings.** This is a known training artifact being fixed in the next round.

**Dead categories.** Crypto, authz, and timing categories have near-zero Youden J on both models — 7B parameters are insufficient for these without full class context. Escalate to frontier models or human analysts.

### Production run

On **369 real production SAST findings** from live codebases:
- **17% rejected as false positive** (~63 findings suppressed)
- Injection-class findings: majority of suppressions, generally accurate
- Authz/crypto findings: some wrong suppressions (do not enable for these categories)

---

## Usage

### Recommended architecture

Use teptez-ai as a **gated FP suppressor**, not a confirmer:

```
Rule-engine finding
        │
        ▼
Is CWE in injection classes?  ──No──▶  Keep finding (don't run model)
        │ Yes
        ▼
Run teptez-ai with full function + taint context
        │
        ├── verdict: false_positive, confidence > 0.75  ──▶  Suppress finding
        ├── verdict: true_positive                       ──▶  Keep finding
        └── verdict: uncertain / confidence < 0.75      ──▶  Escalate to frontier model / analyst
```

**Only suppress on `false_positive`** — never on `true_positive`. The model is biased toward flagging (FPR 0.57), so a `false_positive` verdict is rare and higher-precision.

**Injection-class CWEs only** (where Youden J ≥ 0.13):
- CWE-78 Command Injection
- CWE-79 Cross-Site Scripting
- CWE-89 SQL Injection
- CWE-22 Path Traversal
- CWE-330 Weak Randomness

**Never auto-suppress**:
- CWE-614 Secure Cookie (regression — model worse than random)
- CWE-327/328 Weak Crypto/Hash (near-zero J)
- CWE-862/639 Auth/Authz/IDOR (near-zero J)
- CWE-208 Timing Attacks (near-zero J)

### Running with llama.cpp / Ollama

```bash
# Pull via Ollama
ollama pull hf.co/Kuyash/teptez-ai:Q4_K_M

# Or run directly with llama.cpp
./llama-cli -m teptez-ai-Q4_K_M.gguf \
  --temp 0.1 \
  --top-p 0.9 \
  -n 512 \
  -p "<prompt>"
```

### Python integration

```python
import json
import requests

def triage_finding(finding: dict) -> dict:
    prompt = f"""<|im_start|>system
You are a SAST triage expert. Analyze this finding and return JSON with keys:
verdict (true_positive|false_positive|uncertain), confidence (0.0-1.0),
cwe (string), explanation (string), remediation (string).
<|im_end|>
<|im_start|>user
Finding: {finding['title']}
CWE: {finding.get('cwe_id', 'unknown')}
Severity: {finding.get('severity', 'MEDIUM')}
Code:
{finding.get('code_snippet', '')}

Taint flow: {finding.get('data_flow', 'not available')}
<|im_end|>
<|im_start|>assistant
"""
    response = requests.post("http://localhost:11434/api/generate", json={
        "model": "teptez-ai",
        "prompt": prompt,
        "stream": False,
        "options": {"temperature": 0.1}
    })
    text = response.json()["response"].strip()
    # Strip markdown fences if present
    if text.startswith("```"):
        text = text.split("```")[1]
        if text.startswith("json"):
            text = text[4:]
    return json.loads(text)

# Gate: only run on injection CWEs
INJECTION_CWES = {"CWE-78", "CWE-79", "CWE-89", "CWE-22", "CWE-330"}

def should_suppress(finding: dict) -> bool:
    cwe = finding.get("cwe_id", "")
    if cwe not in INJECTION_CWES:
        return False  # don't touch non-injection
    result = triage_finding(finding)
    return (
        result.get("verdict") == "false_positive"
        and result.get("confidence", 0) >= 0.75
    )
```

---

## Input / Output Format

### Prompt template

```
<|im_start|>system
You are a SAST triage expert. Analyze this finding and return JSON.
<|im_end|>
<|im_start|>user
Finding: {title}
CWE: {cwe_id}
Severity: {severity}
Code:
{code_snippet}

Taint flow: {data_flow}
<|im_end|>
<|im_start|>assistant
```

**Tips for best results:**
- Provide the **full function**, not just the flagged line — avoids "insufficient context" errors
- Include taint flow when available (source → sink path from your SAST tool)
- Keep code under 2048 tokens; truncate from the bottom if needed

### Output schema

```json
{
  "verdict": "true_positive" | "false_positive" | "uncertain",
  "confidence": 0.85,
  "cwe": "CWE-89",
  "explanation": "User input from request.getParameter() flows directly into a string-concatenated SQL query with no parameterization or escaping.",
  "remediation": "Replace string concatenation with a PreparedStatement: `conn.prepareStatement(\"SELECT * FROM users WHERE id = ?\")` and bind the parameter with `stmt.setString(1, userId)`."
}
```

| Field | Type | Description |
|---|---|---|
| `verdict` | string | `true_positive`, `false_positive`, or `uncertain` |
| `confidence` | float | 0.0–1.0; scores < 0.75 should be treated as uncertain |
| `cwe` | string | Classified CWE identifier |
| `explanation` | string | Why the model reached this verdict |
| `remediation` | string | Concrete fix recommendation |

---

## Limitations

### Known issues (as of current release)

**CWE-614 Secure Cookie — severe regression.**
teptez-ai scores Youden J = −0.08 on secure cookie findings, compared to 0.52 for the base model. This is a catastrophic regression caused by training data imbalance. Do not use teptez-ai for HttpOnly/Secure flag findings until this is fixed.

**High overall FPR (0.57).**
The model over-flags — it sees vulnerability in safe code, especially in crypto, auth, and cookie-related code patterns. A `false_positive` verdict is more reliable than a `true_positive` verdict because it swims against the model's bias.

**Dead categories (crypto/authz/timing).**
CWE-327/328/614/862/639/208 have near-zero Youden J. The model lacks sufficient training signal for these categories. Use a frontier model (Claude, GPT-4o, Gemini) or a human analyst for these.

**7B parameter ceiling.**
Subtle IDOR, broken access control, and privilege escalation patterns require understanding class hierarchy, authentication flow, and business logic across multiple files. A 7B model with single-function context cannot reliably detect these.

**GGUF Q4_K_M quantization.**
~4-bit quantization introduces slight accuracy loss vs fp16. For maximum accuracy, use the Q8_0 variant (if available) or the full fp16 model.

**Snippet-only inputs fail.**
If you pass only the flagged 1–3 lines without the surrounding function, the model frequently returns `uncertain` with "insufficient context." Always include the full function body.

---

## Reproducing the Benchmark

```bash
# 1. Clone OWASP Benchmark
git clone https://github.com/OWASP-Benchmark/BenchmarkJava
cd BenchmarkJava && mvn package -DskipTests

# 2. Run evaluation (requires teptez-ai running on Ollama)
python salad/eval/owasp_eval.py      # stratified sample → results.jsonl
python salad/eval/owasp_score.py     # Youden J + per-category table
python salad/eval/owasp_compare.py   # head-to-head vs base model
```

The eval script uses a stratified sample of OWASP BenchmarkJava test cases, covering all CWE categories proportionally. Scoring follows the official OWASP methodology (Youden's J = TPR − FPR).

---

## Roadmap

The following improvements are planned for the next fine-tuning round:

### Round 2 targets

| Improvement | Target metric |
|---|---|
| Flood training with safe-code negatives (50/50 balance) | FPR < 0.30 |
| Fix securecookie regression (restore CWE-614 training data) | J(CWE-614) > 0.40 |
| Add dead-category examples (crypto/authz/timing) | J(CWE-327/328) > 0.10 |
| Calibrate confidence score (train explicit `uncertain` label) | Confidence Brier score < 0.15 |
| Full-function context at train AND inference | Reduce "uncertain" on short snippets |

**Root cause of FPR problem:** Current training set is vuln-heavy (more vulnerable examples than safe ones). The model learned to flag aggressively. Rebalancing to 50/50 with explicit safe variants (parameterized SQL, escaped HTML, validated paths, compare_digest timing-safe comparisons, role-checked endpoints, strong ciphers) is the single highest-leverage fix.

**Securecookie fix:** Restore the original training examples for CWE-614 that were accidentally dropped. Mix with new negative examples. Lower learning rate for this category to avoid forgetting again.

**No catastrophic forgetting:** Round 2 will mix old injection data with new negatives and use a lower learning rate on the balanced set, following standard continual learning practice.

### Round 3 vision

- Full-function + cross-file taint context (requires longer context fine-tune)
- Multi-label output (multiple CWEs per finding)
- Confidence calibration verified against held-out production labels
- Youden J > 0.25 across all injection categories
- FPR < 0.30 overall

---

## Citation

If you use teptez-ai in your research or tooling, please cite:

```bibtex
@misc{teptez-ai-2026,
  title     = {teptez-ai: A Fine-Tuned LLM for SAST Finding Triage},
  author    = {Teptez Security},
  year      = {2026},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/Kuyash/teptez-ai}
}
```

Evaluated against [OWASP Benchmark v1.2](https://owasp.org/www-project-benchmark/) using the official Youden's J scoring methodology.

---

## Related

- [OWASP Benchmark](https://owasp.org/www-project-benchmark/) — the benchmark used for evaluation
- [Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) — the base model

---

*teptez-ai is a security research model. Results may vary across codebases and languages. Always have a human analyst review suppressed findings in critical security contexts.*