初始化项目，由ModelHub XC社区提供模型

Model: Kuyash/teptez-ai Source: Original Platform
2026-07-15 14:53:09 +08:00
commit dec1a19935
14 changed files with 1124 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,335 @@
+---
+language: en
+license: apache-2.0
+base_model: Qwen/Qwen2.5-Coder-7B-Instruct
+tags:
+  - security
+  - sast
+  - code-analysis
+  - vulnerability-detection
+  - triage
+  - gguf
+pipeline_tag: text-generation
+---
+
+# teptez-ai
+
+**SAST finding triage model** — fine-tuned on real production security scan data to classify Static Application Security Testing findings as true positives, false positives, or uncertain, with CWE labels, confidence scores, and remediation guidance.
+
+---
+
+## Overview
+
+Rule-based SAST tools generate enormous volumes of findings, a significant portion of which are false positives. Security analysts spend hours triaging noise instead of fixing real vulnerabilities. teptez-ai is a 7B-parameter LLM fine-tuned specifically to automate this triage step.
+
+Given a SAST finding (title, CWE, severity, code snippet, taint flow), teptez-ai returns a structured JSON verdict:
+
+- **true_positive** — finding is real, exploit path exists
+- **false_positive** — finding is noise, safe to suppress
+- **uncertain** — insufficient context, escalate to analyst
+
+Fine-tuned on production findings from the [Teptez](https://teptez.io) security platform — real codebases, real scan data, real analyst labels.
+
+### Key specs
+
+| Property | Value |
+|---|---|
+| Base model | Qwen2.5-Coder-7B-Instruct |
+| Quantization | GGUF Q4_K_M |
+| Model size | ~4.7 GB |
+| Inference speed | ~100 tok/s (RTX 3090 24GB) |
+| Context window | 8192 tokens |
+| License | Apache 2.0 |
+
+---
+
+## Benchmark Results
+
+Evaluated against **OWASP Benchmark v1.2** — the standard industry benchmark for SAST tools — using the official **Youden's J statistic** (`J = TPR − FPR`).
+
+> J = 0.0 is random. J = 1.0 is perfect. Open-source SAST tools typically score 0.30–0.45.
+
+### Head-to-head vs base model
+
+| Model | TPR | FPR | Youden J | vs base |
+|---|---|---|---|---|
+| **teptez-ai (Q4_K_M)** | **0.68** | **0.57** | **0.109** | **+0.048 (+79%)** |
+| qwen2.5-coder-7b (base) | 0.66 | 0.60 | 0.061 | — |
+
+Fine-tuning delivers a **79% relative improvement** in Youden's J over the base model, primarily by cutting the false positive rate from 0.60 to 0.57 across the full benchmark.
+
+### Per-category breakdown
+
+| CWE Category | teptez-ai J | base J | Delta |
+|---|---|---|---|
+| Command Injection (CWE-78) | **0.40** | 0.22 | +0.18 |
+| SQL Injection (CWE-89) | **0.33** | 0.18 | +0.15 |
+| XSS (CWE-79) | **0.28** | 0.12 | +0.16 |
+| Path Traversal (CWE-22) | **0.22** | 0.09 | +0.13 |
+| Weak Randomness (CWE-330) | **0.13** | 0.05 | +0.08 |
+| Crypto/Hash (CWE-327/328) | 0.00 | 0.01 | -0.01 |
+| Auth/Authz (CWE-862/639) | 0.02 | 0.01 | +0.01 |
+| Timing (CWE-208) | 0.05 | 0.04 | +0.01 |
+| **Secure Cookie (CWE-614)** | **-0.08** | **0.52** | **-0.60 ⚠️** |
+
+**Strong on injection classes.** teptez-ai significantly outperforms the base model across all injection-type CWEs (cmdi/sqli/xss/path/weakrand). These are the highest-volume SAST categories in real codebases.
+
+**Securecookie regression.** CWE-614 (missing HttpOnly/Secure flags) shows a severe regression vs the base model. **Do not use teptez-ai to triage cookie security findings.** This is a known training artifact being fixed in the next round.
+
+**Dead categories.** Crypto, authz, and timing categories have near-zero Youden J on both models — 7B parameters are insufficient for these without full class context. Escalate to frontier models or human analysts.
+
+### Production run
+
+On **369 real production SAST findings** from live codebases:
+- **17% rejected as false positive** (~63 findings suppressed)
+- Injection-class findings: majority of suppressions, generally accurate
+- Authz/crypto findings: some wrong suppressions (do not enable for these categories)
+
+---
+
+## Usage
+
+### Recommended architecture
+
+Use teptez-ai as a **gated FP suppressor**, not a confirmer:
+
+```
+Rule-engine finding
+        │
+        ▼
+Is CWE in injection classes?  ──No──▶  Keep finding (don't run model)
+        │ Yes
+        ▼
+Run teptez-ai with full function + taint context
+        │
+        ├── verdict: false_positive, confidence > 0.75  ──▶  Suppress finding
+        ├── verdict: true_positive                       ──▶  Keep finding
+        └── verdict: uncertain / confidence < 0.75      ──▶  Escalate to frontier model / analyst
+```
+
+**Only suppress on `false_positive`** — never on `true_positive`. The model is biased toward flagging (FPR 0.57), so a `false_positive` verdict is rare and higher-precision.
+
+**Injection-class CWEs only** (where Youden J ≥ 0.13):
+- CWE-78 Command Injection
+- CWE-79 Cross-Site Scripting
+- CWE-89 SQL Injection
+- CWE-22 Path Traversal
+- CWE-330 Weak Randomness
+
+**Never auto-suppress**:
+- CWE-614 Secure Cookie (regression — model worse than random)
+- CWE-327/328 Weak Crypto/Hash (near-zero J)
+- CWE-862/639 Auth/Authz/IDOR (near-zero J)
+- CWE-208 Timing Attacks (near-zero J)
+
+### Running with llama.cpp / Ollama
+
+```bash
+# Pull via Ollama
+ollama pull hf.co/Kuyash/teptez-ai:Q4_K_M
+
+# Or run directly with llama.cpp
+./llama-cli -m teptez-ai-Q4_K_M.gguf \
+  --temp 0.1 \
+  --top-p 0.9 \
+  -n 512 \
+  -p "<prompt>"
+```
+
+### Python integration
+
+```python
+import json
+import requests
+
+def triage_finding(finding: dict) -> dict:
+    prompt = f"""<|im_start|>system
+You are a SAST triage expert. Analyze this finding and return JSON with keys:
+verdict (true_positive|false_positive|uncertain), confidence (0.0-1.0),
+cwe (string), explanation (string), remediation (string).
+<|im_end|>
+<|im_start|>user
+Finding: {finding['title']}
+CWE: {finding.get('cwe_id', 'unknown')}
+Severity: {finding.get('severity', 'MEDIUM')}
+Code:
+{finding.get('code_snippet', '')}
+
+Taint flow: {finding.get('data_flow', 'not available')}
+<|im_end|>
+<|im_start|>assistant
+"""
+    response = requests.post("http://localhost:11434/api/generate", json={
+        "model": "teptez-ai",
+        "prompt": prompt,
+        "stream": False,
+        "options": {"temperature": 0.1}
+    })
+    text = response.json()["response"].strip()
+    # Strip markdown fences if present
+    if text.startswith("```"):
+        text = text.split("```")[1]
+        if text.startswith("json"):
+            text = text[4:]
+    return json.loads(text)
+
+# Gate: only run on injection CWEs
+INJECTION_CWES = {"CWE-78", "CWE-79", "CWE-89", "CWE-22", "CWE-330"}
+
+def should_suppress(finding: dict) -> bool:
+    cwe = finding.get("cwe_id", "")
+    if cwe not in INJECTION_CWES:
+        return False  # don't touch non-injection
+    result = triage_finding(finding)
+    return (
+        result.get("verdict") == "false_positive"
+        and result.get("confidence", 0) >= 0.75
+    )
+```
+
+---
+
+## Input / Output Format
+
+### Prompt template
+
+```
+<|im_start|>system
+You are a SAST triage expert. Analyze this finding and return JSON.
+<|im_end|>
+<|im_start|>user
+Finding: {title}
+CWE: {cwe_id}
+Severity: {severity}
+Code:
+{code_snippet}
+
+Taint flow: {data_flow}
+<|im_end|>
+<|im_start|>assistant
+```
+
+**Tips for best results:**
+- Provide the **full function**, not just the flagged line — avoids "insufficient context" errors
+- Include taint flow when available (source → sink path from your SAST tool)
+- Keep code under 2048 tokens; truncate from the bottom if needed
+
+### Output schema
+
+```json
+{
+  "verdict": "true_positive" | "false_positive" | "uncertain",
+  "confidence": 0.85,
+  "cwe": "CWE-89",
+  "explanation": "User input from request.getParameter() flows directly into a string-concatenated SQL query with no parameterization or escaping.",
+  "remediation": "Replace string concatenation with a PreparedStatement: `conn.prepareStatement(\"SELECT * FROM users WHERE id = ?\")` and bind the parameter with `stmt.setString(1, userId)`."
+}
+```
+
+| Field | Type | Description |
+|---|---|---|
+| `verdict` | string | `true_positive`, `false_positive`, or `uncertain` |
+| `confidence` | float | 0.0–1.0; scores < 0.75 should be treated as uncertain |
+| `cwe` | string | Classified CWE identifier |
+| `explanation` | string | Why the model reached this verdict |
+| `remediation` | string | Concrete fix recommendation |
+
+---
+
+## Limitations
+
+### Known issues (as of current release)
+
+**CWE-614 Secure Cookie — severe regression.**
+teptez-ai scores Youden J = −0.08 on secure cookie findings, compared to 0.52 for the base model. This is a catastrophic regression caused by training data imbalance. Do not use teptez-ai for HttpOnly/Secure flag findings until this is fixed.
+
+**High overall FPR (0.57).**
+The model over-flags — it sees vulnerability in safe code, especially in crypto, auth, and cookie-related code patterns. A `false_positive` verdict is more reliable than a `true_positive` verdict because it swims against the model's bias.
+
+**Dead categories (crypto/authz/timing).**
+CWE-327/328/614/862/639/208 have near-zero Youden J. The model lacks sufficient training signal for these categories. Use a frontier model (Claude, GPT-4o, Gemini) or a human analyst for these.
+
+**7B parameter ceiling.**
+Subtle IDOR, broken access control, and privilege escalation patterns require understanding class hierarchy, authentication flow, and business logic across multiple files. A 7B model with single-function context cannot reliably detect these.
+
+**GGUF Q4_K_M quantization.**
+~4-bit quantization introduces slight accuracy loss vs fp16. For maximum accuracy, use the Q8_0 variant (if available) or the full fp16 model.
+
+**Snippet-only inputs fail.**
+If you pass only the flagged 1–3 lines without the surrounding function, the model frequently returns `uncertain` with "insufficient context." Always include the full function body.
+
+---
+
+## Reproducing the Benchmark
+
+```bash
+# 1. Clone OWASP Benchmark
+git clone https://github.com/OWASP-Benchmark/BenchmarkJava
+cd BenchmarkJava && mvn package -DskipTests
+
+# 2. Run evaluation (requires teptez-ai running on Ollama)
+python salad/eval/owasp_eval.py      # stratified sample → results.jsonl
+python salad/eval/owasp_score.py     # Youden J + per-category table
+python salad/eval/owasp_compare.py   # head-to-head vs base model
+```
+
+The eval script uses a stratified sample of OWASP BenchmarkJava test cases, covering all CWE categories proportionally. Scoring follows the official OWASP methodology (Youden's J = TPR − FPR).
+
+---
+
+## Roadmap
+
+The following improvements are planned for the next fine-tuning round:
+
+### Round 2 targets
+
+| Improvement | Target metric |
+|---|---|
+| Flood training with safe-code negatives (50/50 balance) | FPR < 0.30 |
+| Fix securecookie regression (restore CWE-614 training data) | J(CWE-614) > 0.40 |
+| Add dead-category examples (crypto/authz/timing) | J(CWE-327/328) > 0.10 |
+| Calibrate confidence score (train explicit `uncertain` label) | Confidence Brier score < 0.15 |
+| Full-function context at train AND inference | Reduce "uncertain" on short snippets |
+
+**Root cause of FPR problem:** Current training set is vuln-heavy (more vulnerable examples than safe ones). The model learned to flag aggressively. Rebalancing to 50/50 with explicit safe variants (parameterized SQL, escaped HTML, validated paths, compare_digest timing-safe comparisons, role-checked endpoints, strong ciphers) is the single highest-leverage fix.
+
+**Securecookie fix:** Restore the original training examples for CWE-614 that were accidentally dropped. Mix with new negative examples. Lower learning rate for this category to avoid forgetting again.
+
+**No catastrophic forgetting:** Round 2 will mix old injection data with new negatives and use a lower learning rate on the balanced set, following standard continual learning practice.
+
+### Round 3 vision
+
+- Full-function + cross-file taint context (requires longer context fine-tune)
+- Multi-label output (multiple CWEs per finding)
+- Confidence calibration verified against held-out production labels
+- Youden J > 0.25 across all injection categories
+- FPR < 0.30 overall
+
+---
+
+## Citation
+
+If you use teptez-ai in your research or tooling, please cite:
+
+```bibtex
+@misc{teptez-ai-2026,
+  title     = {teptez-ai: A Fine-Tuned LLM for SAST Finding Triage},
+  author    = {Teptez Security},
+  year      = {2026},
+  publisher = {HuggingFace},
+  url       = {https://huggingface.co/Kuyash/teptez-ai}
+}
+```
+
+Evaluated against [OWASP Benchmark v1.2](https://owasp.org/www-project-benchmark/) using the official Youden's J scoring methodology.
+
+---
+
+## Related
+
+- [OWASP Benchmark](https://owasp.org/www-project-benchmark/) — the benchmark used for evaluation
+- [Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) — the base model
+
+---
+
+*teptez-ai is a security research model. Results may vary across codebases and languages. Always have a human analyst review suppressed findings in critical security contexts.*