183 lines
5.7 KiB
Markdown
183 lines
5.7 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
library_name: pytorch
|
||
pipeline_tag: text-generation
|
||
base_model: sabari2005/cyberslm-base
|
||
tags:
|
||
- cybersecurity
|
||
- security
|
||
- small-language-model
|
||
- instruction-tuned
|
||
- sft
|
||
- causal-lm
|
||
---
|
||
|
||
# CyberSLM-instruct — 33.5M-parameter cybersecurity assistant
|
||
|
||
Instruction-tuned from
|
||
[**sabari2005/cyberslm-base**](https://huggingface.co/sabari2005/cyberslm-base)
|
||
on 23,540 cybersecurity conversations.
|
||
|
||
**Code:** [github.com/Sabari2005/cyberslm](https://github.com/Sabari2005/cyberslm)
|
||
|
||
## Read this before using it
|
||
|
||
At 33.5M parameters this model learned the **shape** of a good answer — markdown
|
||
structure, numbered steps, worked examples, mitigation sections — and is
|
||
**frequently wrong about the content**.
|
||
|
||
Measured over 8 greedy prompts:
|
||
|
||
| category | mean 8-gram repetition | stopped on EOS |
|
||
|---|---:|---:|
|
||
| security | 18.7% | 1 / 4 |
|
||
| general | 18.9% | 1 / 2 |
|
||
| code | 19.9% | 0 / 2 |
|
||
| **overall** | **19.0%** | **2 / 8** |
|
||
|
||
**Do not use this as a factual security reference. Do not run code it
|
||
generates.** It is a demonstration of a correct end-to-end training pipeline and
|
||
a baseline for scaling.
|
||
|
||
### Where it works
|
||
|
||
Well-covered in-domain questions come out correct and terminate cleanly:
|
||
|
||
> **What is SQL injection and how do I prevent it?**
|
||
>
|
||
> SQL injection (SQLi) is a security vulnerability that allows attackers to
|
||
> manipulate database queries by injecting malicious SQL code through input
|
||
> fields. It occurs when user-supplied data is improperly sanitized, allowing
|
||
> attackers to manipulate the database. For example, if an attacker submits a
|
||
> username like `admin' --` as the username, the query becomes:
|
||
> `SELECT * FROM users WHERE username = '[input]' AND password = '[input]'`.
|
||
|
||
> **What is a firewall?**
|
||
>
|
||
> A firewall is a network security device or software that monitors and controls
|
||
> incoming and outgoing network traffic based on predefined security rules. It
|
||
> acts as a barrier between trusted internal networks and untrusted external
|
||
> networks, such as the internet.
|
||
|
||
### Where it fails
|
||
|
||
All observed, reproducible, and the majority case:
|
||
|
||
* **Wrong content, confident tone.** Asked to contrast symmetric and asymmetric
|
||
encryption it answered about hashing and IKE, never addressing the question.
|
||
* **Circular definitions.** "A buffer overflow is a type of buffer overflow that
|
||
could lead to arbitrary code execution."
|
||
* **Topically-adjacent but wrong vocabulary.** Asked how to investigate a
|
||
phishing email it produced `SameSite` and `Strict` — real security terms,
|
||
wrong topic (they are cookie attributes).
|
||
* **Degenerate loops in code.** `port: The port to use` repeated to the token
|
||
limit.
|
||
* **Unreliable termination.** Only 2 of 8 prompts stopped on EOS; the rest ran
|
||
to the token limit.
|
||
|
||
These are consequences of scale, not of the training run — the loss curve is
|
||
healthy and the pipeline is machine-verified (35 architecture checks, 173 tests).
|
||
|
||
## Model details
|
||
|
||
| | |
|
||
|---|---|
|
||
| parameters | 33,531,264 |
|
||
| architecture | 12 layers, d_model 384, 6 heads, SwiGLU 1024, RoPE, RMSNorm, tied head |
|
||
| context | 2048 |
|
||
| vocab | 32,000 (SentencePiece BPE) |
|
||
| base model | sabari2005/cyberslm-base |
|
||
| SFT data | 23,540 conversations, 15.1M supervised tokens |
|
||
| epochs | 3 (2,208 optimizer steps) |
|
||
| optimiser | AdamW, lr 2e-5, 3% warmup, cosine, bf16 |
|
||
| best val loss | 2.2627 (response tokens only) |
|
||
|
||
Loss is computed on assistant responses only; prompts are masked. 88% of tokens
|
||
in the SFT set are supervised.
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
pip install torch sentencepiece
|
||
git clone https://huggingface.co/sabari2005/cyberslm-instruct
|
||
cd cyberslm-instruct
|
||
python infer_chat.py --prompt "What is SQL injection and how do I prevent it?"
|
||
```
|
||
|
||
Interactive:
|
||
|
||
```bash
|
||
python infer_chat.py --interactive
|
||
```
|
||
|
||
Options:
|
||
|
||
```bash
|
||
python infer_chat.py \
|
||
--prompt "What is a buffer overflow?" \
|
||
--max-new-tokens 200 \
|
||
--temperature 0.0 # 0 = greedy, recommended for this model
|
||
```
|
||
|
||
### Prompt format
|
||
|
||
The model was trained on this exact layout, with a real BOS token id prepended
|
||
and EOS terminating each response:
|
||
|
||
```
|
||
### User:
|
||
{question}
|
||
|
||
### Assistant:
|
||
{response}<eos>
|
||
```
|
||
|
||
**Build prompts with the bundled formatter** (`infer_chat.py` does this).
|
||
Hand-assembling the string produces different token ids at every segment
|
||
boundary, because SentencePiece prepends a word-boundary marker per `encode()`
|
||
call — the model then sees something it was never trained on.
|
||
|
||
```python
|
||
import torch
|
||
from configs.sft_config import default_config
|
||
from data.prompt_formatter import PromptFormatter, Tokenizer
|
||
from model.cyberslm import CyberSLM
|
||
|
||
cfg = default_config()
|
||
cfg.tokenizer.model_path = "tokenizer/tokenizer.model"
|
||
cfg.model.max_seq_len = cfg.data.max_seq_len = 2048
|
||
|
||
tok = Tokenizer(cfg.tokenizer.model_path)
|
||
fmt = PromptFormatter(cfg=cfg, tokenizer=tok)
|
||
|
||
model = CyberSLM(cfg.model)
|
||
model.load_state_dict(torch.load("models/instruct.pt", map_location="cpu",
|
||
weights_only=False))
|
||
model.eval()
|
||
|
||
ids = fmt.format_for_inference({"messages": [{"role": "user",
|
||
"content": "What is XSS?"}]})
|
||
out = model.generate(torch.tensor([ids]), max_new_tokens=200,
|
||
temperature=0.0, eos_id=tok.eos_id)
|
||
print(tok.decode(out[0, len(ids):].tolist()))
|
||
```
|
||
|
||
Decoding uses a KV cache — roughly 50–70 tok/s on CPU.
|
||
|
||
## Intended use
|
||
|
||
Research into small language models; a scaling baseline; a demonstration of a
|
||
verified training pipeline. **Not** for security advice, incident response, code
|
||
generation, or anything where accuracy matters.
|
||
|
||
## Training data
|
||
|
||
Not published. Curated cybersecurity instruction data; not redistributed.
|
||
|
||
## License
|
||
|
||
Apache-2.0 for the code and weights. Verify licensing for downstream use against
|
||
the sources the data was curated from.
|