cyberslm-instruct/README.md

---
license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
base_model: sabari2005/cyberslm-base
tags:
- cybersecurity
- security
- small-language-model
- instruction-tuned
- sft
- causal-lm
---

# CyberSLM-instruct — 33.5M-parameter cybersecurity assistant

Instruction-tuned from
[**sabari2005/cyberslm-base**](https://huggingface.co/sabari2005/cyberslm-base)
on 23,540 cybersecurity conversations.

**Code:** [github.com/Sabari2005/cyberslm](https://github.com/Sabari2005/cyberslm)

## Read this before using it

At 33.5M parameters this model learned the **shape** of a good answer — markdown
structure, numbered steps, worked examples, mitigation sections — and is
**frequently wrong about the content**.

Measured over 8 greedy prompts:

| category | mean 8-gram repetition | stopped on EOS |
|---|---:|---:|
| security | 18.7% | 1 / 4 |
| general | 18.9% | 1 / 2 |
| code | 19.9% | 0 / 2 |
| **overall** | **19.0%** | **2 / 8** |

**Do not use this as a factual security reference. Do not run code it
generates.** It is a demonstration of a correct end-to-end training pipeline and
a baseline for scaling.

### Where it works

Well-covered in-domain questions come out correct and terminate cleanly:

> **What is SQL injection and how do I prevent it?**
>
> SQL injection (SQLi) is a security vulnerability that allows attackers to
> manipulate database queries by injecting malicious SQL code through input
> fields. It occurs when user-supplied data is improperly sanitized, allowing
> attackers to manipulate the database. For example, if an attacker submits a
> username like `admin' --` as the username, the query becomes:
> `SELECT * FROM users WHERE username = '[input]' AND password = '[input]'`.

> **What is a firewall?**
>
> A firewall is a network security device or software that monitors and controls
> incoming and outgoing network traffic based on predefined security rules. It
> acts as a barrier between trusted internal networks and untrusted external
> networks, such as the internet.

### Where it fails

All observed, reproducible, and the majority case:

* **Wrong content, confident tone.** Asked to contrast symmetric and asymmetric
  encryption it answered about hashing and IKE, never addressing the question.
* **Circular definitions.** "A buffer overflow is a type of buffer overflow that
  could lead to arbitrary code execution."
* **Topically-adjacent but wrong vocabulary.** Asked how to investigate a
  phishing email it produced `SameSite` and `Strict` — real security terms,
  wrong topic (they are cookie attributes).
* **Degenerate loops in code.** `port: The port to use` repeated to the token
  limit.
* **Unreliable termination.** Only 2 of 8 prompts stopped on EOS; the rest ran
  to the token limit.

These are consequences of scale, not of the training run — the loss curve is
healthy and the pipeline is machine-verified (35 architecture checks, 173 tests).

## Model details

| | |
|---|---|
| parameters | 33,531,264 |
| architecture | 12 layers, d_model 384, 6 heads, SwiGLU 1024, RoPE, RMSNorm, tied head |
| context | 2048 |
| vocab | 32,000 (SentencePiece BPE) |
| base model | sabari2005/cyberslm-base |
| SFT data | 23,540 conversations, 15.1M supervised tokens |
| epochs | 3 (2,208 optimizer steps) |
| optimiser | AdamW, lr 2e-5, 3% warmup, cosine, bf16 |
| best val loss | 2.2627 (response tokens only) |

Loss is computed on assistant responses only; prompts are masked. 88% of tokens
in the SFT set are supervised.

## Usage

```bash
pip install torch sentencepiece
git clone https://huggingface.co/sabari2005/cyberslm-instruct
cd cyberslm-instruct
python infer_chat.py --prompt "What is SQL injection and how do I prevent it?"
```

Interactive:

```bash
python infer_chat.py --interactive
```

Options:

```bash
python infer_chat.py \
    --prompt "What is a buffer overflow?" \
    --max-new-tokens 200 \
    --temperature 0.0      # 0 = greedy, recommended for this model
```

### Prompt format

The model was trained on this exact layout, with a real BOS token id prepended
and EOS terminating each response:

```
### User:
{question}

 ### Assistant:
{response}<eos>
```

**Build prompts with the bundled formatter** (`infer_chat.py` does this).
Hand-assembling the string produces different token ids at every segment
boundary, because SentencePiece prepends a word-boundary marker per `encode()`
call — the model then sees something it was never trained on.

```python
import torch
from configs.sft_config import default_config
from data.prompt_formatter import PromptFormatter, Tokenizer
from model.cyberslm import CyberSLM

cfg = default_config()
cfg.tokenizer.model_path = "tokenizer/tokenizer.model"
cfg.model.max_seq_len = cfg.data.max_seq_len = 2048

tok = Tokenizer(cfg.tokenizer.model_path)
fmt = PromptFormatter(cfg=cfg, tokenizer=tok)

model = CyberSLM(cfg.model)
model.load_state_dict(torch.load("models/instruct.pt", map_location="cpu",
                                 weights_only=False))
model.eval()

ids = fmt.format_for_inference({"messages": [{"role": "user",
                                              "content": "What is XSS?"}]})
out = model.generate(torch.tensor([ids]), max_new_tokens=200,
                     temperature=0.0, eos_id=tok.eos_id)
print(tok.decode(out[0, len(ids):].tolist()))
```

Decoding uses a KV cache — roughly 50–70 tok/s on CPU.

## Intended use

Research into small language models; a scaling baseline; a demonstration of a
verified training pipeline. **Not** for security advice, incident response, code
generation, or anything where accuracy matters.

## Training data

Not published. Curated cybersecurity instruction data; not redistributed.

## License

Apache-2.0 for the code and weights. Verify licensing for downstream use against
the sources the data was curated from.