147 lines
4.6 KiB
Markdown
147 lines
4.6 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
library_name: pytorch
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- cybersecurity
|
||
- security
|
||
- small-language-model
|
||
- from-scratch
|
||
- causal-lm
|
||
- pretrained
|
||
---
|
||
|
||
# CyberSLM-base — 33.5M-parameter cybersecurity language model
|
||
|
||
A decoder-only transformer pretrained from scratch on a cybersecurity corpus.
|
||
This is the **base** model: it continues text. It has not been instruction-tuned
|
||
and will not answer questions.
|
||
|
||
For question answering use
|
||
[**sabari2005/cyberslm-instruct**](https://huggingface.co/sabari2005/cyberslm-instruct).
|
||
|
||
**Code:** [github.com/Sabari2005/cyberslm](https://github.com/Sabari2005/cyberslm)
|
||
|
||
## What this model is
|
||
|
||
Give it the start of a sentence and it continues it:
|
||
|
||
```
|
||
Prompt: "SQL injection is"
|
||
Output: "SQL injection is a common issue in the web interface of Cisco IOS and
|
||
IOS XE Software. It has been declared as critical for its security,
|
||
integrity, and availability. The vulnerability exists because the
|
||
affected software does not properly validate user-supplied input..."
|
||
```
|
||
|
||
Ask it a question and it will continue the *question*, not answer it.
|
||
|
||
## Model details
|
||
|
||
| | |
|
||
|---|---|
|
||
| parameters | 33,531,264 |
|
||
| layers | 12 |
|
||
| d_model | 384 |
|
||
| heads / head_dim | 6 / 64 |
|
||
| FFN (SwiGLU) | 1024 |
|
||
| context | 2048 |
|
||
| vocab | 32,000 (SentencePiece BPE, byte-fallback) |
|
||
| positional encoding | RoPE, base 10000 |
|
||
| normalisation | RMSNorm, pre-norm |
|
||
| LM head | tied to embedding |
|
||
| precision | trained in bf16 |
|
||
|
||
**Training.** 786,432,000 tokens = 4.04 epochs over a 194.8M-token corpus
|
||
(~60% cybersecurity across 16 subdomains, ~20% general English and reasoning,
|
||
~15% programming, ~5% CS fundamentals). 6,000 steps at 131,072 tokens/step,
|
||
AdamW, lr 3e-4 → 3e-5, 600 warmup, cosine decay, grad clip 1.0.
|
||
|
||
Single A100-40GB, 71 minutes, 184,084 tokens/sec.
|
||
|
||
## Evaluation
|
||
|
||
409,600 held-out tokens. Compared against an earlier checkpoint of the same
|
||
architecture, both scored by one process on **identical windows at identical
|
||
context** (a longer conditioning window lowers loss on its own, so scoring each
|
||
at its own maximum would not be a fair comparison):
|
||
|
||
| metric | this model | earlier checkpoint |
|
||
|---|---|---|
|
||
| validation loss | **2.3627** | 2.6255 |
|
||
| perplexity | **10.62** | 13.81 |
|
||
| bits / token | **3.4086** | 3.7878 |
|
||
| top-1 accuracy | **57.21%** | 54.38% |
|
||
| top-5 accuracy | **72.64%** | 69.55% |
|
||
| 8-gram repetition | **23.7%** | 34.0% |
|
||
|
||
Training-time validation loss at step 6,000 was 2.0247, measured on a different
|
||
subset; only the columns above are like-for-like.
|
||
|
||
No benchmark accuracy is claimed — there is no contamination-checked security
|
||
question bank, so nothing beyond next-token metrics is asserted. Single seed.
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
pip install torch sentencepiece
|
||
git clone https://huggingface.co/sabari2005/cyberslm-base
|
||
cd cyberslm-base
|
||
python infer_base.py --prompt "SQL injection is"
|
||
```
|
||
|
||
Options:
|
||
|
||
```bash
|
||
python infer_base.py \
|
||
--prompt "A buffer overflow occurs when" \
|
||
--max-new-tokens 120 \
|
||
--temperature 0.8 \ # 0 = greedy/deterministic
|
||
--top-k 50 --top-p 0.95 \
|
||
--repetition-penalty 1.15
|
||
```
|
||
|
||
### Loading directly
|
||
|
||
```python
|
||
import torch, sentencepiece as spm
|
||
from cyberslm.model.config import CyberSLMConfig
|
||
from cyberslm.model.model import build_model
|
||
|
||
payload = torch.load("models/base.pt", map_location="cpu", weights_only=False)
|
||
model = build_model(CyberSLMConfig(**payload["config"]), device=torch.device("cpu"))
|
||
model.load_state_dict(payload["model_state"])
|
||
model.eval()
|
||
|
||
sp = spm.SentencePieceProcessor(); sp.load("tokenizer/tokenizer.model")
|
||
ids = [sp.bos_id()] + sp.encode("SQL injection is", out_type=int)
|
||
out = model.generate(torch.tensor([ids]), max_new_tokens=60,
|
||
temperature=0.0, eos_id=sp.eos_id())
|
||
print(sp.decode(out[0].tolist()))
|
||
```
|
||
|
||
`generate()` uses a KV cache, so decoding is O(n) — roughly 50–70 tok/s on CPU.
|
||
|
||
## Limitations
|
||
|
||
A 33.5M-parameter model trained on 786M tokens. It produces fluent,
|
||
domain-flavoured security prose and is **not factually reliable**. Output drifts
|
||
into CVE-advisory boilerplate because that pattern is common in the corpus, and
|
||
longer generations repeat (23.7% 8-gram repetition measured).
|
||
|
||
Intended for research into small language models and as a base for further
|
||
scaling or fine-tuning. Not intended for security advice or any use where being
|
||
wrong matters.
|
||
|
||
## Training data
|
||
|
||
Not published. Curated from public cybersecurity, programming and
|
||
general-English sources; not redistributed with the model.
|
||
|
||
## License
|
||
|
||
Apache-2.0 for the code and weights. Verify licensing for downstream use
|
||
against the sources the corpus was curated from.
|