147 lines
4.6 KiB
Markdown
147 lines
4.6 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
library_name: pytorch
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- cybersecurity
|
|||
|
|
- security
|
|||
|
|
- small-language-model
|
|||
|
|
- from-scratch
|
|||
|
|
- causal-lm
|
|||
|
|
- pretrained
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# CyberSLM-base — 33.5M-parameter cybersecurity language model
|
|||
|
|
|
|||
|
|
A decoder-only transformer pretrained from scratch on a cybersecurity corpus.
|
|||
|
|
This is the **base** model: it continues text. It has not been instruction-tuned
|
|||
|
|
and will not answer questions.
|
|||
|
|
|
|||
|
|
For question answering use
|
|||
|
|
[**sabari2005/cyberslm-instruct**](https://huggingface.co/sabari2005/cyberslm-instruct).
|
|||
|
|
|
|||
|
|
**Code:** [github.com/Sabari2005/cyberslm](https://github.com/Sabari2005/cyberslm)
|
|||
|
|
|
|||
|
|
## What this model is
|
|||
|
|
|
|||
|
|
Give it the start of a sentence and it continues it:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
Prompt: "SQL injection is"
|
|||
|
|
Output: "SQL injection is a common issue in the web interface of Cisco IOS and
|
|||
|
|
IOS XE Software. It has been declared as critical for its security,
|
|||
|
|
integrity, and availability. The vulnerability exists because the
|
|||
|
|
affected software does not properly validate user-supplied input..."
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Ask it a question and it will continue the *question*, not answer it.
|
|||
|
|
|
|||
|
|
## Model details
|
|||
|
|
|
|||
|
|
| | |
|
|||
|
|
|---|---|
|
|||
|
|
| parameters | 33,531,264 |
|
|||
|
|
| layers | 12 |
|
|||
|
|
| d_model | 384 |
|
|||
|
|
| heads / head_dim | 6 / 64 |
|
|||
|
|
| FFN (SwiGLU) | 1024 |
|
|||
|
|
| context | 2048 |
|
|||
|
|
| vocab | 32,000 (SentencePiece BPE, byte-fallback) |
|
|||
|
|
| positional encoding | RoPE, base 10000 |
|
|||
|
|
| normalisation | RMSNorm, pre-norm |
|
|||
|
|
| LM head | tied to embedding |
|
|||
|
|
| precision | trained in bf16 |
|
|||
|
|
|
|||
|
|
**Training.** 786,432,000 tokens = 4.04 epochs over a 194.8M-token corpus
|
|||
|
|
(~60% cybersecurity across 16 subdomains, ~20% general English and reasoning,
|
|||
|
|
~15% programming, ~5% CS fundamentals). 6,000 steps at 131,072 tokens/step,
|
|||
|
|
AdamW, lr 3e-4 → 3e-5, 600 warmup, cosine decay, grad clip 1.0.
|
|||
|
|
|
|||
|
|
Single A100-40GB, 71 minutes, 184,084 tokens/sec.
|
|||
|
|
|
|||
|
|
## Evaluation
|
|||
|
|
|
|||
|
|
409,600 held-out tokens. Compared against an earlier checkpoint of the same
|
|||
|
|
architecture, both scored by one process on **identical windows at identical
|
|||
|
|
context** (a longer conditioning window lowers loss on its own, so scoring each
|
|||
|
|
at its own maximum would not be a fair comparison):
|
|||
|
|
|
|||
|
|
| metric | this model | earlier checkpoint |
|
|||
|
|
|---|---|---|
|
|||
|
|
| validation loss | **2.3627** | 2.6255 |
|
|||
|
|
| perplexity | **10.62** | 13.81 |
|
|||
|
|
| bits / token | **3.4086** | 3.7878 |
|
|||
|
|
| top-1 accuracy | **57.21%** | 54.38% |
|
|||
|
|
| top-5 accuracy | **72.64%** | 69.55% |
|
|||
|
|
| 8-gram repetition | **23.7%** | 34.0% |
|
|||
|
|
|
|||
|
|
Training-time validation loss at step 6,000 was 2.0247, measured on a different
|
|||
|
|
subset; only the columns above are like-for-like.
|
|||
|
|
|
|||
|
|
No benchmark accuracy is claimed — there is no contamination-checked security
|
|||
|
|
question bank, so nothing beyond next-token metrics is asserted. Single seed.
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
pip install torch sentencepiece
|
|||
|
|
git clone https://huggingface.co/sabari2005/cyberslm-base
|
|||
|
|
cd cyberslm-base
|
|||
|
|
python infer_base.py --prompt "SQL injection is"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Options:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
python infer_base.py \
|
|||
|
|
--prompt "A buffer overflow occurs when" \
|
|||
|
|
--max-new-tokens 120 \
|
|||
|
|
--temperature 0.8 \ # 0 = greedy/deterministic
|
|||
|
|
--top-k 50 --top-p 0.95 \
|
|||
|
|
--repetition-penalty 1.15
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Loading directly
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import torch, sentencepiece as spm
|
|||
|
|
from cyberslm.model.config import CyberSLMConfig
|
|||
|
|
from cyberslm.model.model import build_model
|
|||
|
|
|
|||
|
|
payload = torch.load("models/base.pt", map_location="cpu", weights_only=False)
|
|||
|
|
model = build_model(CyberSLMConfig(**payload["config"]), device=torch.device("cpu"))
|
|||
|
|
model.load_state_dict(payload["model_state"])
|
|||
|
|
model.eval()
|
|||
|
|
|
|||
|
|
sp = spm.SentencePieceProcessor(); sp.load("tokenizer/tokenizer.model")
|
|||
|
|
ids = [sp.bos_id()] + sp.encode("SQL injection is", out_type=int)
|
|||
|
|
out = model.generate(torch.tensor([ids]), max_new_tokens=60,
|
|||
|
|
temperature=0.0, eos_id=sp.eos_id())
|
|||
|
|
print(sp.decode(out[0].tolist()))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`generate()` uses a KV cache, so decoding is O(n) — roughly 50–70 tok/s on CPU.
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
A 33.5M-parameter model trained on 786M tokens. It produces fluent,
|
|||
|
|
domain-flavoured security prose and is **not factually reliable**. Output drifts
|
|||
|
|
into CVE-advisory boilerplate because that pattern is common in the corpus, and
|
|||
|
|
longer generations repeat (23.7% 8-gram repetition measured).
|
|||
|
|
|
|||
|
|
Intended for research into small language models and as a base for further
|
|||
|
|
scaling or fine-tuning. Not intended for security advice or any use where being
|
|||
|
|
wrong matters.
|
|||
|
|
|
|||
|
|
## Training data
|
|||
|
|
|
|||
|
|
Not published. Curated from public cybersecurity, programming and
|
|||
|
|
general-English sources; not redistributed with the model.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Apache-2.0 for the code and weights. Verify licensing for downstream use
|
|||
|
|
against the sources the corpus was curated from.
|