初始化项目,由ModelHub XC社区提供模型
Model: gabriellarson/Foundation-Sec-8B-Instruct-GGUF Source: Original Platform
This commit is contained in:
59
.gitattributes
vendored
Normal file
59
.gitattributes
vendored
Normal file
@@ -0,0 +1,59 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-F16.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Foundation-Sec-8B-Instruct-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
3
Foundation-Sec-8B-Instruct-F16.gguf
Normal file
3
Foundation-Sec-8B-Instruct-F16.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:017e447456c88ead5e454fd170a28d76af87f2126336e8a494746dc904493fd8
|
||||
size 16070994944
|
||||
3
Foundation-Sec-8B-Instruct-IQ2_M.gguf
Normal file
3
Foundation-Sec-8B-Instruct-IQ2_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:bf10ec2f9dddb82e54b6fc2c5cb89c06a65bf9936e5d3b5803135718794a13ec
|
||||
size 2948873472
|
||||
3
Foundation-Sec-8B-Instruct-IQ2_S.gguf
Normal file
3
Foundation-Sec-8B-Instruct-IQ2_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8a7ae01ec36a0ae179a9cc47a61ed6e1d20d741e2af077d71bc009755320849a
|
||||
size 2759081216
|
||||
3
Foundation-Sec-8B-Instruct-IQ2_XS.gguf
Normal file
3
Foundation-Sec-8B-Instruct-IQ2_XS.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d4d1eafe2329a23da6666d92ca7848202f9e19a1bc5cfc4e01a6d7a6e61b5ffc
|
||||
size 2606320896
|
||||
3
Foundation-Sec-8B-Instruct-IQ2_XXS.gguf
Normal file
3
Foundation-Sec-8B-Instruct-IQ2_XXS.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:47a1f5d7103eca58128418e605739394df4ced7e27aa4800ebcbc5524775af83
|
||||
size 2399751424
|
||||
3
Foundation-Sec-8B-Instruct-IQ3_M.gguf
Normal file
3
Foundation-Sec-8B-Instruct-IQ3_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c80a8a76283a290ed6aca6c07f65086e38b721048e32c12896028dc2ec12d8cb
|
||||
size 3785485568
|
||||
3
Foundation-Sec-8B-Instruct-IQ3_S.gguf
Normal file
3
Foundation-Sec-8B-Instruct-IQ3_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3ee8e090863b9224b9e86ddbbd1c57f41c07f3d417ca082e68f5b199f7e18d32
|
||||
size 3682987264
|
||||
3
Foundation-Sec-8B-Instruct-IQ3_XS.gguf
Normal file
3
Foundation-Sec-8B-Instruct-IQ3_XS.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2f7d24984bdf63fbee192061f60ea1074fda965d910791c5041e161b3289fffe
|
||||
size 3519409408
|
||||
3
Foundation-Sec-8B-Instruct-IQ3_XXS.gguf
Normal file
3
Foundation-Sec-8B-Instruct-IQ3_XXS.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2872e7240a62bd697569167f57172cab8d98d3fc5cdb53de74ab649ec707a8ad
|
||||
size 3275504896
|
||||
3
Foundation-Sec-8B-Instruct-IQ4_NL.gguf
Normal file
3
Foundation-Sec-8B-Instruct-IQ4_NL.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:59e535e9096f187ab5c1fe52e3b66ca035c77dc77f8897ceaf2e60a11fab71af
|
||||
size 4678720768
|
||||
3
Foundation-Sec-8B-Instruct-IQ4_XS.gguf
Normal file
3
Foundation-Sec-8B-Instruct-IQ4_XS.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3ffda7f9c9952e41e6b70d211be09facea176d0e9cff88f8d042ac5e587a1962
|
||||
size 4448378112
|
||||
3
Foundation-Sec-8B-Instruct-Q2_K.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q2_K.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7c69cd7f8555742e0e27158ccfcb763eb21dbac8cb9037af53ae52db6a8d27c5
|
||||
size 3179740416
|
||||
3
Foundation-Sec-8B-Instruct-Q2_K_S.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q2_K_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2e41b484d2cf6c6acccbe27422a03b7391dc1423e79a23b60f3c166ef1eda34b
|
||||
size 2989423872
|
||||
3
Foundation-Sec-8B-Instruct-Q3_K_L.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q3_K_L.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:02aee7e492596c3b0b76095e8a92fd730458748c829adb2d1a78ef2c839198dc
|
||||
size 4322618624
|
||||
3
Foundation-Sec-8B-Instruct-Q3_K_M.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q3_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8f483a31ef71a14ef0c8059a7aff8c4cab4f59c59fdd4d901d3bedd612ce034c
|
||||
size 4019580160
|
||||
3
Foundation-Sec-8B-Instruct-Q3_K_S.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q3_K_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:59e8e0fba90ed49c8d2789c6210d621a75e444d4017a009ede3e280ce2fd8e39
|
||||
size 3665161472
|
||||
3
Foundation-Sec-8B-Instruct-Q4_0.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q4_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:583e9bef8a3645fc695961196643357bab83294c6d740385f54a19a227053b40
|
||||
size 4676623616
|
||||
3
Foundation-Sec-8B-Instruct-Q4_K_M.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q4_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3175e83aa66aefeeddbc17608e3e8a2f52827941c73ccbe14fc9d138c8370206
|
||||
size 4921466112
|
||||
3
Foundation-Sec-8B-Instruct-Q4_K_S.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q4_K_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ff5d4ad1174fb14b21e052d75df4f78e30e5b7299272377eec0a08a2b4e04033
|
||||
size 4693400832
|
||||
3
Foundation-Sec-8B-Instruct-Q5_0.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q5_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:9066575cb3ad94b2afb0eecd2d10e33fea9131145b8cd6bf031c9e1b9b5c0f1e
|
||||
size 5614771456
|
||||
3
Foundation-Sec-8B-Instruct-Q5_K_M.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q5_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b7107f65790f087415f8ffc984fbe899ea2c1be93785d604da1332acf22cc008
|
||||
size 5733784832
|
||||
3
Foundation-Sec-8B-Instruct-Q5_K_S.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q5_K_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:db01543df4e6a037a49d5f452808f544337e7964883caf54438285af26302140
|
||||
size 5600091392
|
||||
3
Foundation-Sec-8B-Instruct-Q6_K.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q6_K.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8b4a372dcce4a4365515a2dcf5c4e9a65768259bcc7a6736ab8df2ae85bfc095
|
||||
size 6596873472
|
||||
3
Foundation-Sec-8B-Instruct-Q8_0.gguf
Normal file
3
Foundation-Sec-8B-Instruct-Q8_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2137ffe55c1654069c808f9da461ef11f64fbda8e37eeef1c5f5b258f07bd89a
|
||||
size 8541891840
|
||||
227
README.md
Normal file
227
README.md
Normal file
@@ -0,0 +1,227 @@
|
||||
---
|
||||
base_model:
|
||||
- fdtn-ai/Foundation-Sec-8B-Instruct
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
license: other
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- security
|
||||
- llama
|
||||
---
|
||||
# Foundation-Sec-8B-Instruct - Model Card
|
||||
|
||||
## Model Information
|
||||
|
||||
Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct (Foundation-Sec-8B-Instruct) is an open-weight, 8-billion parameter instruction-tuned language model specialized for cybersecurity applications.
|
||||
It extends the Foundation-Sec-8B base model with instruction-following capabilities.
|
||||
It leverages prior training to understand security concepts, terminology, and practices across multiple security domains.
|
||||
Further instruction-tuning allows the model to interact with human users in a chat-like interface.
|
||||
Foundation-Sec-8B-Instruct enables organizations to build AI-driven security tools that can be deployed locally, reducing dependency on cloud-based AI services while maintaining high performance on security-related tasks.
|
||||
|
||||
- **Model Name:** Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct (Foundation-Sec-8B-Instruct)
|
||||
- **Model Developer:** Amin Karbasi and Research team at Foundation AI — Cisco
|
||||
- **Model Card Contact:** For questions about the team, model usage, and future directions, contact [`karbasi@cisco.com`](mailto:karbasi@cisco.com). For technical questions about the model, please contact [`sajanaw@cisco.com`](mailto:sajanaw@cisco.com) and [`paulkass@cisco.com`](mailto:paulkass@cisco.com).
|
||||
- **Model Release Date:** August 1st, 2025
|
||||
- **Supported Language(s):** English
|
||||
- **Model Architecture:** Auto-regressive language model that uses an optimized transformer architecture (Meta Llama-3.1-8B backbone)
|
||||
- **Training Objective:** Instruction following and alignment with human preferences
|
||||
- **Training Data Status:** This is a static model trained on an offline dataset. Future versions of the tuned models will be released on updated data.
|
||||
- **License:** See NOTICE.md
|
||||
|
||||
|
||||
## Intended Use
|
||||
|
||||
### Intended Use Cases
|
||||
|
||||
Foundation-Sec-8B-Instruct is designed for security practitioners, researchers, and developers building AI-powered security workflows and applications.
|
||||
Foundation-Sec-8B-Instruct is optimized for three core use case categories:
|
||||
|
||||
- **SOC Acceleration**: Automating triage, summarization, case note generation, and evidence collection.
|
||||
- **Proactive Threat Defense**: Simulating attacks, prioritizing vulnerabilities, mapping TTPs, and modeling attacker behavior.
|
||||
- **Engineering Enablement**: Providing security assistance, validating configurations, assessing compliance evidence, and improving security posture.
|
||||
|
||||
The model is intended for local deployment in environments prioritizing data security, regulatory compliance, and operational control.
|
||||
|
||||
### Downstream Use
|
||||
|
||||
Foundation-Sec-8B-Instruct can be used directly for security-related chat use cases. Example downstream applications include:
|
||||
|
||||
- Summarization
|
||||
- Summarizing detection playbooks and incident reports
|
||||
- Consolidating fragmented analyst notes into structured case summaries
|
||||
- Classification
|
||||
- Mapping threats to MITRE ATT&CK techniques
|
||||
- Prioritizing vulnerabilities based on contextual risk
|
||||
- Classifying security-relevant emails and leaked file contents
|
||||
- Named Entity Recognition
|
||||
- Extracting compliance evidence from documents
|
||||
- Building network behavior profiles from technical manuals
|
||||
- Question & Answer
|
||||
- Assisting SOC analysts with alert triage and investigation
|
||||
- Responding to cloud security and software compliance queries
|
||||
- Reasoning and Text Generation
|
||||
- Generating red-team attack plans and threat models
|
||||
- Predicting attacker next steps in active investigations
|
||||
- Enriching vulnerability scan results with contextual insights
|
||||
|
||||
For questions or assistance with fine-tuning Foundation-Sec-8B-Instruct, please reach out to the team.
|
||||
|
||||
### Out-of-Scope Use
|
||||
|
||||
The following uses are out-of-scope and are neither recommended nor intended use cases:
|
||||
|
||||
1. **Generating harmful content** - The model should not be used to:
|
||||
- Generate malware or other malicious code
|
||||
- Create phishing content or social engineering scripts
|
||||
- Develop attack plans targeting specific organizations
|
||||
- Design exploitation techniques for vulnerabilities without legitimate security research purposes
|
||||
2. **Critical security decisions without human oversight** - The model should not be used for:
|
||||
- Autonomous security decision-making without human review
|
||||
- Critical infrastructure protection without expert supervision
|
||||
- Final determination of security compliance without human verification
|
||||
- Autonomous vulnerability remediation without testing
|
||||
3. **Legal or medical advice** - The model is not qualified to provide:
|
||||
- Legal advice regarding security regulations, compliance requirements, or intellectual property disputes
|
||||
- Legal advice regarding security issues that would reference legal statutes, precedents, or case law necessary to provide legal advice
|
||||
- Medical advice regarding health impacts of security incidents
|
||||
4. **Non-security use cases** - The model is specifically optimized for cybersecurity and may not perform as well on general tasks as models trained for broader applications.
|
||||
5. **Violation of Laws or Regulations** - Any use that violates applicable laws or regulations.
|
||||
|
||||
## How to Get Started with the Model
|
||||
|
||||
Use the code below to get started with the model.
|
||||
|
||||
```python
|
||||
# Import the required libraries
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
# Load the model and tokenizer
|
||||
tokenizer = AutoTokenizer.from_pretrained("fdtn-ai/Foundation-Sec-8B-Instruct")
|
||||
model = AutoModelForCausalLM.from_pretrained("fdtn-ai/Foundation-Sec-8B-Instruct")
|
||||
prompt = "CVE-2015-10011 is a vulnerability about OpenDNS OpenResolve improper log output neutralization. What is the corresponding CWE?"
|
||||
messages = [
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
model_inputs = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
inputs = tokenizer(model_inputs, return_tensors="pt", add_special_tokens=False)
|
||||
output = model.generate(**inputs, temperature=0.1, max_new_tokens=250)
|
||||
resp = tokenizer.batch_decode(output)[0]
|
||||
print(resp.replace(model_inputs, ""))
|
||||
```
|
||||
|
||||
## Training and Evaluation
|
||||
|
||||
### Training Data
|
||||
|
||||
Foundation-Sec-8B-Instruct was trained on a wide variety of public and proprietary question answer/pairs for general and security-specific instruction-following.
|
||||
|
||||
**Data cutoff:** April 10th, 2025.
|
||||
|
||||
A more detailed description of the methodology is available in the technical report.
|
||||
|
||||
### Training Setup
|
||||
|
||||
Foundation-Sec-8B-Instruct is based on the **Llama 3.1 8B** architecture. Training was performed on Cisco Foundation AI’s internal compute cluster.
|
||||
|
||||
Key training details:
|
||||
|
||||
- **Instruction fine-tuning** to follow human instructions
|
||||
- **RLHF** to align model answers to human preferences
|
||||
- **4096-token** sequence length
|
||||
- **Optimizer:** AdamW
|
||||
|
||||
A more detailed description of the methodology is available in the technical report.
|
||||
|
||||
### Evaluation
|
||||
|
||||
Foundation-Sec-8B-Instruct was benchmarked on cybersecurity and general reasoning tasks, using a standardized 0-shot instruction prompting setup (temperature = 0.3).
|
||||
|
||||
| **Benchmark** | **Foundation-sec-8B** | **Llama 3.1 8B** | **GPT-4o-mini** |
|
||||
| --- | --- | --- | --- |
|
||||
| CTI-MCQA | 0.644 | 0.617 | 0.672 |
|
||||
| CTI-RCM | 0.692 | 0.558 | 0.655 |
|
||||
| CTI-VSP | 0.802 | 0.815 | 0.792 |
|
||||
| IF-Eval | 0.811 | 0.791 | 0.834 |
|
||||
| Alpaca Eval 2 | 35.453 | 24.477 | 52.720 |
|
||||
|
||||
**Benchmark Overview:**
|
||||
|
||||
- **CTI-MCQA:** 2,500 multiple-choice questions testing cybersecurity knowledge across frameworks like MITRE ATT&CK, NIST, GDPR, and threat intelligence best practices.
|
||||
- **CTI-RCM:** 1,000 vulnerability root cause mapping examples linking CVEs to CWE categories, assessing deep understanding of security weaknesses.
|
||||
- **CTI-VSP:** A set of 1,000 CVE descriptions where models predict the CVSS v3 Base metrics and compute the overall score, with performance measured by the average absolute difference from the true scores.
|
||||
- **IF-Eval:** 541 instruction-following prompts designed for automated, reproducible assessment of LLM instruction-following capabilities.
|
||||
- **Alpaca Eval 2:** 805 single-turn prompts auto-scored by GPT-4 Turbo against a GPT-4 Turbo reference, validated with 20,000 human preference votes, and closely matching ChatBot Arena results.
|
||||
|
||||
**Key highlights:**
|
||||
|
||||
- **+3 to +11 point gains** over Llama-3.1-8B-Instruct across security-specific benchmarks.
|
||||
- **Exceptional Instruction-Following capabilities** exceeding that of Llama-3.1-8B-Instruct.
|
||||
- **Competitive against small Frontier Models** such as GPT-4o-mini on instruction-following capabilities and cybersecurity tasks.
|
||||
|
||||
For full benchmark details and evaluation methodology, please refer to the technical report.
|
||||
|
||||
## Safety Alignment
|
||||
|
||||
Standard best practices were followed to align the model with general safety values.
|
||||
Despite the alignment, however, safe out-of-the-box performance cannot be guaranteed.
|
||||
Our evaluations show that while the model can achieve reasonable safety performance out-of-the-box, LlamaGuard provides much better protection against malicious requests.
|
||||
It is recommended to deploy this model with additional safeguards (such as LlamaGuard) and human oversight.
|
||||
|
||||
| Model | HarmBench Performance |
|
||||
|---|---|
|
||||
| Llama-3.1-8b-Instruct | 72.43% |
|
||||
| Foundation-Sec-8B-Instruct | 91.98% |
|
||||
| **LlamaGuard** + Foundation-Sec-8B-Instruct | 99.25% |
|
||||
|
||||
|
||||
## Limitations
|
||||
|
||||
Foundation-Sec-8B-Instruct has several limitations that users should be aware of:
|
||||
|
||||
1. **Domain-specific knowledge limitations**:
|
||||
- Foundation-Sec-8B-Instruct may not be familiar with recent vulnerabilities, exploits, or novel attack vectors or security technologies released after its training cutoff date
|
||||
- Knowledge of specialized or proprietary security systems or tools may be limited
|
||||
2. **Potential biases**:
|
||||
- The model may reflect biases present in security literature and documentation
|
||||
- The model may be trained on known attack patterns and have difficulty recognizing novel attack vectors
|
||||
- Security practices and recommendations may be biased toward certain technological ecosystems
|
||||
- Geographic and cultural biases in security approaches may be present
|
||||
3. **Security risks**:
|
||||
- The model cannot verify the identity or intentions of users
|
||||
- Adversarial prompting techniques might potentially bypass safety mechanisms
|
||||
- The model may unintentionally provide information that could be misused if proper prompting guardrails are not implemented
|
||||
4. **Contextual blindness:**
|
||||
- The model may struggle to understand the complex interrelationships between systems, users, and data in order to provide accurate context.
|
||||
5. **Technical limitations**:
|
||||
- Performance varies based on how security concepts are described in prompts
|
||||
- May not fully understand complex, multi-step security scenarios without clear explanation
|
||||
- Cannot access external systems or actively scan environments
|
||||
- Cannot independently verify factual accuracy of its outputs
|
||||
6. **Ethical considerations**:
|
||||
- Dual-use nature of security knowledge requires careful consideration of appropriate use cases
|
||||
|
||||
### Recommendations
|
||||
|
||||
To address the limitations of Foundation-Sec-8B-Instruct, we recommend:
|
||||
|
||||
1. **Human oversight**:
|
||||
- Always have qualified security professionals review model outputs before implementation
|
||||
- Use the model as an assistive tool rather than a replacement for expert human judgment
|
||||
- Implement a human-in-the-loop approach for security-critical applications
|
||||
2. **System design safeguards**:
|
||||
- Implement additional validation layers for applications built with this model
|
||||
- Consider architectural constraints that limit the model's ability to perform potentially harmful actions (excessive agency)
|
||||
- Deploy the model in environments with appropriate access controls
|
||||
3. **Prompt engineering**:
|
||||
- Use carefully designed prompts that encourage ethical security practices
|
||||
- Include explicit instructions regarding responsible disclosure and ethical hacking principles
|
||||
- Structure interactions to minimize the risk of inadvertently harmful outputs
|
||||
4. **Knowledge supplementation**:
|
||||
- Supplement the model with up-to-date security feeds and databases
|
||||
- Implement retrieval-augmented generation for current threat intelligence sources
|
||||
5. **Usage policies**:
|
||||
- Develop and enforce clear acceptable use policies for applications using this model
|
||||
- Implement monitoring and auditing for high-risk applications
|
||||
- Create documentation for end users about the model's limitations
|
||||
Reference in New Issue
Block a user