初始化项目，由ModelHub XC社区提供模型
Model: Aravindan/smol-lm2-360-instruct Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
204
README.md
Normal file
204
README.md
Normal file
@@ -0,0 +1,204 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
|
||||
license: apache-2.0
|
||||
|
||||
base_model: HuggingFaceTB/SmolLM2-360M
|
||||
|
||||
pipeline_tag: text-generation
|
||||
|
||||
tags:
|
||||
- cybersecurity
|
||||
- instruction-tuning
|
||||
- smollm2
|
||||
- causal-lm
|
||||
- transformers
|
||||
- pytorch
|
||||
- llm
|
||||
- ai-security
|
||||
|
||||
datasets:
|
||||
- Trendyol/Trendyol-Cybersecurity-Instruction-Tuning-Dataset
|
||||
|
||||
library_name: transformers
|
||||
---
|
||||
|
||||
# smol-lm2-360-instruct
|
||||
|
||||
smol-lm2-360-instruct is an instruction-tuned language model built by fine-tuning **SmolLM2-360M Base** on a cybersecurity instruction dataset. The objective is to improve the model's ability to answer cybersecurity-related questions while following instruction-style prompts.
|
||||
|
||||
> **Note:** This is an educational project for learning Instruction Fine-Tuning (IFT) and should not be considered a production-grade security assistant.
|
||||
|
||||
---
|
||||
|
||||
# Model Details
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| Base Model | HuggingFaceTB/SmolLM2-360M |
|
||||
| Model Type | Causal Language Model |
|
||||
| Fine-tuning | Full Parameter Instruction Fine-Tuning |
|
||||
| Domain | Cybersecurity |
|
||||
| Framework | Hugging Face Transformers |
|
||||
| Training Library | Transformers Trainer |
|
||||
| Language | English |
|
||||
|
||||
---
|
||||
|
||||
# Training Objective
|
||||
|
||||
The model was trained using a supervised instruction-tuning objective.
|
||||
|
||||
Each training example follows the structure:
|
||||
|
||||
```text
|
||||
<|system|>
|
||||
You are a cybersecurity assistant.
|
||||
|
||||
<|user|>
|
||||
Explain SQL Injection.
|
||||
|
||||
<|assistant|>
|
||||
SQL Injection is...
|
||||
```
|
||||
|
||||
During training:
|
||||
|
||||
- The **System** prompt provides behavioral context.
|
||||
- The **User** prompt provides the instruction.
|
||||
- Loss is computed **only on the Assistant response**, while the System and User tokens are masked from the loss.
|
||||
|
||||
This enables the model to learn instruction following while conditioning on the provided context.
|
||||
|
||||
---
|
||||
|
||||
# Training Configuration
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Epochs | 4 |
|
||||
| Learning Rate | 2e-4 |
|
||||
| Scheduler | Cosine |
|
||||
| Warmup | 10% |
|
||||
| Batch Size | 2 |
|
||||
| Gradient Accumulation | 8 |
|
||||
| Precision | FP16 |
|
||||
| Optimizer | AdamW |
|
||||
|
||||
---
|
||||
|
||||
# Dataset
|
||||
|
||||
The model was trained on a cybersecurity instruction dataset consisting of:
|
||||
|
||||
- System prompts
|
||||
- User instructions
|
||||
- Assistant responses
|
||||
|
||||
The dataset covers topics such as:
|
||||
|
||||
- Network Security
|
||||
- Malware Analysis
|
||||
- Threat Intelligence
|
||||
- Secure Coding
|
||||
- Incident Response
|
||||
- Vulnerability Management
|
||||
- Security Best Practices
|
||||
- MITRE ATT&CK
|
||||
- NIST Cybersecurity Framework
|
||||
|
||||
---
|
||||
|
||||
# Example
|
||||
|
||||
## Prompt
|
||||
|
||||
```text
|
||||
<|system|>
|
||||
You are a cybersecurity expert.
|
||||
|
||||
<|user|>
|
||||
Explain the difference between symmetric and asymmetric encryption.
|
||||
|
||||
<|assistant|>
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
```text
|
||||
Symmetric encryption uses a single shared key for both encryption and decryption, making it computationally efficient and suitable for encrypting large amounts of data. Asymmetric encryption uses a public/private key pair, enabling secure key exchange and digital signatures but at a higher computational cost.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
model_name = "Aravindan/smol-lm2-360-instruct"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_name)
|
||||
|
||||
prompt = """
|
||||
<|system|>
|
||||
You are a cybersecurity assistant.
|
||||
|
||||
<|user|>
|
||||
Explain SQL Injection.
|
||||
|
||||
<|assistant|>
|
||||
"""
|
||||
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=200,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Limitations
|
||||
|
||||
This model:
|
||||
|
||||
- is intended for educational and research purposes
|
||||
- may generate incorrect or outdated cybersecurity advice
|
||||
- has not undergone safety alignment comparable to production LLMs
|
||||
- should not be used as the sole source for security decisions
|
||||
|
||||
Always verify security recommendations with authoritative sources.
|
||||
|
||||
---
|
||||
|
||||
# Future Improvements
|
||||
|
||||
- Larger instruction datasets
|
||||
- Multi-turn conversation training
|
||||
- Preference Optimization (DPO/ORPO)
|
||||
- Retrieval-Augmented Generation (RAG)
|
||||
- LoRA and QLoRA fine-tuning
|
||||
- Evaluation on cybersecurity benchmarks
|
||||
|
||||
---
|
||||
|
||||
# Acknowledgements
|
||||
|
||||
- Hugging Face
|
||||
- SmolLM2 Team
|
||||
- Transformers
|
||||
- Datasets
|
||||
- PyTorch
|
||||
|
||||
---
|
||||
|
||||
# Citation
|
||||
|
||||
If you use this model in your work, please cite this repository.
|
||||
34
config.json
Normal file
34
config.json
Normal file
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 0,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 0,
|
||||
"head_dim": 64,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 960,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 2560,
|
||||
"is_llama_config": true,
|
||||
"max_position_embeddings": 8192,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 15,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 5,
|
||||
"pad_token_id": null,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_interleaved": false,
|
||||
"rope_parameters": {
|
||||
"rope_theta": 100000,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "5.12.1",
|
||||
"use_cache": false,
|
||||
"vocab_size": 49152
|
||||
}
|
||||
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 0,
|
||||
"eos_token_id": 0,
|
||||
"transformers_version": "5.12.1"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:bcc86b2aa72442cab3d6b6cf1c06bf1e9e946cf0282ff7129e615bb161f456e7
|
||||
size 723674912
|
||||
244965
tokenizer.json
Normal file
244965
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
34
tokenizer_config.json
Normal file
34
tokenizer_config.json
Normal file
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"add_prefix_space": false,
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<|endoftext|>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|endoftext|>",
|
||||
"errors": "replace",
|
||||
"extra_special_tokens": [
|
||||
"<|endoftext|>",
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<repo_name>",
|
||||
"<reponame>",
|
||||
"<file_sep>",
|
||||
"<filename>",
|
||||
"<gh_stars>",
|
||||
"<issue_start>",
|
||||
"<issue_comment>",
|
||||
"<issue_closed>",
|
||||
"<jupyter_start>",
|
||||
"<jupyter_text>",
|
||||
"<jupyter_code>",
|
||||
"<jupyter_output>",
|
||||
"<jupyter_script>",
|
||||
"<empty_output>"
|
||||
],
|
||||
"is_local": false,
|
||||
"local_files_only": false,
|
||||
"model_max_length": 8192,
|
||||
"pad_token": "<|endoftext|>",
|
||||
"tokenizer_class": "GPT2Tokenizer",
|
||||
"unk_token": "<|endoftext|>",
|
||||
"vocab_size": 49152
|
||||
}
|
||||
Reference in New Issue
Block a user