Llama-Primus-Reasoning/README.md

license, datasets, language, base_model, pipeline_tag, library_name, tags, extra_gated_fields

license	datasets	language	base_model	pipeline_tag	library_name	tags	extra_gated_fields
mit	trendmicro-ailab/Primus-Reasoning trendmicro-ailab/Primus-Seed trendmicro-ailab/Primus-FineWeb trendmicro-ailab/Primus-Instruct	en	trendmicro-ailab/Llama-Primus-Merged	text-generation	transformers	cybersecurity pretraining	Affiliation Country I want to use this model for Job title geo text country type options select Research Commercial label value Other other type options select Student Research graduate AI researcher AI developer/engineer Cybersecurity researcher Reporter Other ip_location

Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training

First cybersecurity reasoning model!

TL;DR: Llama-Primus-Reasoning is a reasoning model distilled from the reasoning steps with reflection data generated by o1-preview & DeepSeek-R1 on cybersecurity tasks (Primus-Reasoning), based on Llama-Primus-Merged. It demonstrates a 🚀15.8% improvement in security certification (CISSP).

🔥 For more details, please refer to the paper: [📄Paper].

📢 News (2025/06/02): We have expanded the Primus-Reasoning dataset with additional samples from DeepSeek-R1. Accordingly, we have replaced Llama-Primus-Reasoning with a new version distilled jointly from DeepSeek-R1 and o1-preview. This version achieves the best CISSP performance, with a 15.8% improvement.

Introduction

Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, with promising applications in specialized domains such as finance, law, and biomedicine. However, in the domain of cybersecurity, we noticed a lack of open-source datasets specifically designed for LLM pre-training—even though much research has shown that LLMs acquire their knowledge during pre-training. To fill this gap, we present a collection of datasets covering multiple stages of cybersecurity LLM training, including pre-training (Primus-Seed and Primus-FineWeb), instruction fine-tuning (Primus-Instruct), and reasoning data for distillation (Primus-Reasoning). Based on these datasets and Llama-3.1-8B-Instruct, we developed Llama-Primus-Base, Llama-Primus-Merged, and Llama-Primus-Reasoning. This model card is Llama-Primus-Reasoning.

Note: No TrendMicro customer information is included.

Cybersecurity Benchmark Results

Model	CISSP	Avg. Tokens
w/o CoT, 5-shot
Llama-3.1-8B-Instruct	0.7073	1
Llama-Primus-Merged	0.7191 ↑1.67%	1
w/ CoT, 0-shot
Llama-3.1-8B-Instruct	0.7288 ↑3.03%	279.69
└─ + Distilled from o1-preview	0.7583 ↑7.21%	646.94
└─ + Distilled from DeepSeek-R1	0.7859 ↑11.1%	1667.56
└─ + Distilled from (o1 + R1)	0.7780 ↑10.0%	1615.54
Llama-Primus-Merged	0.7603 ↑7.49%	241.92
└─ + Distilled from o1-preview	0.7780 ↑10.0%	726.96
└─ + Distilled from DeepSeek-R1	0.8075 ↑14.2%	1483.94
└─ + Distilled from (o1 + R1)	0.8193 ↑15.8%	1467.40
Raw Models for Comparison
o1-preview	0.8035	1054.91
DeepSeek-R1	0.8212	1229.32
DeepSeek-R1-Distill-Llama-8B	0.7399 ↑4.61%	1542.10

Effect of Primus-Reasoning fine-tuning, evaluated on CISSP. ↑ indicates the percentage improvement over Llama without CoT and in the 5-shot setting. The best improvement is highlighted in bold.

About Primus

Primus is Trend Micro's pioneering family of lightweight, state-of-the-art open cybersecurity language models and datasets. Developed through our cutting-edge research initiatives and advanced technology, these resources share the innovative foundation that powers our enterprise-class Trend Cybertron solution. As an industry leader in cybersecurity, Trend Micro is proud to contribute these powerful, efficiency-optimized models and datasets to the community, while maintaining the excellence and reliability that define our global security standards.

License

This model is based on the MIT license, but you must also comply with the Llama 3.1 Community License Agreement.