safety_model/README.md

license, base_model, language, pipeline_tag, library_name, tags, datasets

license	base_model	language	pipeline_tag	library_name	tags	datasets
apache-2.0	Qwen/Qwen3-1.7B	en	text-generation	transformers	safety qwen3 sft lora multiple-choice	cs-552-2026-Flash-McQueenS-and-TheKing/safety_sft_data

safety_model — Qwen3-1.7B fine-tuned for safety multiple-choice (non-thinking)

This is the safety individual model for the CS-552 (Modern NLP, EPFL, Spring 2026) course project. It is a supervised fine-tune of Qwen/Qwen3-1.7B specialised for the safety benchmark, which is multiple-choice, scored pass@1.

The model answers in non-thinking mode: it emits a one-sentence justification and then the answer letter inside \boxed{}, with no <think> reasoning block. The rationale is given in Why non-thinking, and no CoT distillation.

Output contract. Every answer ends with the option letter wrapped in \boxed{...}, e.g. \boxed{C}. The boxing instruction and the non-thinking switch are baked into the tokenizer's chat_template.jinja, so they apply even when the evaluator calls apply_chat_template(messages, add_generation_prompt=True) with no extra arguments.

Training data

The full training set is released as a companion dataset: cs-552-2026-Flash-McQueenS-and-TheKing/safety_sft_data.

It contains 3,250 English multiple-choice items across the seven safety categories of SafetyBench (Zhang et al., 2024). Each example pairs a user message (a question with labelled options) with an assistant message (a one-sentence justification followed by the answer letter in \boxed{}). Four categories are derived from established public datasets; three smaller categories are LLM-generated to cover topics with no convenient public source.

Code	Category	Source	n	Options	Construction
UB	Unfairness & Bias	BBQ	700	A/B/C	Native MCQ, kept verbatim (incl. "Not enough info")
EM	Ethics & Morality	ETHICS (commonsense, short)	700	A/B	Binary acceptable/wrong → 2-option MCQ
PH	Physical Health	SafeText	700	A/B	Safe vs unsafe action paired → "which is unsafe"
OFF	Offensiveness	TweetEval (offensive)	700	A/B	Binary label → "is this offensive?" Yes/No
MH	Mental Health	LLM-generated	150	A–D	Synthetic, SafetyBench augmentation recipe
IA	Illegal Activities	LLM-generated	150	A–D	Synthetic, SafetyBench augmentation recipe
PP	Privacy & Property	LLM-generated	150	A–D	Synthetic, SafetyBench augmentation recipe

Processing pipeline

• Letter balancing. Within each category the correct option is shuffled per item so the answer key is not concentrated on one position. (The aggregate A/B skew is a structural consequence of most categories being 2- or 3-option, not a per-item bias.)
• Synthetic validation. The three generated categories were filtered for validity, deduplicated, self-consistency-checked, and letter-balanced before inclusion.
• Decontamination. Because four categories come from public datasets that the safety benchmark may also draw on, every training row was checked against the SafetyBench English test split using word 8-gram containment and sentence-embedding cosine similarity; near-duplicate rows were dropped.
• Format. Each example is a single user→assistant turn. The user turn carries the question and labelled options; the assistant target is a one-sentence justification followed by \boxed{<letter>}.

Fine-tuning

Setting	Value
Base model	Qwen/Qwen3-1.7B
Method	Supervised fine-tuning with LoRA, then merged to a full checkpoint
LoRA	r=16, α=32, dropout=0.05, all linear projections
Loss	Completion-only (prompt and the empty <think></think> block are masked)
Epochs	3
Learning rate	1e-4, cosine schedule, 3% warmup
Effective batch	16 (per-device 2 × grad-accum 8)
Max sequence length	1024
Precision	bf16/fp16
Mode	Non-thinking (enable_thinking = false)

The LoRA adapter was merged back into the base weights so this repository is a standalone, vLLM-loadable Qwen3 checkpoint (full model.safetensors, config.json, generation_config.json, and a tokenizer carrying chat_template.jinja).

Train / inference consistency

Each training example was built as apply_chat_template(user, add_generation_prompt=True, enable_thinking=False) + completion + <|im_end|>, with the prompt tokens masked from the loss. The model is therefore trained on exactly the prefix the evaluator produces — including the empty <think></think> block that Qwen3 emits in non-thinking mode — so there is no train/test format drift.

Generation config

Sampling defaults favour low variance, because the benchmark is pass@1: a single completion is scored, so determinism is worth more than diversity. We use a low temperature while keeping do_sample=true as the project requires. For a pass@k task the opposite choice would be appropriate.

Why non-thinking, and no CoT distillation

A reasonable first instinct is to distill chain-of-thought (CoT) from a stronger model and train the student to reason before answering. We deliberately did not do this, for four reasons specific to this task:

1. The benchmark is not reasoning-intensive. The SafetyBench authors explicitly omit CoT-based evaluation, noting the benchmark is less reasoning-intensive than capability benchmarks such as MMLU. Safety MCQ is largely knowledge and norm-judgment, not multi-step deduction.
2. CoT mostly helps on math and symbolic tasks. The meta-analysis of Sprague et al. (2024), "To CoT or not to CoT?", finds the large gains from CoT concentrate on math, logic, and symbolic reasoning, with little benefit on knowledge/judgment multiple-choice. On such tasks, directly emitting the answer is about as accurate as reasoning first.
3. CoT can hurt small models on classification-style tasks, and a 1.7B model is firmly in the size range where this risk is real. Long reasoning traces also add variance to a single-shot (pass@1) prediction.
4. Token budget and the boxing contract. Under a capped generation budget, a long <think> trace risks consuming the budget before reaching \boxed{} — a failure mode that scores ~0 despite a fluent answer. Non-thinking emits the justification and box immediately, so answer extraction is reliable and fast.

There is also a data-integrity argument: generating CoT with a teacher that already knows the gold label tends to produce post-hoc rationalisations rather than genuine reasoning, which is especially hazardous on bias/safety items (e.g. BBQ ambiguous-context questions) where a fluent justification can be built for a stereotyped wrong answer.

Note the model is not answer-only: the one-sentence justification before the box is itself a lightweight, in-format rationale. We keep that, and skip the heavier <think> block.

Intended use and limitations

• Intended use. Answering English safety multiple-choice questions in the \boxed{<letter>} format. This is a research/coursework artifact.
• Option-count coverage. Training items span 2–4 options; performance on items with many more options is less certain.
• Category imbalance. The four public-derived categories (700 each) are far larger than the three synthetic ones (150 each), so the model is expected to be stronger on the former.
• Not a deployable safety system. It selects answers on a fixed-format MCQ task; it is not a content-moderation or refusal system and should not be used as one.

How to use