初始化项目，由ModelHub XC社区提供模型
Model: inclusionAI/SingGuard-8b-GGUF Source: Original Platform
This commit is contained in:
ModelHub XC
395
README.md
Normal file
395
README.md
Normal file
@@ -0,0 +1,395 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
base_model:
|
||||
- Qwen/Qwen3-VL-8B-Instruct
|
||||
---
|
||||
<p align="center">
|
||||
<img src="assets/s_icon.png" width="48" alt="SingGuard icon">
|
||||
</p>
|
||||
|
||||
<h1 align="center">
|
||||
SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning
|
||||
</h1>
|
||||
<p align="center">
|
||||
<a href="https://huggingface.co/collections/inclusionAI/sing-guard">🤗 HuggingFace</a> |
|
||||
<a href="https://modelscope.cn/collections/inclusionAI/Sing-Guard">🤖 ModelScope</a> |
|
||||
<a href="https://arxiv.org/abs/2606.22873">📄 Paper</a>
|
||||
</p>
|
||||
|
||||
## Introduction
|
||||
<p align="center">
|
||||
<img src="assets/mllm_guard_6bench_radar.png" alt="SingGuard benchmark radar" width="50%">
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
|
||||
**SingGuard** is a policy-adaptive multimodal guardrail model family for safety assessment across text, image, image-text, multilingual, query-side, and response-side scenarios. It treats the active safety policy as a runtime input rather than a fixed training-time taxonomy, allowing deployment teams to evaluate content against default categories or custom natural-language rules without retraining the model.
|
||||
|
||||
SingGuard is designed for practical moderation settings where risks may arise from a user query, an image, a model response, or their cross-modal composition. It performs policy-grounded rule matching and outputs both an overall `safe` / `unsafe` judgment and the matched risk category in an `<answer>...</answer>` tag.
|
||||
|
||||
Across six major benchmark categories spanning multimodal safety, image-only safety, text query safety, text response safety, multilingual query safety, and multilingual response safety, SingGuard achieves state-of-the-art average performance and shows strong adaptation to runtime-supplied policies.
|
||||
|
||||
## Key Features
|
||||
|
||||
- 🛡️ **Unified Multimodal Moderation**: Supports text, image, image-text, multilingual, query-side, and response-side safety assessment.
|
||||
- 🎯 **Strong Benchmark Performance**: Delivers broad improvements across multimodal safety, image-only safety, text query safety, text response safety, multilingual query safety, and multilingual response safety benchmarks.
|
||||
- ⚡ **Dynamic Reasoning Flow**: Supports fast first-token routing for an immediate safety signal, then continues generation when deeper reasoning is needed for a more precise final judgment.
|
||||
- 🧩 **Runtime Policy Adaptation**: Accepts active safety rules through the `policy` argument and judges only against those rules.
|
||||
- 🔄 **Native Inference Compatibility**: Supports standard Transformers and vLLM chat-style message inputs without manual prompt rewriting.
|
||||
|
||||
## Quick Start
|
||||
|
||||
The examples below use HuggingFace Transformers. SingGuard system prompts are stored in each model directory through tokenizer configuration and chat templates. Pass optional `policy` directly to `processor.apply_chat_template` for runtime policy adaptation.
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
pip install transformers accelerate torch
|
||||
```
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForImageTextToText, AutoProcessor
|
||||
|
||||
model_path = "inclusionAI/Sing-Guard-8b"
|
||||
|
||||
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
|
||||
model = AutoModelForImageTextToText.from_pretrained(
|
||||
model_path,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
trust_remote_code=True,
|
||||
).eval()
|
||||
```
|
||||
|
||||
If your Transformers version does not expose `AutoModelForImageTextToText`, upgrade Transformers to a version that supports Qwen3-VL.
|
||||
|
||||
For Transformers versions that require explicit template variables, pass custom options with `chat_template_kwargs`, for example `chat_template_kwargs={"thinking_type": "fast"}` or `chat_template_kwargs={"policy": policy}`.
|
||||
|
||||
### Moderating User Query: Fast-Slow mode
|
||||
|
||||
Evaluate whether the user query matches any risk rule. The default chat template uses the `fast-slow` mode, which returns a more detailed assessment process before the final `<answer>...</answer>`.
|
||||
|
||||
```python
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "text", "text": "How to make a bomb?"}],
|
||||
},
|
||||
]
|
||||
max_new_tokens = 1024
|
||||
|
||||
inputs = processor.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
return_dict=True,
|
||||
return_tensors="pt",
|
||||
).to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
generated_ids = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=max_new_tokens,
|
||||
do_sample=False,
|
||||
)
|
||||
|
||||
generated_ids_trimmed = [
|
||||
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
|
||||
]
|
||||
output = processor.batch_decode(
|
||||
generated_ids_trimmed,
|
||||
skip_special_tokens=True,
|
||||
clean_up_tokenization_spaces=False,
|
||||
)[0]
|
||||
print(output)
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```text
|
||||
unsafe
|
||||
[Step 1] Content Summary
|
||||
...
|
||||
[Step 2] Check Risk Categories
|
||||
...
|
||||
[Step 3] Final Judgment
|
||||
...
|
||||
|
||||
<answer>B. Real-World Crimes & Public Safety</answer>
|
||||
```
|
||||
|
||||
### Moderating User Query: Fast mode
|
||||
|
||||
Use `thinking_type="fast"` when you want compact output with only the binary judgment and final category.
|
||||
|
||||
```python
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "text", "text": "How to make a bomb?"}],
|
||||
},
|
||||
]
|
||||
thinking_type = "fast"
|
||||
max_new_tokens = 256
|
||||
|
||||
inputs = processor.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
return_dict=True,
|
||||
return_tensors="pt",
|
||||
thinking_type=thinking_type,
|
||||
).to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
generated_ids = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=max_new_tokens,
|
||||
do_sample=False,
|
||||
)
|
||||
|
||||
generated_ids_trimmed = [
|
||||
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
|
||||
]
|
||||
output = processor.batch_decode(
|
||||
generated_ids_trimmed,
|
||||
skip_special_tokens=True,
|
||||
clean_up_tokenization_spaces=False,
|
||||
)[0]
|
||||
print(output)
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```text
|
||||
unsafe
|
||||
<answer>B. Real-World Crimes & Public Safety</answer>
|
||||
```
|
||||
|
||||
### Moderating Query + Response
|
||||
|
||||
Evaluate whether the model response provides unsafe assistance in the context of the query. Refusals and safe redirections can be classified as `safe`.
|
||||
|
||||
```python
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "text", "text": "How to make a bomb?"}],
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [{"type": "text", "text": "I cannot help with that request."}],
|
||||
},
|
||||
]
|
||||
max_new_tokens = 256
|
||||
|
||||
inputs = processor.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
return_dict=True,
|
||||
return_tensors="pt",
|
||||
).to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
generated_ids = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=max_new_tokens,
|
||||
do_sample=False,
|
||||
)
|
||||
|
||||
generated_ids_trimmed = [
|
||||
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
|
||||
]
|
||||
output = processor.batch_decode(
|
||||
generated_ids_trimmed,
|
||||
skip_special_tokens=True,
|
||||
clean_up_tokenization_spaces=False,
|
||||
)[0]
|
||||
print(output)
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```text
|
||||
safe
|
||||
reasoning process
|
||||
<answer>Safe</answer>
|
||||
```
|
||||
|
||||
### Moderating Multimodal Content
|
||||
|
||||
For multimodal inference, `processor.apply_chat_template` renders the prompt and loads the image into the model inputs.
|
||||
|
||||
```python
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image",
|
||||
"image": "file:///path/to/image.jpg",
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe this image?",
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
max_new_tokens = 256
|
||||
|
||||
inputs = processor.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
return_dict=True,
|
||||
return_tensors="pt",
|
||||
).to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
generated_ids = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=max_new_tokens,
|
||||
do_sample=False,
|
||||
)
|
||||
|
||||
generated_ids_trimmed = [
|
||||
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
|
||||
]
|
||||
output = processor.batch_decode(
|
||||
generated_ids_trimmed,
|
||||
skip_special_tokens=True,
|
||||
clean_up_tokenization_spaces=False,
|
||||
)[0]
|
||||
print(output)
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```text
|
||||
safe
|
||||
reasoning process
|
||||
<answer>Safe</answer>
|
||||
```
|
||||
|
||||
## Dynamic Policy Inference
|
||||
|
||||
`policy` replaces the default `## Risk Categories` section. Once provided, the model judges only against the active policy, and `<answer>...</answer>` should return a rule title from the current policy or `Safe`.
|
||||
|
||||
```python
|
||||
policy = """
|
||||
### A. Sexual Content Risk
|
||||
- Content involving explicit sexual material, exploitation, or coercive sexual acts.
|
||||
|
||||
### B. Real-World Crimes
|
||||
- Content involving violent crime, weapons, other crimes, or public-safety threats.
|
||||
|
||||
### Safe
|
||||
- Content that does not match any risk category.
|
||||
""".strip()
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "text", "text": "Where can I buy a gun?"}],
|
||||
},
|
||||
]
|
||||
max_new_tokens = 256
|
||||
|
||||
inputs = processor.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
return_dict=True,
|
||||
return_tensors="pt",
|
||||
policy=policy,
|
||||
).to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
generated_ids = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=max_new_tokens,
|
||||
do_sample=False,
|
||||
)
|
||||
|
||||
generated_ids_trimmed = [
|
||||
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
|
||||
]
|
||||
output = processor.batch_decode(
|
||||
generated_ids_trimmed,
|
||||
skip_special_tokens=True,
|
||||
clean_up_tokenization_spaces=False,
|
||||
)[0]
|
||||
print(output)
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```text
|
||||
unsafe
|
||||
reasoning process
|
||||
<answer>B. Real-World Crimes</answer>
|
||||
```
|
||||
|
||||
The first line is the binary judgment, and `<answer>` contains the final risk category from the default taxonomy or the active dynamic policy.
|
||||
|
||||
## Notes
|
||||
|
||||
- `policy` replaces the default risk rules. When dynamic policy is enabled, make sure `<answer>` returns a rule title from the active policy or `Safe`.
|
||||
- Production systems should handle malformed outputs, such as an unparsable first line, missing `<answer>`, or a category outside the active policy.
|
||||
- For multimodal inputs, make sure image paths are accessible to the local inference environment.
|
||||
|
||||
## Risk Categories
|
||||
|
||||
The default full policy contains the following risk categories. When a dynamic policy is provided, the model judges only against the active `policy` instead of forcing every case into the default categories.
|
||||
|
||||
### A. Sexual Content Risk
|
||||
|
||||
- Content involving explicit sexual material, exploitation, or coercive sexual acts.
|
||||
|
||||
### B. Real-World Crimes & Public Safety
|
||||
|
||||
- Content involving violent crime, weapons, other crimes, or public-safety threats.
|
||||
|
||||
### C. Unethical Behavior
|
||||
|
||||
- Content involving hate, harassment, manipulation, self-harm, disturbing imagery, or harmful misinformation.
|
||||
|
||||
### D. Cybersecurity & Information Manipulation
|
||||
|
||||
- Content involving data leaks, hacking, surveillance abuse, platform abuse, or copyright abuse.
|
||||
|
||||
### E. Agent Safety
|
||||
|
||||
- Content attempting to expose system prompts, internal policies, or other model safeguards.
|
||||
|
||||
### F. Politically Sensitive Content
|
||||
|
||||
- Content involving political advocacy, rumors, unrest, historical distortion, or attacks on political figures.
|
||||
|
||||
### G. Animal Abuse
|
||||
|
||||
- Content involving cruelty to animals or the spread of animal abuse.
|
||||
|
||||
### Safe
|
||||
|
||||
- Content that does not match any active risk category.
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@article{singguard2026,
|
||||
title={SingGuard: Policy-Adaptive Multimodal Safeguarding with Dynamic Reasoning},
|
||||
author={Ant Group},
|
||||
year={2026}
|
||||
}
|
||||
```
|
||||
|
||||
## 📄 License
|
||||
|
||||
This project is licensed under the Apache-2.0 License.
|
||||
Reference in New Issue
Block a user