115 lines
4.3 KiB
Markdown
115 lines
4.3 KiB
Markdown
---
|
||
base_model: Qwen/Qwen3-0.6B
|
||
datasets:
|
||
- blackhao0426/user-preference-564k
|
||
language:
|
||
- en
|
||
license: apache-2.0
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- preference-extraction
|
||
- personalization
|
||
- user-modeling
|
||
---
|
||
|
||
# Preference Extractor (Qwen3-0.6B Full SFT)
|
||
|
||
[](https://arxiv.org/abs/2603.20939)
|
||
[](https://github.com/YurenHao0426/VARS)
|
||
[](https://huggingface.co/datasets/blackhao0426/user-preference-564k)
|
||
|
||
A lightweight preference extraction model finetuned from [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B). Given a dialogue window, it outputs structured JSON preference tuples describing user style preferences.
|
||
|
||
This model was introduced in the paper [User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction](https://arxiv.org/abs/2603.20939). It is a core component of **VARS** (Vector-Adapted Retrieval Scoring), a framework for personalized LLM assistants.
|
||
|
||
## Task
|
||
|
||
Given recent conversation turns between a user and an AI assistant, extract user preferences as structured condition-action rules in JSON format:
|
||
|
||
```json
|
||
{
|
||
"preferences": [
|
||
{"condition": "when explaining math", "action": "show step-by-step derivation", "confidence": 0.9},
|
||
{"condition": "general", "action": "respond in Chinese", "confidence": 0.85}
|
||
]
|
||
}
|
||
```
|
||
|
||
Each preference is a `(condition, action, confidence)` tuple:
|
||
- **condition**: when this preference applies (e.g., "when writing code", "general", "when debugging")
|
||
- **action**: the preferred behavior (e.g., "use Python with type hints", "show step-by-step derivation")
|
||
- **confidence**: extraction confidence score (0.0–1.0)
|
||
|
||
## Performance
|
||
|
||
Evaluated on a held-out test set:
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| JSON validity | 99.7% |
|
||
| Recall | 97.5% |
|
||
| Precision | 37.7% |
|
||
| Eval loss | 0.1611 |
|
||
|
||
The model intentionally favors **high recall** over precision: it over-extracts candidate preferences, and the downstream reranker and user vector in the VARS pipeline filter irrelevant ones.
|
||
|
||
## Usage
|
||
|
||
```python
|
||
import torch
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model_name = "blackhao0426/pref-extractor-qwen3-0.6b-full-sft"
|
||
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
||
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto", trust_remote_code=True)
|
||
|
||
# system_prompt should describe the extraction task
|
||
system_prompt = "You are a preference extractor. Given the dialogue window, extract preferences into JSON format."
|
||
|
||
conversation = """User: Can you explain the quicksort algorithm? I prefer step-by-step breakdowns with Python code.
|
||
Assistant: Sure! Here's quicksort step by step...
|
||
User: Great, but can you add type hints to the code?"""
|
||
|
||
messages = [
|
||
{"role": "system", "content": system_prompt},
|
||
{"role": "user", "content": conversation},
|
||
]
|
||
|
||
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||
|
||
with torch.no_grad():
|
||
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
||
|
||
# Decode only the newly generated tokens
|
||
result = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
|
||
# Parse JSON from result to get preference list
|
||
```
|
||
|
||
## Training
|
||
|
||
| Parameter | Value |
|
||
|-----------|-------|
|
||
| Base model | Qwen/Qwen3-0.6B |
|
||
| Training data | [blackhao0426/user-preference-564k](https://huggingface.co/datasets/blackhao0426/user-preference-564k) (564K examples) |
|
||
| Learning rate | 2e-05 |
|
||
| Batch size | 128 (32 per device x 4 GPUs) |
|
||
| Epochs | 1 |
|
||
| LR scheduler | Cosine with 5% warmup |
|
||
| Optimizer | AdamW (fused) |
|
||
| Framework | LLaMA-Factory |
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{hao2026userpreferencemodelingconversational,
|
||
title={User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction},
|
||
author={Yuren Hao and Shuhaib Mehri and ChengXiang Zhai and Dilek Hakkani-Tür},
|
||
year={2026},
|
||
eprint={2603.20939},
|
||
archivePrefix={arXiv},
|
||
primaryClass={cs.CL},
|
||
url={https://arxiv.org/abs/2603.20939},
|
||
}
|
||
``` |