115 lines
4.3 KiB
Markdown
115 lines
4.3 KiB
Markdown
|
---
|
|||
|
|
base_model: Qwen/Qwen3-0.6B
|
|||
|
|
datasets:
|
|||
|
|
- blackhao0426/user-preference-564k
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
license: apache-2.0
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
library_name: transformers
|
|||
|
|
tags:
|
|||
|
|
- preference-extraction
|
|||
|
|
- personalization
|
|||
|
|
- user-modeling
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Preference Extractor (Qwen3-0.6B Full SFT)
|
|||
|
|
|
|||
|
|
[](https://arxiv.org/abs/2603.20939)
|
|||
|
|
[](https://github.com/YurenHao0426/VARS)
|
|||
|
|
[](https://huggingface.co/datasets/blackhao0426/user-preference-564k)
|
|||
|
|
|
|||
|
|
A lightweight preference extraction model finetuned from [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B). Given a dialogue window, it outputs structured JSON preference tuples describing user style preferences.
|
|||
|
|
|
|||
|
|
This model was introduced in the paper [User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction](https://arxiv.org/abs/2603.20939). It is a core component of **VARS** (Vector-Adapted Retrieval Scoring), a framework for personalized LLM assistants.
|
|||
|
|
|
|||
|
|
## Task
|
|||
|
|
|
|||
|
|
Given recent conversation turns between a user and an AI assistant, extract user preferences as structured condition-action rules in JSON format:
|
|||
|
|
|
|||
|
|
```json
|
|||
|
|
{
|
|||
|
|
"preferences": [
|
|||
|
|
{"condition": "when explaining math", "action": "show step-by-step derivation", "confidence": 0.9},
|
|||
|
|
{"condition": "general", "action": "respond in Chinese", "confidence": 0.85}
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Each preference is a `(condition, action, confidence)` tuple:
|
|||
|
|
- **condition**: when this preference applies (e.g., "when writing code", "general", "when debugging")
|
|||
|
|
- **action**: the preferred behavior (e.g., "use Python with type hints", "show step-by-step derivation")
|
|||
|
|
- **confidence**: extraction confidence score (0.0–1.0)
|
|||
|
|
|
|||
|
|
## Performance
|
|||
|
|
|
|||
|
|
Evaluated on a held-out test set:
|
|||
|
|
|
|||
|
|
| Metric | Value |
|
|||
|
|
|--------|-------|
|
|||
|
|
| JSON validity | 99.7% |
|
|||
|
|
| Recall | 97.5% |
|
|||
|
|
| Precision | 37.7% |
|
|||
|
|
| Eval loss | 0.1611 |
|
|||
|
|
|
|||
|
|
The model intentionally favors **high recall** over precision: it over-extracts candidate preferences, and the downstream reranker and user vector in the VARS pipeline filter irrelevant ones.
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import torch
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
model_name = "blackhao0426/pref-extractor-qwen3-0.6b-full-sft"
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto", trust_remote_code=True)
|
|||
|
|
|
|||
|
|
# system_prompt should describe the extraction task
|
|||
|
|
system_prompt = "You are a preference extractor. Given the dialogue window, extract preferences into JSON format."
|
|||
|
|
|
|||
|
|
conversation = """User: Can you explain the quicksort algorithm? I prefer step-by-step breakdowns with Python code.
|
|||
|
|
Assistant: Sure! Here's quicksort step by step...
|
|||
|
|
User: Great, but can you add type hints to the code?"""
|
|||
|
|
|
|||
|
|
messages = [
|
|||
|
|
{"role": "system", "content": system_prompt},
|
|||
|
|
{"role": "user", "content": conversation},
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|||
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|||
|
|
|
|||
|
|
with torch.no_grad():
|
|||
|
|
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
|||
|
|
|
|||
|
|
# Decode only the newly generated tokens
|
|||
|
|
result = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
|
|||
|
|
# Parse JSON from result to get preference list
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
|
|||
|
|
| Parameter | Value |
|
|||
|
|
|-----------|-------|
|
|||
|
|
| Base model | Qwen/Qwen3-0.6B |
|
|||
|
|
| Training data | [blackhao0426/user-preference-564k](https://huggingface.co/datasets/blackhao0426/user-preference-564k) (564K examples) |
|
|||
|
|
| Learning rate | 2e-05 |
|
|||
|
|
| Batch size | 128 (32 per device x 4 GPUs) |
|
|||
|
|
| Epochs | 1 |
|
|||
|
|
| LR scheduler | Cosine with 5% warmup |
|
|||
|
|
| Optimizer | AdamW (fused) |
|
|||
|
|
| Framework | LLaMA-Factory |
|
|||
|
|
|
|||
|
|
## Citation
|
|||
|
|
|
|||
|
|
```bibtex
|
|||
|
|
@misc{hao2026userpreferencemodelingconversational,
|
|||
|
|
title={User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction},
|
|||
|
|
author={Yuren Hao and Shuhaib Mehri and ChengXiang Zhai and Dilek Hakkani-Tür},
|
|||
|
|
year={2026},
|
|||
|
|
eprint={2603.20939},
|
|||
|
|
archivePrefix={arXiv},
|
|||
|
|
primaryClass={cs.CL},
|
|||
|
|
url={https://arxiv.org/abs/2603.20939},
|
|||
|
|
}
|
|||
|
|
```
|