---
base_model: Qwen/Qwen3-0.6B
datasets:
- blackhao0426/user-preference-564k
language:
- en
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers
tags:
- preference-extraction
- personalization
- user-modeling
---

# Preference Extractor (Qwen3-0.6B Full SFT)

[![arXiv](https://img.shields.io/badge/arXiv-2603.20939-b31b1b.svg)](https://arxiv.org/abs/2603.20939)
[![Code](https://img.shields.io/badge/GitHub-VARS-black?logo=github)](https://github.com/YurenHao0426/VARS)
[![Dataset](https://img.shields.io/badge/🤗_Dataset-user--preference--564k-blue)](https://huggingface.co/datasets/blackhao0426/user-preference-564k)

A lightweight preference extraction model finetuned from [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B). Given a dialogue window, it outputs structured JSON preference tuples describing user style preferences.

This model was introduced in the paper [User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction](https://arxiv.org/abs/2603.20939). It is a core component of **VARS** (Vector-Adapted Retrieval Scoring), a framework for personalized LLM assistants.

## Task

Given recent conversation turns between a user and an AI assistant, extract user preferences as structured condition-action rules in JSON format:

```json
{
  "preferences": [
    {"condition": "when explaining math", "action": "show step-by-step derivation", "confidence": 0.9},
    {"condition": "general", "action": "respond in Chinese", "confidence": 0.85}
  ]
}
```

Each preference is a `(condition, action, confidence)` tuple:
- **condition**: when this preference applies (e.g., "when writing code", "general", "when debugging")
- **action**: the preferred behavior (e.g., "use Python with type hints", "show step-by-step derivation")
- **confidence**: extraction confidence score (0.0–1.0)

## Performance

Evaluated on a held-out test set:

| Metric | Value |
|--------|-------|
| JSON validity | 99.7% |
| Recall | 97.5% |
| Precision | 37.7% |
| Eval loss | 0.1611 |

The model intentionally favors **high recall** over precision: it over-extracts candidate preferences, and the downstream reranker and user vector in the VARS pipeline filter irrelevant ones.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "blackhao0426/pref-extractor-qwen3-0.6b-full-sft"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="bfloat16", device_map="auto", trust_remote_code=True)

# system_prompt should describe the extraction task
system_prompt = "You are a preference extractor. Given the dialogue window, extract preferences into JSON format."

conversation = """User: Can you explain the quicksort algorithm? I prefer step-by-step breakdowns with Python code.
Assistant: Sure! Here's quicksort step by step...
User: Great, but can you add type hints to the code?"""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": conversation},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)

# Decode only the newly generated tokens
result = tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)
# Parse JSON from result to get preference list
```

## Training

| Parameter | Value |
|-----------|-------|
| Base model | Qwen/Qwen3-0.6B |
| Training data | [blackhao0426/user-preference-564k](https://huggingface.co/datasets/blackhao0426/user-preference-564k) (564K examples) |
| Learning rate | 2e-05 |
| Batch size | 128 (32 per device x 4 GPUs) |
| Epochs | 1 |
| LR scheduler | Cosine with 5% warmup |
| Optimizer | AdamW (fused) |
| Framework | LLaMA-Factory |

## Citation

```bibtex
@misc{hao2026userpreferencemodelingconversational,
      title={User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction},
      author={Yuren Hao and Shuhaib Mehri and ChengXiang Zhai and Dilek Hakkani-Tür},
      year={2026},
      eprint={2603.20939},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.20939},
}
```