81 lines
3.4 KiB
Markdown
81 lines
3.4 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
base_model:
|
|
- Klingspor/Qwen3-4B-SFT
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- 20-questions
|
|
- rl
|
|
- grpo
|
|
- starpo
|
|
- multi-turn
|
|
- information-seeking
|
|
- reinforcement-learning
|
|
- credit-assignment
|
|
---
|
|
# 20 Questions StarPO - Qwen3-4B
|
|
|
|
This model is a reinforcement learning fine-tuned version of [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) for the **20 Questions** task, trained with **StarPO** (a variant of GRPO adapted for multi-turn settings). Released as part of the paper *"Intrinsic Credit Assignment for Long Horizon Interaction"*.
|
|
|
|
## Overview
|
|
|
|
The model plays the role of a **Questioner** in a game of 20 Questions: it asks up to 20 yes-or-no questions to deduce a secret word (a common English noun). This model was trained with RL starting from the [SFT checkpoint](https://huggingface.co/bethgelab/20q-sft-qwen3-4b) and serves as a baseline in the paper.
|
|
|
|
## Training
|
|
|
|
- **Base model:** [Qwen3-4B-SFT](https://huggingface.co/Klingspor/Qwen3-4B-SFT) (SFT on Qwen3-4B)
|
|
- **Method:** StarPO (multi-turn GRPO / Group Relative Policy Optimization)
|
|
- **Training data:** 1,000 words from the COCA+ RL training set (no overlap with test set)
|
|
- **Test set:** 433 held-out words from the Gemini test split
|
|
- **Judge/Oracle:** Qwen3-14B (with chain-of-thought reasoning)
|
|
- **Framework:** [VERL](https://github.com/volcengine/verl) (Volcano Engine Reinforcement Learning)
|
|
|
|
## Intended Use
|
|
|
|
This model is intended for:
|
|
- Playing 20 Questions as a questioner agent
|
|
- Research on multi-turn interactive language agents and RL for LLMs
|
|
- Comparison baseline for credit assignment methods in multi-step RL
|
|
|
|
## How to Use
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_name = "Klingspor/StarPO-4B"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
model = AutoModelForCausalLM.from_pretrained(model_name)
|
|
|
|
system_prompt = """You are the Questioner in a game of 20 Questions, and your goal is to determine the secret word.
|
|
The secret is randomly drawn from the most frequent nouns of the English language.
|
|
|
|
Ask clear, concise, and strategic yes/no questions that will help you narrow down the possibilities.
|
|
Consider previous answers to inform your subsequent questions, and keep track of the information you gather.
|
|
Focus on deductive reasoning, start with a broad question and refine your queries as you progress."""
|
|
|
|
user_prompt = """Ask a question to gain additional information about the secret or guess what the secret is.
|
|
|
|
Instructions:
|
|
1. Ask a question that can be answered with "Yes" or "No" to help you deduce the secret word.
|
|
2. Your answer must be a single question. Do not provide any additional commentary or reasoning.
|
|
|
|
Ask your question: """
|
|
|
|
messages = [
|
|
{"role": "system", "content": system_prompt},
|
|
{"role": "user", "content": user_prompt},
|
|
]
|
|
|
|
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|
inputs = tokenizer(text, return_tensors="pt")
|
|
outputs = model.generate(**inputs, max_new_tokens=128)
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
## Links
|
|
|
|
- **Paper:** [Intrinsic Credit Assignment for Long Horizon Interaction](https://bethgelab.github.io/delta-belief-rl/)
|
|
- **Code:** [github.com/bethgelab/delta-belief-rl](https://github.com/bethgelab/delta-belief-rl/)
|
|
- **Model collection:** [bethgelab/delta-belief-rl](https://huggingface.co/collections/bethgelab/delta-belief-rl) |