Seed-AlfWorld-3B/README.md

---
license: other
license_name: qwen-research
license_link: https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE
language:
- en
metrics:
- accuracy
base_model:
- Qwen/Qwen2.5-3B-Instruct
library_name: transformers
pipeline_tag: text-generation
tags:
- agent
- reinforcement-learning
- long-horizon
- embodied-ai
- self-distillation
- on-policy-distillation
- alfworld
- seed
- qwen2.5
---

# SEED-ALFWorld-3B

## Overview

Built with Qwen2.5-3B-Instruct, SEED-ALFWorld-3B is trained for text-based
embodied interaction in ALFWorld using **SEED** (Self-Evolving On-Policy
Distillation). Training unfolds in two stages:

- **Hindsight-skill SFT:** the model learns to analyze a completed trajectory
  and summarize it as a concise, episode-level skill.
- **Self-evolving on-policy distillation:** in each reinforcement-learning
  iteration, the current policy both acts in the environment and analyzes the
  trajectories it has just collected. The generated skills are used to re-score
  the same sampled actions, providing dense token-level supervision alongside
  GRPO. Once updated, the improved policy takes on both roles in the next
  iteration, allowing its behavior and hindsight supervision to evolve
  together.

At inference time, the model acts from the standard interaction history alone,
with no analyzer, skill bank, retrieval module, or additional skill prompt.

## Key Features

- **Self-evolving supervision:** policy updates improve both action selection
  and the analyzer used to generate the next round of hindsight skills.
- **On-policy skill generation:** supervision follows the states, actions, and
  failure modes encountered by the current policy.
- **Dense token-level credit:** skill-induced log-probability shifts provide
  decision-specific guidance beyond terminal rewards.
- **No inference overhead:** privileged hindsight skills are used only during
  training and are internalized by the released policy.
## Performance

The following results are success rates (%) on the ALFWorld seen test
split. `Avg.` is the unweighted macro-average over the six task families. An
asterisk indicates that the method also receives a skill prompt at evaluation;
SEED does not use skills at evaluation.

| Method | Pick | Look | Clean | Heat | Cool | Pick2 | Avg. |
|---|---:|---:|---:|---:|---:|---:|---:|
| Vanilla | 44.4 | 11.1 | 6.2 | 15.4 | 28.6 | 12.5 | 21.9 |
| Skill-Prompt* | 51.7 | 66.7 | 48.4 | 0.0 | 4.3 | 10.0 | 28.9 |
| OPSD | 48.8 | 41.7 | 16.7 | 0.0 | 15.8 | 16.7 | 28.1 |
| GRPO | 91.2 | 62.5 | 96.2 | 61.9 | 65.0 | 47.4 | 75.0 |
| Skill-GRPO | 88.9 | 71.4 | 58.8 | 70.6 | 40.7 | 29.2 | 60.2 |
| Skill-GRPO* | 94.3 | 57.1 | 100.0 | 66.7 | 73.1 | 57.1 | 80.5 |
| GRPO+OPSD | 100.0 | 82.4 | 85.7 | 75.0 | 70.0 | 60.0 | 81.2 |
| Skill-SD | 88.2 | 50.0 | 96.2 | 52.4 | 65.0 | 57.9 | 73.4 |
| RLSD | 87.9 | 75.0 | 90.9 | 75.0 | 73.1 | 68.4 | 79.7 |
| SDAR | 97.1 | 62.5 | 100.0 | 61.9 | 75.0 | 84.2 | 84.4 |
| **SEED (ours)** | **100.0** | **100.0** | **100.0** | **100.0** | **70.6** | **80.0** | **91.8** |

On the ALFWorld unseen split, the same 3B SEED checkpoint reaches an
86.2 macro-average, compared with 70.9 for the matched GRPO baseline.

| Method | Pick | Look | Clean | Heat | Cool | Pick2 | Avg. |
|---|---:|---:|---:|---:|---:|---:|---:|
| ReAct | 17.4 | 6.7 | 8.8 | 7.4 | 9.1 | 0.0 | 8.2 |
| GRPO | 73.9 | 60.0 | 82.4 | 59.3 | 72.7 | 76.9 | 70.9 |
| **SEED (ours)** | **90.4** | **78.3** | **79.5** | **94.3** | **86.2** | **88.2** | **86.2** |

## Quickstart

Install the runtime dependencies:

```bash
pip install "transformers>=4.37.0" accelerate torch
```

Load the checkpoint and generate one ALFWorld action:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Jinyang23/Seed-AlfWorld-3B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)


def build_alfworld_prompt(
    task_description,
    step_count,
    history_length,
    action_history,
    current_step,
    current_observation,
    admissible_actions,
):
    actions = ", ".join(admissible_actions)
    return f"""You are an expert agent operating in the ALFRED Embodied Environment. Your task is
to: {task_description}
Prior to this step, you have already taken {step_count} step(s). Below are the most recent
{history_length} observations and the corresponding actions you took: {action_history}
You are now at step {current_step} and your current observation is: {current_observation}
Your admissible actions of the current situation are: [{actions}].

Now it's your turn to take an action.

You should first reason step-by-step about the current situation. This reasoning process
MUST be enclosed within <think> </think> tags.
Once you've finished your reasoning, you should choose an admissible action for current
step and present it within <action> </action> tags."""


prompt = build_alfworld_prompt(
    task_description="put a candle in toilet.",
    step_count=0,
    history_length=0,
    action_history="None.",
    current_step=1,
    current_observation="You are in the middle of a room.",
    admissible_actions=["look", "go to shelf 1", "go to toilet 1"],
)

text = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=False,
)
new_tokens = outputs[:, inputs.input_ids.shape[1]:]
response = tokenizer.batch_decode(new_tokens, skip_special_tokens=True)[0]
print(response)
```

In a complete evaluation loop, replace the example values with the current
ALFWorld task, recent observation-action history, observation, and admissible
actions at every step. Only execute the action emitted inside `<action>` tags.

The model can also be served with vLLM:

```bash
vllm serve Jinyang23/Seed-AlfWorld-3B \
  --served-model-name Seed-AlfWorld-3B \
  --dtype bfloat16
```

## Limitations

- The model is specialized for ALFWorld-style text interaction and may not
  transfer reliably to unrelated environments or prompting formats.

## License

This checkpoint is a derivative of Qwen2.5-3B-Instruct and is distributed under
the [Qwen Research License Agreement](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE).

## Citation

Citation metadata will be added after the public paper release.

## Acknowledgements

The SEED implementation builds on
[veRL](https://github.com/volcengine/verl),
[verl-agent](https://github.com/langfengQ/verl-agent), and
[SDAR](https://github.com/ZJU-REAL/SDAR).

## Links

- Model: https://huggingface.co/Jinyang23/Seed-AlfWorld-3B
- Base model: https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
- Paper: https://arxiv.org/pdf/2607.14777
- Code: https://github.com/jinyangwu/SEED