96 lines
3.0 KiB
Markdown
96 lines
3.0 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
base_model: Qwen/Qwen2.5-3B-Instruct
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- reinforcement-learning
|
||
|
|
- grpo
|
||
|
|
- customer-service
|
||
|
|
- tool-use
|
||
|
|
- sft
|
||
|
|
- qlora
|
||
|
|
---
|
||
|
|
|
||
|
|
# RL-CAS TRL Agent
|
||
|
|
|
||
|
|
Fine-tuned **Qwen2.5-3B-Instruct** that selects enterprise tools to resolve customer service queries. Trained via **SFT + GRPO** on expert trajectories from an RL-trained SAC agent.
|
||
|
|
|
||
|
|
> **Project:** [huggingface.co/spaces/praveenkrovvidi/rl-cas](https://huggingface.co/spaces/praveenkrovvidi/rl-cas)
|
||
|
|
|
||
|
|
## Training Pipeline
|
||
|
|
|
||
|
|
1. **SAC-Discrete** (Round 1) -- MLP [256,256] trained for 3,000 episodes on a 14-signal dense reward environment with 13 actions (7 API tools + 5 terminal + 1 meta)
|
||
|
|
2. **SFT** (Round 2a) -- QLoRA fine-tuning on SAC expert trajectories (reward >= 5.0), 3 epochs, lr=2e-4
|
||
|
|
3. **GRPO** (Round 2b) -- RL with live environment rewards, 482 steps, lr=5e-5, 4 completions/prompt
|
||
|
|
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
| | |
|
||
|
|
|---|---|
|
||
|
|
| **Base model** | Qwen/Qwen2.5-3B-Instruct |
|
||
|
|
| **Architecture** | Qwen2ForCausalLM, 36 layers, 2048 hidden, GQA (16 heads / 2 KV) |
|
||
|
|
| **Parameters** | 3B (fp16, 5.8 GB) |
|
||
|
|
| **Context length** | 32,768 tokens |
|
||
|
|
| **Fine-tuning** | QLoRA (rank=16, alpha=32, 4-bit NF4) merged to fp16 |
|
||
|
|
| **Hardware** | NVIDIA A10G (24 GB VRAM) |
|
||
|
|
|
||
|
|
## Input / Output
|
||
|
|
|
||
|
|
**Input:** System prompt + user query describing a customer service issue.
|
||
|
|
|
||
|
|
**Output:** JSON with chain-of-thought reasoning and action ID.
|
||
|
|
|
||
|
|
```json
|
||
|
|
{"reasoning": "Customer needs order status. I should look up their order first.", "action_id": 0}
|
||
|
|
```
|
||
|
|
|
||
|
|
### Action Space
|
||
|
|
|
||
|
|
| ID | Action | Type |
|
||
|
|
|----|--------|------|
|
||
|
|
| 0 | `invoke_order_service` | tool |
|
||
|
|
| 1 | `invoke_refund_service` | tool |
|
||
|
|
| 2 | `invoke_customer_service` | tool |
|
||
|
|
| 3 | `invoke_auth_service` | tool |
|
||
|
|
| 4 | `invoke_logistics_service` | tool |
|
||
|
|
| 5 | `invoke_inventory_service` | tool |
|
||
|
|
| 6 | `invoke_knowledge_base` | tool |
|
||
|
|
| 7 | `resolve_query` | terminal |
|
||
|
|
| 8 | `request_customer_information` | terminal |
|
||
|
|
| 9 | `route_to_billing_department` | terminal |
|
||
|
|
| 10 | `route_to_technical_support` | terminal |
|
||
|
|
| 11 | `escalate_to_human_agent` | terminal |
|
||
|
|
| 12 | `consult_llm` | meta |
|
||
|
|
|
||
|
|
## Benchmark (100 Queries, 8 Categories)
|
||
|
|
|
||
|
|
| Agent | Avg Reward | Resolution | Avg Steps |
|
||
|
|
|-------|-----------|-----------|----------|
|
||
|
|
| GPT-4o-mini (zero-shot) | 19.43 | 100% | 2.5 |
|
||
|
|
| SAC (MLP policy) | 19.36 | 100% | 3.0 |
|
||
|
|
| Gemini-2.5-flash | 16.46 | 100% | 2.2 |
|
||
|
|
| **This model (TRL)** | **15.58** | **97%** | **3.1** |
|
||
|
|
|
||
|
|
## Intended Use
|
||
|
|
|
||
|
|
Designed for the RL-CAS customer service environment. Selects tools from a fixed action space -- does **not** generate free-form customer responses.
|
||
|
|
|
||
|
|
## Limitations
|
||
|
|
|
||
|
|
- 97% resolution rate (struggles with `auth` and `refund` multi-tool sequences)
|
||
|
|
- Requires the RL-CAS mock backend for tool execution
|
||
|
|
- Trained on synthetic customer service data, not real customer interactions
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
|
||
|
|
```bibtex
|
||
|
|
@misc{rlcas2026,
|
||
|
|
title={RL-CAS: Reinforcement Learning for Customer Service Agent},
|
||
|
|
author={Praveen Krovvidi},
|
||
|
|
year={2026},
|
||
|
|
url={https://huggingface.co/praveenkrovvidi/rl-cas-trl-agent}
|
||
|
|
}
|
||
|
|
```
|