---
license: apache-2.0
language:
  - en
base_model: Qwen/Qwen2.5-3B-Instruct
pipeline_tag: text-generation
tags:
  - reinforcement-learning
  - grpo
  - customer-service
  - tool-use
  - sft
  - qlora
---

# RL-CAS TRL Agent

Fine-tuned **Qwen2.5-3B-Instruct** that selects enterprise tools to resolve customer service queries. Trained via **SFT + GRPO** on expert trajectories from an RL-trained SAC agent.

> **Project:** [huggingface.co/spaces/praveenkrovvidi/rl-cas](https://huggingface.co/spaces/praveenkrovvidi/rl-cas)

## Training Pipeline

1. **SAC-Discrete** (Round 1) -- MLP [256,256] trained for 3,000 episodes on a 14-signal dense reward environment with 13 actions (7 API tools + 5 terminal + 1 meta)
2. **SFT** (Round 2a) -- QLoRA fine-tuning on SAC expert trajectories (reward >= 5.0), 3 epochs, lr=2e-4
3. **GRPO** (Round 2b) -- RL with live environment rewards, 482 steps, lr=5e-5, 4 completions/prompt

## Model Details

| | |
|---|---|
| **Base model** | Qwen/Qwen2.5-3B-Instruct |
| **Architecture** | Qwen2ForCausalLM, 36 layers, 2048 hidden, GQA (16 heads / 2 KV) |
| **Parameters** | 3B (fp16, 5.8 GB) |
| **Context length** | 32,768 tokens |
| **Fine-tuning** | QLoRA (rank=16, alpha=32, 4-bit NF4) merged to fp16 |
| **Hardware** | NVIDIA A10G (24 GB VRAM) |

## Input / Output

**Input:** System prompt + user query describing a customer service issue.

**Output:** JSON with chain-of-thought reasoning and action ID.

```json
{"reasoning": "Customer needs order status. I should look up their order first.", "action_id": 0}
```

### Action Space

| ID | Action | Type |
|----|--------|------|
| 0 | `invoke_order_service` | tool |
| 1 | `invoke_refund_service` | tool |
| 2 | `invoke_customer_service` | tool |
| 3 | `invoke_auth_service` | tool |
| 4 | `invoke_logistics_service` | tool |
| 5 | `invoke_inventory_service` | tool |
| 6 | `invoke_knowledge_base` | tool |
| 7 | `resolve_query` | terminal |
| 8 | `request_customer_information` | terminal |
| 9 | `route_to_billing_department` | terminal |
| 10 | `route_to_technical_support` | terminal |
| 11 | `escalate_to_human_agent` | terminal |
| 12 | `consult_llm` | meta |

## Benchmark (100 Queries, 8 Categories)

| Agent | Avg Reward | Resolution | Avg Steps |
|-------|-----------|-----------|----------|
| GPT-4o-mini (zero-shot) | 19.43 | 100% | 2.5 |
| SAC (MLP policy) | 19.36 | 100% | 3.0 |
| Gemini-2.5-flash | 16.46 | 100% | 2.2 |
| **This model (TRL)** | **15.58** | **97%** | **3.1** |

## Intended Use

Designed for the RL-CAS customer service environment. Selects tools from a fixed action space -- does **not** generate free-form customer responses.

## Limitations

- 97% resolution rate (struggles with `auth` and `refund` multi-tool sequences)
- Requires the RL-CAS mock backend for tool execution
- Trained on synthetic customer service data, not real customer interactions

## Citation

```bibtex
@misc{rlcas2026,
  title={RL-CAS: Reinforcement Learning for Customer Service Agent},
  author={Praveen Krovvidi},
  year={2026},
  url={https://huggingface.co/praveenkrovvidi/rl-cas-trl-agent}
}
```