rl-cas-trl-agent/README.md

license, language, base_model, pipeline_tag, tags

license	language	base_model	pipeline_tag	tags
apache-2.0	en	Qwen/Qwen2.5-3B-Instruct	text-generation	reinforcement-learning grpo customer-service tool-use sft qlora

RL-CAS TRL Agent

Fine-tuned Qwen2.5-3B-Instruct that selects enterprise tools to resolve customer service queries. Trained via SFT + GRPO on expert trajectories from an RL-trained SAC agent.

Project: huggingface.co/spaces/praveenkrovvidi/rl-cas

Training Pipeline

1. SAC-Discrete (Round 1) -- MLP [256,256] trained for 3,000 episodes on a 14-signal dense reward environment with 13 actions (7 API tools + 5 terminal + 1 meta)
2. SFT (Round 2a) -- QLoRA fine-tuning on SAC expert trajectories (reward >= 5.0), 3 epochs, lr=2e-4
3. GRPO (Round 2b) -- RL with live environment rewards, 482 steps, lr=5e-5, 4 completions/prompt

Model Details

Base model	Qwen/Qwen2.5-3B-Instruct
Architecture	Qwen2ForCausalLM, 36 layers, 2048 hidden, GQA (16 heads / 2 KV)
Parameters	3B (fp16, 5.8 GB)
Context length	32,768 tokens
Fine-tuning	QLoRA (rank=16, alpha=32, 4-bit NF4) merged to fp16
Hardware	NVIDIA A10G (24 GB VRAM)

Input / Output

Input: System prompt + user query describing a customer service issue.

Output: JSON with chain-of-thought reasoning and action ID.

{"reasoning": "Customer needs order status. I should look up their order first.", "action_id": 0}

Action Space

ID	Action	Type
0	invoke_order_service	tool
1	invoke_refund_service	tool
2	invoke_customer_service	tool
3	invoke_auth_service	tool
4	invoke_logistics_service	tool
5	invoke_inventory_service	tool
6	invoke_knowledge_base	tool
7	resolve_query	terminal
8	request_customer_information	terminal
9	route_to_billing_department	terminal
10	route_to_technical_support	terminal
11	escalate_to_human_agent	terminal
12	consult_llm	meta

Benchmark (100 Queries, 8 Categories)

Agent	Avg Reward	Resolution	Avg Steps
GPT-4o-mini (zero-shot)	19.43	100%	2.5
SAC (MLP policy)	19.36	100%	3.0
Gemini-2.5-flash	16.46	100%	2.2
This model (TRL)	15.58	97%	3.1

Intended Use

Designed for the RL-CAS customer service environment. Selects tools from a fixed action space -- does not generate free-form customer responses.

Limitations

• 97% resolution rate (struggles with auth and refund multi-tool sequences)
• Requires the RL-CAS mock backend for tool execution
• Trained on synthetic customer service data, not real customer interactions

Citation

@misc{rlcas2026,
  title={RL-CAS: Reinforcement Learning for Customer Service Agent},
  author={Praveen Krovvidi},
  year={2026},
  url={https://huggingface.co/praveenkrovvidi/rl-cas-trl-agent}
}