77 lines
2.2 KiB
Markdown
77 lines
2.2 KiB
Markdown
---
|
|
language:
|
|
- hi
|
|
- ta
|
|
- en
|
|
tags:
|
|
- customer-support
|
|
- hinglish
|
|
- tanglish
|
|
- fine-tuned
|
|
- qwen2.5
|
|
- qlora
|
|
base_model: Qwen/Qwen2.5-3B-Instruct
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# SupportLM — Hinglish/Tanglish Customer Support Model
|
|
|
|
A fine-tuned Qwen2.5-3B model for Indian e-commerce customer support
|
|
in code-switched language (Hinglish and Tanglish).
|
|
|
|
## What it does
|
|
|
|
Given a customer support message in Hinglish or Tanglish, the model:
|
|
1. Classifies the issue type into one of 8 categories
|
|
2. Extracts the order ID if present in the message
|
|
3. Generates an empathetic response in the same language the customer used
|
|
|
|
## Benchmark Results
|
|
|
|
Evaluated on 159 held-out test examples never seen during training.
|
|
|
|
| Metric | GPT-4o-mini | SFT Model | DPO Model |
|
|
|--------|-------------|-----------|-----------|
|
|
| Classification accuracy | 74.8% | **79.9%** | 78.0% |
|
|
| Order ID extraction | 98.7% | **99.4%** | 99.4% |
|
|
| Language match rate | 96.2% | 99.4% | **100.0%** |
|
|
|
|
SFT model chosen for deployment — best classification accuracy.
|
|
|
|
## Issue Types
|
|
|
|
order_status, return_request, payment_issue, delivery_complaint,
|
|
product_defect, cancellation_request, wrong_item, discount_query
|
|
|
|
## Training Details
|
|
|
|
- Base model: Qwen/Qwen2.5-3B-Instruct
|
|
- Training data: 1277 synthetic Hinglish/Tanglish examples
|
|
- Training method: QLoRA (r=16, alpha=32) + DPO (beta=0.1)
|
|
- Hardware: Google Colab T4 GPU (free tier)
|
|
- Training time: ~40 minutes total (SFT + DPO)
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
import json
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("GuruVarshini/supportlm-qwen2.5-3b")
|
|
tokenizer = AutoTokenizer.from_pretrained("GuruVarshini/supportlm-qwen2.5-3b")
|
|
|
|
message = "bhai mere order ka kya hua yaar 3 din ho gaye"
|
|
|
|
messages = [
|
|
{"role": "system", "content": "You are a customer support agent for an Indian e-commerce platform. Given a customer message, respond with a JSON object containing issue_type, order_id, and response."},
|
|
{"role": "user", "content": message}
|
|
]
|
|
|
|
input_ids = tokenizer.apply_chat_template(
|
|
messages,
|
|
return_tensors="pt",
|
|
add_generation_prompt=True
|
|
)
|
|
output = model.generate(input_ids, max_new_tokens=200)
|
|
print(tokenizer.decode(output[0][input_ids.shape[1]:], skip_special_tokens=True))
|