115 lines
3.7 KiB
Markdown
115 lines
3.7 KiB
Markdown
---
|
||
license: mit
|
||
language:
|
||
- en
|
||
tags:
|
||
- reinforcement-learning
|
||
- grpo
|
||
- cloud-management
|
||
- multi-agent
|
||
- sustainability
|
||
- openenv
|
||
base_model: Qwen/Qwen2.5-0.5B-Instruct
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# ⚡ CloudEdge GRPO Controller
|
||
|
||
**A Qwen2.5-0.5B model fine-tuned with Group Relative Policy Optimization (GRPO) to manage cloud infrastructure crises.**
|
||
|
||
Built for the **Meta PyTorch OpenEnv Hackathon Grand Finale**.
|
||
|
||
## Model Description
|
||
|
||
This model is a reinforcement-learning-trained controller for the **CloudEdge** cloud crisis simulator. It learns to select optimal infrastructure actions (scale_up, scale_down, optimize_energy, migrate_region) by balancing three competing objectives:
|
||
|
||
| Objective | Target | Agent |
|
||
|-----------|--------|-------|
|
||
| Latency | < 150ms | ResourceAgent |
|
||
| Cost | < $400/hr | CostAgent |
|
||
| Carbon | < 220 units | SustainabilityAgent |
|
||
|
||
### Training Method
|
||
|
||
- **Algorithm:** GRPO (Group Relative Policy Optimization) via [TRL](https://github.com/huggingface/trl)
|
||
- **Base Model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
|
||
- **Training Steps:** 512
|
||
- **Generations per prompt:** 4
|
||
- **Reward Function:** Shaped multi-objective reward with gap closure + worst-metric bonus
|
||
|
||
### Shaped Reward Function
|
||
|
||
```
|
||
reward = Σ (gap_closure × weight) + worst_metric_bonus
|
||
```
|
||
|
||
| Action | Reward (crisis state) | Model Learns |
|
||
|--------|----------------------|-------------|
|
||
| `optimize_energy` | **+7.5** | "Best action — addresses cost + carbon simultaneously" |
|
||
| `scale_down` | **+5.75** | "Good — reduces cost effectively" |
|
||
| `migrate_region` | **+3.75** | "Moderate — helps carbon but hurts cost" |
|
||
| `scale_up` | **+1.5** | "Worst — increases cost and carbon" |
|
||
|
||
### Training Results
|
||
|
||
The model converged to selecting `optimize_energy` as the dominant policy when all metrics are above target — which is the mathematically optimal action given the shaped reward function.
|
||
|
||
## How to Use
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model = AutoModelForCausalLM.from_pretrained("kartikraut09/ecocloud-grpo-qwen")
|
||
tokenizer = AutoTokenizer.from_pretrained("kartikraut09/ecocloud-grpo-qwen")
|
||
|
||
prompt = """<|im_start|>system
|
||
You are the CloudEdge controller managing a cloud platform in crisis.
|
||
Pick the BEST single action for the current state. Respond with ONLY the action name.
|
||
|
||
Actions:
|
||
scale_up → latency -40, cost +30, carbon +20
|
||
scale_down → latency +25, cost -35, carbon -15
|
||
optimize_energy → latency +10, cost -20, carbon -40
|
||
migrate_region → latency +15, cost +10, carbon -50
|
||
|
||
Targets: latency<150ms, cost<$400, carbon<220<|im_end|>
|
||
<|im_start|>user
|
||
Cloud state: latency=280ms, cost=$620/hr, carbon=380, load=critical. Best action?<|im_end|>
|
||
<|im_start|>assistant
|
||
"""
|
||
|
||
inputs = tokenizer(prompt, return_tensors="pt")
|
||
outputs = model.generate(**inputs, max_new_tokens=16, temperature=0.1)
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
# Output: optimize_energy
|
||
```
|
||
|
||
## Technical Details
|
||
|
||
- **Architecture:** Qwen2 (0.5B parameters)
|
||
- **Framework:** PyTorch + HuggingFace Transformers + TRL
|
||
- **Environment:** OpenEnv-compatible Gymnasium-style simulator
|
||
- **Training Hardware:** Google Colab T4 GPU
|
||
- **Training Time:** ~15 minutes (512 steps)
|
||
|
||
## Project Links
|
||
|
||
- **GitHub:** [KartikRaut09/ecocloud-war-room](https://github.com/KartikRaut09/ecocloud-war-room)
|
||
- **Hackathon:** Meta PyTorch OpenEnv Hackathon Grand Finale
|
||
- **Themes:** Multi-Agent Interactions · Long-Horizon Planning · World Modeling
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{cloudedge2026,
|
||
title={CloudEdge: Multi-Agent LLM Simulator for Sustainable Cloud Crisis Management},
|
||
author={Kartik Raut},
|
||
year={2026},
|
||
url={https://github.com/KartikRaut09/ecocloud-war-room}
|
||
}
|
||
```
|
||
|
||
## License
|
||
|
||
MIT License
|