27 lines
1.1 KiB
Markdown
27 lines
1.1 KiB
Markdown
|
---
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
# Hybrid Policy Distillation for LLMs
|
||
|
|
|
||
|
|
This repository contains the weights for the model described in the paper [Hybrid Policy Distillation for LLMs](https://huggingface.co/papers/2604.20244).
|
||
|
|
|
||
|
|
Hybrid Policy Distillation (HPD) is a framework for compressing large language models (LLMs) that reformulates knowledge distillation (KD) as a reweighted log-likelihood objective at the token level. It integrates the complementary advantages of forward and reverse KL to balance mode coverage and mode-seeking, demonstrating improved computational efficiency and final performance across diverse model families and scales.
|
||
|
|
|
||
|
|
## Resources
|
||
|
|
- **Paper:** [Hybrid Policy Distillation for LLMs](https://huggingface.co/papers/2604.20244)
|
||
|
|
- **Code:** [GitHub Repository](https://github.com/zwhong714/Hybrid-Policy-Distillation)
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
|
||
|
|
If you find this work useful in your research, please cite:
|
||
|
|
|
||
|
|
```bibtex
|
||
|
|
@article{hong2024hybrid,
|
||
|
|
title={Hybrid Policy Distillation for LLMs},
|
||
|
|
author={Hong, Zhiwei and others},
|
||
|
|
journal={arXiv preprint arXiv:2604.20244},
|
||
|
|
year={2024}
|
||
|
|
}
|
||
|
|
```
|