125 lines
4.0 KiB
Markdown
125 lines
4.0 KiB
Markdown
---
|
|
library_name: transformers
|
|
base_model:
|
|
- openai-community/gpt2
|
|
---
|
|
# Growing LLM Model Card
|
|
|
|
## Model Description
|
|
|
|
The **Growing LLM** is a GPT-2 based language model that implements neural plasticity-inspired dynamic growth during training. This model starts with a pre-trained GPT-2 (124M parameters) and dynamically adds new transformer blocks while freezing the original parameters, allowing the model to acquire new knowledge without catastrophic forgetting.
|
|
|
|
### Key Features
|
|
|
|
- **Dynamic Growth**: Adds new transformer blocks during training
|
|
- **Knowledge Preservation**: Freezes original parameters to retain pre-trained knowledge
|
|
- **Flexible Triggers**: Supports fixed schedule and plateau detection growth triggers
|
|
- **Regularization Options**: Supports Knowledge Distillation and Elastic Weight Consolidation (EWC)
|
|
- **Comprehensive Metrics**: Tracks training, validation, growth events, and scaling analysis
|
|
|
|
## Training Details
|
|
|
|
### Training Data
|
|
- Dataset: WikiText-2-raw-v1
|
|
- Max sequence length: 128 tokens
|
|
|
|
### Training Configuration
|
|
- Base model: GPT-2 (124M parameters)
|
|
- Learning rate: 5e-5
|
|
- Batch size: 8
|
|
- Optimizer: AdamW with weight decay 0.01
|
|
- Max steps: 2000
|
|
- Growth frequency: Every 500 steps
|
|
- Maximum growth events: 3
|
|
|
|
### Growth Mechanism
|
|
1. **Fixed Schedule**: Grow every N training steps
|
|
2. **Plateau Detection**: Grow when validation loss shows no improvement for Y steps
|
|
|
|
### Regularization (Optional)
|
|
- **Knowledge Distillation**: Uses teacher-student architecture with temperature scaling
|
|
- **Elastic Weight Consolidation (EWC)**: Penalizes changes to important parameters
|
|
|
|
## Model Architecture
|
|
|
|
- Base: GPT-2 (12 layers, 12 heads, 768 hidden dim)
|
|
- Growth: Added 3 new transformer blocks (one per growth event)
|
|
- Final: 15 layers, 145.7M total parameters
|
|
|
|
## Training Results
|
|
|
|
### Summary Metrics
|
|
| Metric | Initial | Final |
|
|
|--------|---------|-------|
|
|
| Training Loss | 7.16 | 1.95 |
|
|
| Validation Loss | 6.99 | 2.03 |
|
|
| Validation Perplexity | ~1000 | 7.58 |
|
|
| Total Parameters | 124.4M | 145.7M |
|
|
|
|
### Training Time
|
|
- Total time: ~60 minutes (3596 seconds)
|
|
- Best validation loss: 2.00
|
|
- Best validation perplexity: 7.42
|
|
|
|
### Growth Events
|
|
| Growth # | Step | Layers | Parameters Added | Val Loss Delta |
|
|
|---------|------|--------|-----------------|----------------|
|
|
| 1 | 500 | 12 → 13 | +7.1M | +0.00003 |
|
|
| 2 | 1000 | 13 → 14 | +7.1M | +0.00002 |
|
|
| 3 | 1500 | 14 → 15 | +7.1M | +0.000001 |
|
|
|
|
|
|
### RESULTS SUMMARY
|
|
|
|
| Model | Perplexity | Loss |
|
|
|-------|------------|------|
|
|
| Base GPT-2 | 56.39 | 4.0323 |
|
|
| Growing LLM | 33.39 | 3.5082 |
|
|
|
|
Perplexity improvement: 40.8%
|
|
|
|
**Key Observation**: The validation loss delta after each growth event is minimal (~0.00003), demonstrating successful knowledge retention. The model continues to learn new capabilities without catastrophic forgetting.
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import GPT2LMHeadModel, AutoTokenizer
|
|
|
|
# Load model and tokenizer
|
|
model = GPT2LMHeadModel.from_pretrained("aicinema69/gpt2-growing")
|
|
tokenizer = AutoTokenizer.from_pretrained("aicinema69/gpt2-growing")
|
|
|
|
# Generate text
|
|
input_text = "Once upon a time"
|
|
inputs = tokenizer(input_text, return_tensors="pt")
|
|
outputs = model.generate(**inputs, max_new_tokens=50)
|
|
print(tokenizer.decode(outputs[0]))
|
|
```
|
|
|
|
## Limitations
|
|
|
|
- Growth events may cause temporary performance dips that recover with continued training
|
|
- Requires sufficient training data to benefit from additional parameters
|
|
- More parameters = higher memory and compute requirements
|
|
|
|
## License
|
|
|
|
This model is based on GPT-2 which has the [OpenAI GPT-2 License](https://github.com/openai/gpt-2/blob/master/LICENSE).
|
|
|
|
## Citation
|
|
|
|
If you use this model in your research, please cite:
|
|
|
|
```bibtex
|
|
@misc{growing_llm,
|
|
author = {Satyam Singh},
|
|
title = {Growing LLM: Dynamic Model Growth for Continual Learning},
|
|
year = {2026},
|
|
publisher = {HuggingFace},
|
|
howpublished = {\url{https://huggingface.co/aicinema69/gpt2-growing}}
|
|
}
|
|
```
|
|
|
|
## Contact
|
|
|
|
For questions or issues, please open a GitHub issue or contact the model author. |