68 lines
1.9 KiB
Markdown
68 lines
1.9 KiB
Markdown
---
|
|
license: llama2
|
|
---
|
|
|
|
## Introducing GenZ Infinite
|
|
|
|
The model is a finetuned version of Genz-13B-v2 with a context size of 16K. The model architecture is updated to have lamda attention from the LM-Infinite paper which gives the model capability of 120K+ sequence length without affecting the preplexity
|
|
|
|
## Generate responses
|
|
|
|
Use the generate.py file from the [github repo](https://github.com/BudEcosystem/genz-infinite)
|
|
|
|
```
|
|
python generate.py --base_model budecosystem/genz-13b-infinite
|
|
|
|
```
|
|
|
|
You can integrate the model in your code my loading convert_llama_model function.
|
|
|
|
```python
|
|
import torch
|
|
from transformers import GenerationConfig, AutoModelForCausalLM, AutoTokenizer
|
|
from model.llama import convert_llama_model
|
|
|
|
local_branch = 2048
|
|
global_branch = 10
|
|
limit_distance = 2048
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
"budecosystem/genz-13b-infinite",
|
|
torch_dtype=torch.float16,
|
|
device_map="auto",
|
|
)
|
|
model = convert_llama_model(model, local_branch, global_branch)
|
|
|
|
```
|
|
|
|
## Evaluation
|
|
|
|
|
|
| Task | 4096 | 5120 | 8192 | 16384 |
|
|
| :----:|:---------:| :--------:| :--------:| :--------:|
|
|
|Passkey retreival | 100 | 75 | 48 | 30 |
|
|
|
|
|
|
## Training details
|
|
|
|
The model is trained of 4 A100 80GB for approximately 55hrs.
|
|
|
|
| Hyperparameters | Value |
|
|
| :----------------------------| :-----: |
|
|
| per_device_train_batch_size | 1 |
|
|
| gradient_accumulation_steps | 1 |
|
|
| epoch | 3 |
|
|
| steps | 8550 |
|
|
| learning_rate | 2e-4 |
|
|
| lr schedular type | cosine |
|
|
| warmup steps | 1000 |
|
|
| optimizer | adamw |
|
|
| fp16 | True |
|
|
| GPU | 4 A100 80GB |
|
|
|
|
|
|
### Acknowledgments
|
|
|
|
We'd like to thank the open-source community and the researchers whose foundational work laid the path to this model. Special shoutout to the authors of [LM-Infinite paper](https://arxiv.org/abs/2308.16137) and the [GitHub repo](https://github.com/Glaciohound/LM-Infinite)
|
|
|