159 lines
3.9 KiB
Markdown
159 lines
3.9 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
license: apache-2.0
|
||
base_model: openai-community/gpt2-xl
|
||
tags:
|
||
- gpt2
|
||
- gpt2-xl
|
||
- instruction-tuned
|
||
- alpaca
|
||
- causal-lm
|
||
- text-generation
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
|
||
|
||
# BananaMind-1.0-Instruct
|
||
|
||
BananaMind-1.0-Instruct is a full-finetuned GPT-2 XL instruction model trained on Alpaca-style instruction data.
|
||
|
||
|
||
It is our first Scaling Up Plan Model.
|
||
We plan to do full pretraining in the future.
|
||
|
||
The model is based on `openai-community/gpt2-xl` and is intended for basic English instruction-following text generation.
|
||
|
||
|
||
It is our first usable chat model.
|
||
|
||
It includes a 1024 token context window.
|
||
|
||
📚 Training Data
|
||
|
||
For the Full Finetune, we used the Alpaca Cleaned Dataset for ~0.75 Epochs not full because it started plateauing.
|
||
|
||
|
||
## Evaluation
|
||
|
||
Evaluations were run with `lm-evaluation-harness` using the Hugging Face model backend in bfloat16.
|
||
|
||
### Benchmark Summary
|
||
|
||
| Benchmark | Setting | Metric | Score |
|
||
|---|---:|---|---:|
|
||
| ARC-Easy | 0-shot | acc | 57.62% |
|
||
| ARC-Easy | 0-shot | acc_norm | 50.38% |
|
||
| ARC-Challenge | 0-shot | acc | 27.13% |
|
||
| ARC-Challenge | 0-shot | acc_norm | 28.24% |
|
||
| HellaSwag | 0-shot | acc | 40.33% |
|
||
| HellaSwag | 0-shot | acc_norm | 50.68% |
|
||
| PIQA | 0-shot | acc | 71.00% |
|
||
| PIQA | 0-shot | acc_norm | 70.13% |
|
||
| Winogrande | 0-shot | acc | 58.25% |
|
||
| MMLU | 0-shot | acc | 25.54% |
|
||
|
||
## Prompt Format
|
||
|
||
Use this format:
|
||
|
||
Below is an instruction that describes a task. Write a response that appropriately completes the request.
|
||
|
||
### Instruction:
|
||
{instruction}
|
||
|
||
### Response:
|
||
|
||
## Example Usage
|
||
|
||
Install dependencies:
|
||
|
||
pip install -U transformers accelerate safetensors torch
|
||
|
||
Run inference:
|
||
|
||
import torch
|
||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
|
||
repo = "BananaMind/BananaMind-1.0-Instruct"
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(repo, use_fast=True)
|
||
tokenizer.pad_token = tokenizer.eos_token
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
repo,
|
||
dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
)
|
||
|
||
instruction = "Explain what photosynthesis is in simple terms."
|
||
|
||
prompt = f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
|
||
|
||
### Instruction:
|
||
{instruction}
|
||
|
||
### Response:
|
||
"""
|
||
|
||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||
|
||
with torch.no_grad():
|
||
output = model.generate(
|
||
**inputs,
|
||
max_new_tokens=140,
|
||
do_sample=False,
|
||
repetition_penalty=1.1,
|
||
pad_token_id=tokenizer.eos_token_id,
|
||
eos_token_id=tokenizer.eos_token_id,
|
||
)
|
||
|
||
text = tokenizer.decode(output[0], skip_special_tokens=True)
|
||
print(text.split("### Response:", 1)[-1].strip())
|
||
|
||
## Suggested Generation Settings
|
||
|
||
For stable answers:
|
||
|
||
- `do_sample=False`
|
||
- `repetition_penalty=1.1`
|
||
- `max_new_tokens=80` to `160`
|
||
|
||
For slightly more creative answers:
|
||
|
||
- `do_sample=True`
|
||
- `temperature=0.4`
|
||
- `top_p=0.85`
|
||
- `repetition_penalty=1.15`
|
||
- `max_new_tokens=80` to `160`
|
||
|
||
|
||
BananaMind-1.0-Instruct was trained as a full finetune rather than a LoRA adapter. The model was trained using an Alpaca-style instruction-response format.
|
||
|
||
Older training checkpoints may still be available under the `checkpoints/` folder.
|
||
|
||
|
||
## Samples
|
||
|
||
The sky can vary in color from blue to purple, depending on the time of day and the location. The sky can also be cloudy or clear, with stars visible through the clouds.
|
||
The number of r’s in the word "strawberry" is 3. (yes i dont know how it knows this correctly)
|
||
I am an AI assistant designed to assist users in various tasks and provide them with information, entertainment, and assistance.
|
||
1 + 1 is equal to 2.
|
||
Here's a simple Python script that will print "Hello World" to the console:
|
||
|
||
```python
|
||
#!/usr/bin/env python
|
||
|
||
import time
|
||
|
||
print("Hello World")
|
||
```
|
||
|
||
|
||
|
||
|
||
|
||
## License
|
||
|
||
Apache 2.0 |