---
language:
- en
license: apache-2.0
base_model: openai-community/gpt2-xl
tags:
- gpt2
- gpt2-xl
- instruction-tuned
- alpaca
- causal-lm
- text-generation
pipeline_tag: text-generation
---


![Banner](banner.png)
# BananaMind-1.0-Instruct

BananaMind-1.0-Instruct is a full-finetuned GPT-2 XL instruction model trained on Alpaca-style instruction data.


It is our first Scaling Up Plan Model.
We plan to do full pretraining in the future.

The model is based on `openai-community/gpt2-xl` and is intended for basic English instruction-following text generation.


It is our first usable chat model.

It includes a 1024 token context window.

📚 Training Data

For the Full Finetune, we used the Alpaca Cleaned Dataset for ~0.75 Epochs not full because it started plateauing.


## Evaluation

Evaluations were run with `lm-evaluation-harness` using the Hugging Face model backend in bfloat16.

### Benchmark Summary

| Benchmark | Setting | Metric | Score |
|---|---:|---|---:|
| ARC-Easy | 0-shot | acc | 57.62% |
| ARC-Easy | 0-shot | acc_norm | 50.38% |
| ARC-Challenge | 0-shot | acc | 27.13% |
| ARC-Challenge | 0-shot | acc_norm | 28.24% |
| HellaSwag | 0-shot | acc | 40.33% |
| HellaSwag | 0-shot | acc_norm | 50.68% |
| PIQA | 0-shot | acc | 71.00% |
| PIQA | 0-shot | acc_norm | 70.13% |
| Winogrande | 0-shot | acc | 58.25% |
| MMLU | 0-shot | acc | 25.54% |

## Prompt Format

Use this format:

    Below is an instruction that describes a task. Write a response that appropriately completes the request.

    ### Instruction:
    {instruction}

    ### Response:

## Example Usage

Install dependencies:

    pip install -U transformers accelerate safetensors torch

Run inference:

    import torch
    from transformers import AutoTokenizer, AutoModelForCausalLM

    repo = "BananaMind/BananaMind-1.0-Instruct"

    tokenizer = AutoTokenizer.from_pretrained(repo, use_fast=True)
    tokenizer.pad_token = tokenizer.eos_token

    model = AutoModelForCausalLM.from_pretrained(
        repo,
        dtype=torch.bfloat16,
        device_map="auto",
    )

    instruction = "Explain what photosynthesis is in simple terms."

    prompt = f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.

    ### Instruction:
    {instruction}

    ### Response:
    """

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=140,
            do_sample=False,
            repetition_penalty=1.1,
            pad_token_id=tokenizer.eos_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )

    text = tokenizer.decode(output[0], skip_special_tokens=True)
    print(text.split("### Response:", 1)[-1].strip())

## Suggested Generation Settings

For stable answers:

- `do_sample=False`
- `repetition_penalty=1.1`
- `max_new_tokens=80` to `160`

For slightly more creative answers:

- `do_sample=True`
- `temperature=0.4`
- `top_p=0.85`
- `repetition_penalty=1.15`
- `max_new_tokens=80` to `160`


BananaMind-1.0-Instruct was trained as a full finetune rather than a LoRA adapter. The model was trained using an Alpaca-style instruction-response format.

Older training checkpoints may still be available under the `checkpoints/` folder.


## Samples

The sky can vary in color from blue to purple, depending on the time of day and the location. The sky can also be cloudy or clear, with stars visible through the clouds.
The number of r’s in the word "strawberry" is 3. (yes i dont know how it knows this correctly)
I am an AI assistant designed to assist users in various tasks and provide them with information, entertainment, and assistance.
1 + 1 is equal to 2.
Here's a simple Python script that will print "Hello World" to the console:

```python
#!/usr/bin/env python

import time

print("Hello World")
```





## License

Apache 2.0