42 lines
1.1 KiB
Markdown
42 lines
1.1 KiB
Markdown
---
|
|
license: mit
|
|
base_model: openai-community/gpt2
|
|
datasets:
|
|
- OpenAssistant/oasst1
|
|
language:
|
|
- en
|
|
pipeline_tag: text-generation
|
|
library_name: transformers
|
|
tags:
|
|
- gpt2
|
|
- gpt-2
|
|
- chatbot
|
|
- text-generation-inference
|
|
---
|
|
|
|
# GPT-2 SFT on OASST1
|
|
|
|
This model is a fine-tuned version of [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) on the [OpenAssistant/oasst1](https://huggingface.co/datasets/OpenAssistant/oasst1) dataset.
|
|
|
|
## Training Procedure
|
|
|
|
- **Base Model:** GPT-2 Small (124M parameters)
|
|
- **Dataset:** OpenAssistant OASST1 (English only)
|
|
- **Format:** `user: ... assistant: <s> ... </s>`
|
|
- **Epochs:** 15
|
|
- **Learning Rate:** 5e-5
|
|
- **Batch Size:** 4 (effective 16 with gradient accumulation)
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
|
model_name = "gitcoreai/gpt-2-small-sft"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
|
model = AutoModelForCausalLM.from_pretrained(model_name)
|
|
|
|
prompt = "user: hello, how are you?\nassistant: <s>"
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
|
outputs = model.generate(**inputs, max_new_tokens=60)
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |