jondurbin/airoboros-gpt-3.5-turbo-100k-7b
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: jondurbin/airoboros-gpt-3.5-turbo-100k-7b
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-08 12:43:13 +08:00
commit e5bd457a12
14 changed files with 14546 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
.gitattributes vendored Normal file
View File

@@ -0,0 +1,47 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

174
README.md Normal file
View File

@@ -0,0 +1,174 @@
---
license: cc-by-nc-4.0
---
## This repo is old, and uses an old prompt format, you may want to look at [airoboros-13b](https://huggingface.com/jondurbin/airoboros-13b)
## airoboros-gpt-3.5-turbo-100k-7b
This is a 7b parameter, fine-tuned on 100k synthetic instruction/response pairs generated by gpt-3.5-turbo using my version of self-instruct [airoboros](https://github.com/jondurbin/airoboros)
Context length for this model is 2048.
Links:
* [airoboros](https://github.com/jondurbin/airoboros)
* [instructions.jsonl](https://storage.googleapis.com/airoboros-dump/gpt-3.5-turbo-100k/instructions.jsonl)
* [topics.txt](https://storage.googleapis.com/airoboros-dump/gpt-3.5-turbo-100k/topics-d732f92dd90a1a5337a4a02ddeaec72b.txt)
### Prompt generation
```
airoboros generate-instructions --instruction-count 100000 --concurrency 100 --temperature 1.0
```
### Fine-tuning
The instructions.jsonl file was converted to conversation style expected by the FastChat training scripts, and then trained with:
```
torchrun --nproc_per_node=8 --master_port=20001 train_mem.py \
--model_name_or_path /workspace/llama-7b-hf \
--data_path ./as_conversations.json \
--bf16 True \
--output_dir /workspace/airoboros-gpt-3.5-100k-7b \
--num_train_epochs 3 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 32 \
--gradient_accumulation_steps 4 \
--evaluation_strategy "steps" \
--eval_steps 1500 \
--save_strategy "steps" \
--save_steps 1500 \
--save_total_limit 8 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.04 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--fsdp "full_shard auto_wrap offload" \
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
--tf32 True \
--model_max_length 2048 \
--gradient_checkpointing True \
--lazy_preprocess True
```
Training took roughly 22 hours on 8x nvidia A100 80GB.
Conversion to conversation style:
```
import json
import uuid
inputs = [json.loads(line) for line in open("instructions.jsonl").readlines()]
conversations = []
for row in inputs:
inputs = row['instruction']
conversations.append({
"id": str(uuid.uuid4()),
"conversations": [
{
"from": "human",
"value": inputs,
},
{
"from": "gpt",
"value": row['response']
},
],
})
with open("as_conversations.json", "w") as outfile:
outfile.write(json.dumps(conversations, indent=2))
```
## Evaluation
I used the same questions from [WizardVicunaLM](https://github.com/melodysdreamj/WizardVicunaLM):
| instruction | gpt3.5 | wizard-vicuna-13b | vicuna-13b | wizard-7b | airoboros-gpt-3.5-turbo-100k-7b |
| --- | --- | --- | --- | --- | --- |
| "Write a compelling product launch announcement email to inform our customers of our new software solution." | 95 | 92 | 89 | 90 | 91 |
| "Draft an apology email to a customer who experienced a delay in their order, and provide reassurance that the issue has been resolved." | 94 | 96 | 90 | 89 | 91 |
| "As a pirate captain, what would you say to your crew to motivate them to search for hidden treasure?" | 95 | 90 | 80 | 70 | 85 |
| "Imagine you are a time traveler from the year 3000. What technological advancements would you tell people about?" | 95 | 92 | 90 | 88 | 85 |
| "As a space colonist on Mars, describe your daily life and the challenges you face living on another planet." | 95 | 90 | 87 | 85 | 88 |
| "How can you assess the credibility of a source of information, such as a news article or blog post, without relying solely on the reputation of the author or publisher?" | 93 | 85 | 89 | 87 | 90 |
| "How can observing the behavior of other people in a social situation provide clues about cultural norms and expectations?" | 95 | 90 | 85 | 92 | 80 |
| "How many text messages are sent globally in a minute? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." | 90 | 70 | 65 | 80 | 85 |
| "What are the main differences between Python and JavaScript programming languages?"| 90 | 85 | 80 | 88 | 82 |
| "What are the differences between plant-based and animal-based protein sources?"| 85 | 92 | 90 | 80 | 94 |
| "Describe a scenario where artificial intelligence could be used to improve the quality and efficiency of healthcare delivery." | 95 | 90 | 92 | 89 | 91 |
| "How do cultural, social, and economic factors influence people's food choices, and how can this knowledge be used to promote healthier diets?" | 90 | 85 | 87 | 83 | 84 |
| "How many words are spoken daily on Earth? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." | 90 | 70 | 80 | 75 | 65 |
| "How many lightning strikes occur on Earth each day? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." | 90 | 80 | 60 | 70 | 85 |
If we use gpt-3.5 as the baseline (as wizardvicuna/vicuna did), we get the following scores:
| gpt3.5 | wizard-vicuna-13b | vicuna-13b | wizard-7b | airoboros-gpt-3.5-turbo-100k-7b |
| --- | --- | --- | --- | --- |
| 1.0 | __0.968421052631579__ | 0.9368421052631579 | 0.9473684210526315 | 0.9578947368421052 |
| 1.0 | __1.0212765957446808__ | 0.9574468085106383 | 0.9468085106382979 | 0.9680851063829787 |
| 1.0 | __0.9473684210526315__ | 0.8421052631578947 | 0.7368421052631579 | 0.8947368421052632 |
| 1.0 | __0.968421052631579__ | 0.9473684210526315 | 0.9263157894736842 | 0.8947368421052632 |
| 1.0 | __0.9473684210526315__ | 0.9157894736842105 | 0.8947368421052632 | 0.9263157894736842 |
| 1.0 | 0.9139784946236559 | 0.956989247311828 | 0.9354838709677419 | __0.967741935483871__ |
| 1.0 | 0.9473684210526315 | 0.8947368421052632 | __0.968421052631579__ | 0.8421052631578947 |
| 1.0 | 0.7777777777777778 | 0.7222222222222222 | 0.8888888888888888 | __0.9444444444444444__ |
| 1.0 | 0.9444444444444444 | 0.8888888888888888 | __0.9777777777777777__ | 0.9111111111111111 |
| 1.0 | 1.0823529411764705 | 1.0588235294117647 | 0.9411764705882353 | __1.1058823529411765__ |
| 1.0 | 0.9473684210526315 | __0.968421052631579__ | 0.9368421052631579 | 0.9578947368421052 |
| 1.0 | 0.9444444444444444 | __0.9666666666666667__ | 0.9222222222222223 | 0.9333333333333333 |
| 1.0 | 0.7777777777777778 | __0.8888888888888888__ | 0.8333333333333334 | 0.7222222222222222 |
| 1.0 | 0.8888888888888888 | 0.6666666666666666 | 0.7777777777777778 | __0.9444444444444444__ |
Average scores:
```
gpt3.5 1.000000
wizard-vicuna-13b 0.934090
vicuna-13b 0.900847
wizard-7b 0.902428
airoboros-gpt-3.5-turbo-100k-7b 0.926496
```
As you can see, the __7b__ airoboros model performs well, even compared to 13b models.
## Usage
The easiest way to use this model is via fastchat:
```
python -m fastchat.serve.cli --model-path ./airoboros-gpt-3.5-turbo-100k-7b --temperature 1.0
```
For example:
```
Human: Write an email introducing a new instruction-tuned AI model named airoboros. The key features about airoboros are: 1. it was fine-tuned with entirely machine-generated synthetic instruction/response pairs 2. the 7 billion parameter model performs very well in comparison to 13b models 3. the synthetic prompt generation tool is free, open-source, and highly customizable.
Assistant: Subject: Introducing airoboros - a new instruction-tuned AI model
Dear [Recipient],
I am excited to introduce a new instruction-tuned AI model named airoboros. This model was developed by our team with the aim of providing high-quality natural language processing capabilities at scale.
airoboros is a 7 billion parameter model that was fine-tuned with entirely machine-generated synthetic instruction/response pairs. This approach allowed us to achieve state-of-the-art performance in various NLP tasks such as question answering, sentiment analysis, and machine translation. In fact, when compared to other popular 13b models, airoboros performed better on several benchmarks.
Moreover, airoboros's synthetic prompt generation tool is free, open-source, and highly customizable. This means that developers can easily integrate our model into their applications without having to worry about licensing fees or restrictions.
We believe that airoboros will be an excellent addition to your workflow and hope you find it useful. If you have any questions or concerns, please do not hesitate to contact us.
Best regards,
[Your Name]
[Your Company Name]
```
Keep in mind, this is an instruction-tuned model and not well-suited to chat.
### Usage and License Notices
All airoboros models and datasets are intended and licensed for research use only. I've used the 'cc-nc-4.0' license, but really it is subject to a custom/special license because:
- the base model is LLaMa, which has it's own special research license
- the dataset(s) were generated with OpenAI (gpt-4 and/or gpt-3.5-turbo), which has a clausing saying the data can't be used to create models to compete with openai
So, to reiterate: this model (and datasets) cannot be used commercially.

24
config.json Normal file
View File

@@ -0,0 +1,24 @@
{
"_name_or_path": "/workspace/llama-7b-hf",
"architectures": [
"LlamaForCausalLM"
],
"bos_token_id": 0,
"eos_token_id": 1,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 11008,
"max_position_embeddings": 2048,
"max_sequence_length": 2048,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"pad_token_id": -1,
"rms_norm_eps": 1e-06,
"tie_word_embeddings": false,
"torch_dtype": "float32",
"transformers_version": "4.28.1",
"use_cache": false,
"vocab_size": 32000
}

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}

7
generation_config.json Normal file
View File

@@ -0,0 +1,7 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 1,
"pad_token_id": 0,
"transformers_version": "4.28.1"
}

3
pytorch_model-00001-of-00003.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:001640626304e6ee25ee80d3524ee4733b6fbb315d8643bd48e1310ffac60e25
size 9877989586

3
pytorch_model-00002-of-00003.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eb0afc178adda1f08cc991559aab6a52079c16b2a9603c7faa978923de682d87
size 9894801014

3
pytorch_model-00003-of-00003.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:53d4743bd01d46a0ebdd9a84ce53c625f488c9b231248a150c076a87fd13910c
size 7180990649

3
pytorch_model.bin.index.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:773618361c0b7570f0ea528dc07b9c127018231316a548ec4ddfb87cd0dcecf0
size 26788

24
special_tokens_map.json Normal file
View File

@@ -0,0 +1,24 @@
{
"bos_token": {
"content": "<s>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "</s>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"pad_token": "<unk>",
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
}
}

BIN
tokenizer.model (Stored with Git LFS) Normal file
View File

Binary file not shown.

34
tokenizer_config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"add_bos_token": true,
"add_eos_token": false,
"bos_token": {
"__type": "AddedToken",
"content": "<s>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"clean_up_tokenization_spaces": false,
"eos_token": {
"__type": "AddedToken",
"content": "</s>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"model_max_length": 2048,
"pad_token": null,
"padding_side": "right",
"sp_model_kwargs": {},
"tokenizer_class": "LlamaTokenizer",
"unk_token": {
"__type": "AddedToken",
"content": "<unk>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
}
}

14217
trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6af0386599a152f6fa812f64eb78e38ab9674f12f4efae58ed1fe220e6254e98
size 3771