初始化项目，由ModelHub XC社区提供模型
Model: migtissera/Tess-3-7B-SFT Source: Original Platform
This commit is contained in:
ModelHub XC
203
README.md
Normal file
203
README.md
Normal file
@@ -0,0 +1,203 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- generated_from_trainer
|
||||
base_model: mistralai/Mistral-7B-v0.3
|
||||
model-index:
|
||||
- name: home/migel/tess-2.5-mistral-7B-phase-1
|
||||
results: []
|
||||
---
|
||||
|
||||
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
||||
should probably proofread and complete it, then remove this comment. -->
|
||||
|
||||
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
|
||||
<details><summary>See axolotl config</summary>
|
||||
|
||||
axolotl version: `0.4.1`
|
||||
```yaml
|
||||
base_model: mistralai/Mistral-7B-v0.3
|
||||
model_type: AutoModelForCausalLM
|
||||
tokenizer_type: AutoTokenizer
|
||||
tokenizer_use_fast: false
|
||||
|
||||
load_in_8bit: false
|
||||
load_in_4bit: false
|
||||
strict: false
|
||||
model_config:
|
||||
|
||||
datasets:
|
||||
- path: /home/migel/ai_datasets/tess-v1.5b-chatml.jsonl
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
- path: /home/migel/ai_datasets/Tess-3.0/Tess-3.0-multi_turn_chatml.jsonl
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
- path: /home/migel/ai_datasets/Tess-3.0/Tess-3.0-single_turn_chatml.jsonl
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
|
||||
chat_template: chatml
|
||||
|
||||
dataset_prepared_path: last_run_prepared_mistral
|
||||
val_set_size: 0.0
|
||||
output_dir: /home/migel/tess-2.5-mistral-7B-phase-1
|
||||
|
||||
resume_from_checkpoint: /home/migel/tess-2.5-mistral-7B-phase-1/checkpoint-440
|
||||
auto_resume_from_checkpoints: true
|
||||
|
||||
sequence_len: 16384
|
||||
sample_packing: true
|
||||
pad_to_sequence_len: true
|
||||
|
||||
gradient_accumulation_steps: 4
|
||||
micro_batch_size: 4
|
||||
num_epochs: 1
|
||||
logging_steps: 1
|
||||
optimizer: adamw_8bit
|
||||
lr_scheduler: constant
|
||||
learning_rate: 1e-6
|
||||
|
||||
wandb_project: mistral-7b
|
||||
wandb_watch:
|
||||
wandb_run_id:
|
||||
wandb_log_model:
|
||||
|
||||
train_on_inputs: false
|
||||
group_by_length: false
|
||||
bf16: auto
|
||||
fp16:
|
||||
tf32: false
|
||||
|
||||
gradient_checkpointing: true
|
||||
gradient_checkpointing_kwargs:
|
||||
use_reentrant: false
|
||||
early_stopping_patience:
|
||||
resume_from_checkpoint:
|
||||
local_rank:
|
||||
logging_steps: 1
|
||||
xformers_attention:
|
||||
flash_attention: true
|
||||
saves_per_epoch: 10
|
||||
evals_per_epoch: 10
|
||||
save_total_limit: 3
|
||||
save_steps:
|
||||
eval_sample_packing: false
|
||||
debug:
|
||||
deepspeed: /home/migel/axolotl/deepspeed_configs/zero3_bf16.json
|
||||
weight_decay: 0.0
|
||||
fsdp:
|
||||
fsdp_config:
|
||||
special_tokens:
|
||||
bos_token: "<|im_start|>"
|
||||
eos_token: "<|im_end|>"
|
||||
pad_token: "<|end_of_text|>"
|
||||
|
||||
|
||||
```
|
||||
|
||||
</details><br>
|
||||
|
||||
# Tess-3-7B-SFT
|
||||
|
||||
|
||||
|
||||
Tess-3-7B is a finetuned version of the Mistral-7B-v0.3 base model. This version is the first phase of the final Tess-3 model, and have been trained with supervised fine-tuning (SFT) on a curated dataset of ~500K samples. The total SFT dataset contains about 1B tokens.
|
||||
|
||||
This model has 32K context length.
|
||||
|
||||
|
||||
# Sample code to run inference
|
||||
|
||||
Note that this model uses ChatML prompt format.
|
||||
|
||||
```python
|
||||
import torch, json
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from stop_word import StopWordCriteria
|
||||
|
||||
model_path = "migtissera/Tess-3-7B-SFT"
|
||||
output_file_path = "/home/migel/conversations.jsonl"
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_path,
|
||||
torch_dtype=torch.float16,
|
||||
device_map="auto",
|
||||
load_in_4bit=False,
|
||||
trust_remote_code=True,
|
||||
)
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
|
||||
|
||||
terminators = [
|
||||
tokenizer.convert_tokens_to_ids("<|im_end|>")
|
||||
]
|
||||
|
||||
def generate_text(instruction):
|
||||
tokens = tokenizer.encode(instruction)
|
||||
tokens = torch.LongTensor(tokens).unsqueeze(0)
|
||||
tokens = tokens.to("cuda")
|
||||
|
||||
instance = {
|
||||
"input_ids": tokens,
|
||||
"top_p": 1.0,
|
||||
"temperature": 0.75,
|
||||
"generate_len": 1024,
|
||||
"top_k": 50,
|
||||
}
|
||||
|
||||
length = len(tokens[0])
|
||||
with torch.no_grad():
|
||||
rest = model.generate(
|
||||
input_ids=tokens,
|
||||
max_length=length + instance["generate_len"],
|
||||
use_cache=True,
|
||||
do_sample=True,
|
||||
top_p=instance["top_p"],
|
||||
temperature=instance["temperature"],
|
||||
top_k=instance["top_k"],
|
||||
num_return_sequences=1,
|
||||
pad_token_id=tokenizer.eos_token_id,
|
||||
eos_token_id=terminators,
|
||||
)
|
||||
output = rest[0][length:]
|
||||
string = tokenizer.decode(output, skip_special_tokens=True)
|
||||
return f"{string}"
|
||||
|
||||
conversation = f"""<|im_start|>system\nYou are Tesoro, a helful AI assitant. You always provide detailed answers without hesitation.<|im_end|>\n<|im_start|>user\n"""
|
||||
|
||||
while True:
|
||||
user_input = input("You: ")
|
||||
llm_prompt = f"{conversation}{user_input}<|im_end|>\n<|im_start|>assistant\n"
|
||||
answer = generate_text(llm_prompt)
|
||||
print(answer)
|
||||
conversation = f"{llm_prompt}{answer}\n"
|
||||
json_data = {"prompt": user_input, "answer": answer}
|
||||
|
||||
with open(output_file_path, "a") as output_file:
|
||||
output_file.write(json.dumps(json_data) + "\n")
|
||||
```
|
||||
|
||||
# Join My General AI Discord (NeuroLattice):
|
||||
https://discord.gg/Hz6GrwGFKD
|
||||
|
||||
# Limitations & Biases:
|
||||
|
||||
While this model aims for accuracy, it can occasionally produce inaccurate or misleading results.
|
||||
|
||||
Despite diligent efforts in refining the pretraining data, there remains a possibility for the generation of inappropriate, biased, or offensive content.
|
||||
|
||||
Exercise caution and cross-check information when necessary. This is an uncensored model.
|
||||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
|
||||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_migtissera__Tess-3-7B-SFT)
|
||||
|
||||
| Metric |Value|
|
||||
|-------------------|----:|
|
||||
|Avg. |17.10|
|
||||
|IFEval (0-Shot) |39.46|
|
||||
|BBH (3-Shot) |24.12|
|
||||
|MATH Lvl 5 (4-Shot)| 3.32|
|
||||
|GPQA (0-shot) | 2.80|
|
||||
|MuSR (0-shot) |10.28|
|
||||
|MMLU-PRO (5-shot) |22.60|
|
||||
|
||||
Reference in New Issue
Block a user