204 lines
5.7 KiB
Markdown
204 lines
5.7 KiB
Markdown
---
|
|
license: apache-2.0
|
|
tags:
|
|
- generated_from_trainer
|
|
base_model: mistralai/Mistral-7B-v0.3
|
|
model-index:
|
|
- name: home/migel/tess-2.5-mistral-7B-phase-1
|
|
results: []
|
|
---
|
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
|
should probably proofread and complete it, then remove this comment. -->
|
|
|
|
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
|
|
<details><summary>See axolotl config</summary>
|
|
|
|
axolotl version: `0.4.1`
|
|
```yaml
|
|
base_model: mistralai/Mistral-7B-v0.3
|
|
model_type: AutoModelForCausalLM
|
|
tokenizer_type: AutoTokenizer
|
|
tokenizer_use_fast: false
|
|
|
|
load_in_8bit: false
|
|
load_in_4bit: false
|
|
strict: false
|
|
model_config:
|
|
|
|
datasets:
|
|
- path: /home/migel/ai_datasets/tess-v1.5b-chatml.jsonl
|
|
type: sharegpt
|
|
conversation: chatml
|
|
- path: /home/migel/ai_datasets/Tess-3.0/Tess-3.0-multi_turn_chatml.jsonl
|
|
type: sharegpt
|
|
conversation: chatml
|
|
- path: /home/migel/ai_datasets/Tess-3.0/Tess-3.0-single_turn_chatml.jsonl
|
|
type: sharegpt
|
|
conversation: chatml
|
|
|
|
chat_template: chatml
|
|
|
|
dataset_prepared_path: last_run_prepared_mistral
|
|
val_set_size: 0.0
|
|
output_dir: /home/migel/tess-2.5-mistral-7B-phase-1
|
|
|
|
resume_from_checkpoint: /home/migel/tess-2.5-mistral-7B-phase-1/checkpoint-440
|
|
auto_resume_from_checkpoints: true
|
|
|
|
sequence_len: 16384
|
|
sample_packing: true
|
|
pad_to_sequence_len: true
|
|
|
|
gradient_accumulation_steps: 4
|
|
micro_batch_size: 4
|
|
num_epochs: 1
|
|
logging_steps: 1
|
|
optimizer: adamw_8bit
|
|
lr_scheduler: constant
|
|
learning_rate: 1e-6
|
|
|
|
wandb_project: mistral-7b
|
|
wandb_watch:
|
|
wandb_run_id:
|
|
wandb_log_model:
|
|
|
|
train_on_inputs: false
|
|
group_by_length: false
|
|
bf16: auto
|
|
fp16:
|
|
tf32: false
|
|
|
|
gradient_checkpointing: true
|
|
gradient_checkpointing_kwargs:
|
|
use_reentrant: false
|
|
early_stopping_patience:
|
|
resume_from_checkpoint:
|
|
local_rank:
|
|
logging_steps: 1
|
|
xformers_attention:
|
|
flash_attention: true
|
|
saves_per_epoch: 10
|
|
evals_per_epoch: 10
|
|
save_total_limit: 3
|
|
save_steps:
|
|
eval_sample_packing: false
|
|
debug:
|
|
deepspeed: /home/migel/axolotl/deepspeed_configs/zero3_bf16.json
|
|
weight_decay: 0.0
|
|
fsdp:
|
|
fsdp_config:
|
|
special_tokens:
|
|
bos_token: "<|im_start|>"
|
|
eos_token: "<|im_end|>"
|
|
pad_token: "<|end_of_text|>"
|
|
|
|
|
|
```
|
|
|
|
</details><br>
|
|
|
|
# Tess-3-7B-SFT
|
|
|
|
|
|
|
|
Tess-3-7B is a finetuned version of the Mistral-7B-v0.3 base model. This version is the first phase of the final Tess-3 model, and have been trained with supervised fine-tuning (SFT) on a curated dataset of ~500K samples. The total SFT dataset contains about 1B tokens.
|
|
|
|
This model has 32K context length.
|
|
|
|
|
|
# Sample code to run inference
|
|
|
|
Note that this model uses ChatML prompt format.
|
|
|
|
```python
|
|
import torch, json
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
from stop_word import StopWordCriteria
|
|
|
|
model_path = "migtissera/Tess-3-7B-SFT"
|
|
output_file_path = "/home/migel/conversations.jsonl"
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_path,
|
|
torch_dtype=torch.float16,
|
|
device_map="auto",
|
|
load_in_4bit=False,
|
|
trust_remote_code=True,
|
|
)
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
|
|
|
|
terminators = [
|
|
tokenizer.convert_tokens_to_ids("<|im_end|>")
|
|
]
|
|
|
|
def generate_text(instruction):
|
|
tokens = tokenizer.encode(instruction)
|
|
tokens = torch.LongTensor(tokens).unsqueeze(0)
|
|
tokens = tokens.to("cuda")
|
|
|
|
instance = {
|
|
"input_ids": tokens,
|
|
"top_p": 1.0,
|
|
"temperature": 0.75,
|
|
"generate_len": 1024,
|
|
"top_k": 50,
|
|
}
|
|
|
|
length = len(tokens[0])
|
|
with torch.no_grad():
|
|
rest = model.generate(
|
|
input_ids=tokens,
|
|
max_length=length + instance["generate_len"],
|
|
use_cache=True,
|
|
do_sample=True,
|
|
top_p=instance["top_p"],
|
|
temperature=instance["temperature"],
|
|
top_k=instance["top_k"],
|
|
num_return_sequences=1,
|
|
pad_token_id=tokenizer.eos_token_id,
|
|
eos_token_id=terminators,
|
|
)
|
|
output = rest[0][length:]
|
|
string = tokenizer.decode(output, skip_special_tokens=True)
|
|
return f"{string}"
|
|
|
|
conversation = f"""<|im_start|>system\nYou are Tesoro, a helful AI assitant. You always provide detailed answers without hesitation.<|im_end|>\n<|im_start|>user\n"""
|
|
|
|
while True:
|
|
user_input = input("You: ")
|
|
llm_prompt = f"{conversation}{user_input}<|im_end|>\n<|im_start|>assistant\n"
|
|
answer = generate_text(llm_prompt)
|
|
print(answer)
|
|
conversation = f"{llm_prompt}{answer}\n"
|
|
json_data = {"prompt": user_input, "answer": answer}
|
|
|
|
with open(output_file_path, "a") as output_file:
|
|
output_file.write(json.dumps(json_data) + "\n")
|
|
```
|
|
|
|
# Join My General AI Discord (NeuroLattice):
|
|
https://discord.gg/Hz6GrwGFKD
|
|
|
|
# Limitations & Biases:
|
|
|
|
While this model aims for accuracy, it can occasionally produce inaccurate or misleading results.
|
|
|
|
Despite diligent efforts in refining the pretraining data, there remains a possibility for the generation of inappropriate, biased, or offensive content.
|
|
|
|
Exercise caution and cross-check information when necessary. This is an uncensored model.
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_migtissera__Tess-3-7B-SFT)
|
|
|
|
| Metric |Value|
|
|
|-------------------|----:|
|
|
|Avg. |17.10|
|
|
|IFEval (0-Shot) |39.46|
|
|
|BBH (3-Shot) |24.12|
|
|
|MATH Lvl 5 (4-Shot)| 3.32|
|
|
|GPQA (0-shot) | 2.80|
|
|
|MuSR (0-shot) |10.28|
|
|
|MMLU-PRO (5-shot) |22.60|
|
|
|