VAGOsolutions/SauerkrautLM-Phi-3-medium
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: VAGOsolutions/SauerkrautLM-Phi-3-medium
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-22 16:12:12 +08:00
commit ffaf0ff830
14 changed files with 93799 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

92
README.md Normal file
View File

@@ -0,0 +1,92 @@
---
license: mit
language:
- de
- en
---
![SauerkrautLM-Phi-3-medium](https://vago-solutions.ai/wp-content/uploads/2024/06/SauerkrautLM-phi3-medium.png "SauerkrautLM-Phi-3-medium")
## VAGO solutions SauerkrautLM-Phi-3-medium
Introducing **SauerkrautLM-Phi-3-medium** – our Sauerkraut version of the powerful [unsloth/Phi-3-medium-4k-instruct](https://huggingface.co/unsloth/Phi-3-medium-4k-instruct)!
- Aligned with DPO using [**Spectrum**](https://github.com/cognitivecomputations/spectrum) QLoRA (by Eric Hartford, Lucas Atkins, Fernando Fernandes Neto and David Golchinfar) **targeting 50% of the layers.**
# Table of Contents
1. [Overview of all SauerkrautLM-Phi-3-medium](#all-SauerkrautLM-Phi-3-medium)
2. [Model Details](#model-details)
- [Training procedure](#training-procedure)
3. [Evaluation](#evaluation)
5. [Disclaimer](#disclaimer)
6. [Contact](#contact)
7. [Collaborations](#collaborations)
8. [Acknowledgement](#acknowledgement)
## All SauerkrautLM-Phi-3-medium
| Model | HF | EXL2 | GGUF | AWQ |
|-------|-------|-------|-------|-------|
| SauerkrautLM-Phi-3-medium | [Link](https://huggingface.co/VAGOsolutions/SauerkrautLM-Phi-3-medium) | coming soon | coming soon | coming soon |
## Model Details
**SauerkrautLM-Phi-3-medium**
- **Model Type:** SauerkrautLM-Phi-3-medium is a finetuned Model based on [unsloth/Phi-3-medium-4k-instruct](https://huggingface.co/unsloth/Phi-3-medium-4k-instruct)
- **Language(s):** German, English
- **License:** MIT
- **Contact:** [VAGO solutions](https://vago-solutions.ai)
### Training procedure:
- We trained this model with [**Spectrum**](https://github.com/cognitivecomputations/spectrum) QLoRA DPO Fine-Tuning for 1 epoch with 70k samples targeting 50% of the layers with a high Learningrate of 5e-04.
This relatively high learning rate was feasible due to the selective targeting of layers; had we applied this rate to all layers, the gradients would have exploded.
**Fine-Tuning Details**
Epochs: 1
Data Size: 70,000 samples
Targeted Layers: 50%
Learning Rate: 5e-04
Warm-up Ratio: 0.03
The strategy of targeting only half of the layers also enabled us to use a very low warm-up ratio of 0.03, contributing to the overall stability of the fine-tuning process.
**Results**
This fine-tuning approach resulted in a noticeable improvement in the model's reasoning capabilities.
The model's performance was evaluated using a variety of benchmark suites, including the newly introduced [MixEval](https://mixeval.github.io/), which shows a 96% correlation with Chatbot Arena.
MixEval uses regular updated test data, providing a reliable benchmark for model performance.
## Evaluation
**Open LLM Leaderboard and German RAG:**
![SauerkrautLM-Phi-3-medium_h6_ger_rag](https://vago-solutions.ai/wp-content/uploads/2024/06/HF6-RAG.png "SauerkrautLM-Phi-3-medium_h6_ger_rag")
**Mix Eval Hard**
![SauerkrautLM-Phi-3-medium_mixeval_hard](https://vago-solutions.ai/wp-content/uploads/2024/06/MixedEval.png "SauerkrautLM-Phi-3-medium_mixeval_hard")
**GPT4ALL**
![SauerkrautLM-Phi-3-medium_gpt4all](https://vago-solutions.ai/wp-content/uploads/2024/06/GPT4ALL.png "SauerkrautLM-Phi-3-medium_gpt4all")
**AGIEval**
![SauerkrautLM-Phi-3-medium_agieval](https://vago-solutions.ai/wp-content/uploads/2024/06/AgiEval.png "SauerkrautLM-Phi-3-medium_agieval")
## Disclaimer
We must inform users that despite our best efforts in data cleansing, the possibility of uncensored content slipping through cannot be entirely ruled out.
However, we cannot guarantee consistently appropriate behavior. Therefore, if you encounter any issues or come across inappropriate content, we kindly request that you inform us through the contact information provided.
Additionally, it is essential to understand that the licensing of these models does not constitute legal advice. We are not held responsible for the actions of third parties who utilize our models.
## Contact
If you are interested in customized LLMs for business applications, please get in contact with us via our websites. We are also grateful for your feedback and suggestions.
## Collaborations
We are also keenly seeking support and investment for our startup, VAGO solutions where we continuously advance the development of robust language models designed to address a diverse range of purposes and requirements. If the prospect of collaboratively navigating future challenges excites you, we warmly invite you to reach out to us at [VAGO solutions](https://vago-solutions.ai/#Kontakt)
## Acknowledgement
Many thanks to [unsloth](https://huggingface.co/unsloth/) and [Microsoft](https://huggingface.co/microsoft) for providing such valuable model to the Open-Source community.

27
config.json Normal file
View File

@@ -0,0 +1,27 @@
{
"_name_or_path": "SauerkrautLM-Phi-3-medium",
"architectures": [
"MistralForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 1,
"eos_token_id": 32000,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 17920,
"max_position_embeddings": 4096,
"model_type": "mistral",
"num_attention_heads": 40,
"num_hidden_layers": 40,
"num_key_value_heads": 10,
"pad_token_id": 32009,
"rms_norm_eps": 1e-05,
"rope_theta": 10000.0,
"sliding_window": 2048,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.41.2",
"use_cache": true,
"vocab_size": 32064
}

3
model-00001-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2f3299b8496e4f20d976f8c75f7cf92d6911d75a1bacbcee7f60d3d2c22c30d7
size 4929757784

3
model-00002-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a8a076a6e10c8b0d549613dd33cc2f3e4543a84a11120c0270ca3d668cc191d3
size 4954672440

3
model-00003-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d5ba632e55386033db7cc00288d7e54bcb874e677c5e28370475f2e351050354
size 4967790112

3
model-00004-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d2b1fa3b36d7cfd644898790730e595461a76962f91d6e3d5439e88b4b94332c
size 4889147016

3
model-00005-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7903e2ffef9570712bb7d36edf86c4603b40590ffe44e3847a6e221ac05c23d4
size 4954672440

3
model-00006-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:167c2343adbb7017402a1c5d8714a3e3f91536d01a6d022c7e8ead02e93c4019
size 3224478688

1
model.safetensors.index.json Normal file
View File

File diff suppressed because one or more lines are too long

30
special_tokens_map.json Normal file
View File

@@ -0,0 +1,30 @@
{
"bos_token": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|placeholder6|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false
},
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

93463
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
tokenizer.model (Stored with Git LFS) Normal file
View File

Binary file not shown.

130
tokenizer_config.json Normal file
View File

@@ -0,0 +1,130 @@
{
"add_bos_token": false,
"add_eos_token": false,
"added_tokens_decoder": {
"0": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": false
},
"32000": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32001": {
"content": "<|assistant|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": true
},
"32002": {
"content": "<|placeholder1|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": true
},
"32003": {
"content": "<|placeholder2|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": true
},
"32004": {
"content": "<|placeholder3|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": true
},
"32005": {
"content": "<|placeholder4|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": true
},
"32006": {
"content": "<|system|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": true
},
"32007": {
"content": "<|end|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": true
},
"32008": {
"content": "<|placeholder5|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": true
},
"32009": {
"content": "<|placeholder6|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": true
},
"32010": {
"content": "<|user|>",
"lstrip": false,
"normalized": false,
"rstrip": true,
"single_word": false,
"special": true
}
},
"bos_token": "<s>",
"chat_template": "{% for message in messages %}{% if (message['role'] == 'user') %}{{'<|user|>' + '\n' + message['content'] + '<|end|>' + '\n' + '<|assistant|>' + '\n'}}{% elif (message['role'] == 'assistant') %}{{message['content'] + '<|end|>' + '\n'}}{% endif %}{% endfor %}",
"clean_up_tokenization_spaces": false,
"eos_token": "<|endoftext|>",
"legacy": true,
"model_max_length": 4096,
"pad_token": "<|placeholder6|>",
"padding_side": "left",
"sp_model_kwargs": {},
"tokenizer_class": "LlamaTokenizer",
"unk_token": "<unk>",
"use_default_system_prompt": false
}