初始化项目，由ModelHub XC社区提供模型

Model: VAGOsolutions/SauerkrautLM-1.5b
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,136 @@
+---
+license: apache-2.0
+language:
+- de
+- en
+tags:
+- spectrum
+- continuous pretraining
+- sft
+- dpo
+---
+
+
+![SauerkrautLM-1.5b](https://vago-solutions.ai/wp-content/uploads/2024/06/SauerkrautLM-1.5b-pic.png "SauerkrautLM-1.5b")
+## VAGO solutions SauerkrautLM-1.5b
+
+**DEMO Model** - *to showcase the potential of resource-efficient Continuous Pre-Training of Large Language Models using **Spectrum CPT***
+
+Introducing **SauerkrautLM-1.5b** – our Sauerkraut version of the powerful [Qwen/Qwen2-1.5B](https://huggingface.co/Qwen/Qwen2-1.5B)!
+
+- Continuous Pretraining on German Data with [**Spectrum**](https://github.com/cognitivecomputations/spectrum) CPT (by Eric Hartford, Lucas Atkins, Fernando Fernandes Neto and David Golchinfar) **targeting 25% of the layers.**
+- Finetuned with SFT
+- Aligned with DPO
+
+# Table of Contents
+1. [Overview of all SauerkrautLM-1.5b](#all-SauerkrautLM-1.5b)
+2. [Model Details](#model-details)
+   - [Training procedure](#training-procedure)
+3. [Evaluation](#evaluation)
+5. [Disclaimer](#disclaimer)
+6. [Contact](#contact)
+7. [Collaborations](#collaborations)
+8. [Acknowledgement](#acknowledgement)
+
+
+## All SauerkrautLM-1.5b
+
+| Model | HF    | EXL2  | GGUF  | AWQ  |
+|-------|-------|-------|-------|-------|
+| SauerkrautLM-1.5b | [Link](https://huggingface.co/VAGOsolutions/SauerkrautLM-1.5b) | coming soon | [Link](https://huggingface.co/VAGOsolutions/SauerkrautLM-1.5b.GGUF) | coming soon |
+
+## Model Details
+**SauerkrautLM-1.5b**
+- **Model Type:** SauerkrautLM-1.5b is a finetuned Model based on [Qwen/Qwen2-1.5B](https://huggingface.co/Qwen/Qwen2-1.5B)
+- **Language(s):** German, English
+- **License:** Apache 2.0
+- **Contact:** [VAGO solutions](https://vago-solutions.ai)
+
+## Training Procedure
+
+This model is a demo intended to showcase the potential of resource-efficient training of large language models using Spectrum CPT. Here's a brief on the procedure:
+
+**Continuous Pre-training (CPT) on German Data**: 
+
+Utilizing Spectrum by Eric Hartford, Lucas Atkins, Fernando Fernandes Neto, and David Golchinfar, the model targeted 25% of its layers during training. This approach allowed significant resource savings:
+Spectrum with 25% layer targeting consumed 309.78GB at a batch size of 2048.
+Full Fine-tuning targeting 100% of layers used 633.55GB at the same batch size.
+Using Spectrum, we enhanced the German language capabilities of the Qwen2-1.5B model via CPT while achieving substantial resource savings. 
+Spectrum enabled faster training and cost reductions. By not targeting all layers for CPT, we managed to prevent substantial performance degradation in the model's primary language (English), thus markedly improving its German proficiency.
+
+The model was further trained with **6.1 billion German tokens**, costing $1152 GPU-Rent for CPT. 
+In the German Rag evaluation, it is on par with 8 billion parameter models and, with its 1.5 billion parameter size, is well-suited for mobile deployment on smartphones and tablets.
+
+Despite the large volume of German CPT data, the model competes well against the Qwen2-1.5B-Instruct model and performs significantly better in German.
+
+**Post-CPT Training**: 
+
+The model underwent 3 epochs of Supervised Fine-Tuning (SFT) with 700K samples.
+
+
+**Further Steps**: 
+
+The model was aligned with Direct Preference Optimization (DPO) using 70K samples.
+
+## Objective and Results
+
+The primary goal of this training was to demonstrate that with Spectrum CPT targeting 25% of the layers, even a relatively small model with 1.5 billion parameters can significantly enhance language capabilities while using a fraction of the resources of the classic CPT approach. 
+This method has an even more pronounced effect on larger models. It is feasible to teach a model a new language by training just a quarter of the available layers.
+
+The model has substantially improved German skills as demonstrated in RAG evaluations and numerous recognized benchmarks. In some English benchmarks, it even surpasses the Qwen2-1.5B-Instruct model.
+
+**Spectrum CPT can efficiently teach a new language to a large language model (LLM) while preserving the majority of its previously acquired knowledge.**
+
+Stay tuned for the next big models employing Spectrum CPT!
+
+**NOTE**
+
+For the demo, the performance of the model is sufficient.  
+For productive use, more German tokens can be trained on the SauerkrautLM-1.5b as required in order to teach the model even firmer German while only having a relative influence on the performance of the model (25% of the layers). 
+The SauerkrautLM-1.5b offers an excellent starting point for this.
+
+
+## Evaluation
+
+**VRAM usage Spectrum CPT vs. FFT CPT - with a batchsize of 2048**
+
+
+![SauerkrautLM-1.5b_vram](https://vago-solutions.ai/wp-content/uploads/2024/06/VRAM-Usage_new.png "SauerkrautLM-1.5b_vram")
+
+**Open LLM Leaderboard H6:**
+
+![SauerkrautLM-1.5b_h6](https://vago-solutions.ai/wp-content/uploads/2024/06/H6-Benchmarks.png "SauerkrautLM-1.5b_h6")
+
+
+**German H4**
+
+![SauerkrautLM-1.5b_h4](https://vago-solutions.ai/wp-content/uploads/2024/06/H4_ger_new.png "SauerkrautLM-1.5b_h4")
+
+
+**German RAG:**
+
+![SauerkrautLM-1.5b_ger_rag](https://vago-solutions.ai/wp-content/uploads/2024/06/ger_rag_eval.png "SauerkrautLM-1.5b_ger_rag")
+
+
+**GPT4ALL**
+
+![SauerkrautLM-1.5b_gpt4all](https://vago-solutions.ai/wp-content/uploads/2024/06/GPT4All-1.png "SauerkrautLM-1.5b_gpt4all")
+
+
+**AGIEval**
+
+![SauerkrautLM-1.5b_agieval](https://vago-solutions.ai/wp-content/uploads/2024/06/AGIEval-1.png "SauerkrautLM-1.5b_agieval")
+
+## Disclaimer
+We must inform users that despite our best efforts in data cleansing, the possibility of uncensored content slipping through cannot be entirely ruled out.
+However, we cannot guarantee consistently appropriate behavior. Therefore, if you encounter any issues or come across inappropriate content, we kindly request that you inform us through the contact information provided.
+Additionally, it is essential to understand that the licensing of these models does not constitute legal advice. We are not held responsible for the actions of third parties who utilize our models.
+ 
+## Contact
+If you are interested in customized LLMs for business applications, please get in contact with us via our website. We are also grateful for your feedback and suggestions.
+ 
+## Collaborations
+We are also keenly seeking support and investment for our startup, VAGO solutions where we continuously advance the development of robust language models designed to address a diverse range of purposes and requirements. If the prospect of collaboratively navigating future challenges excites you, we warmly invite you to reach out to us at [VAGO solutions](https://vago-solutions.de/#Kontakt)
+
+## Acknowledgement
+Many thanks to [Qwen](https://huggingface.co/Qwen) for providing such valuable model to the Open-Source community.

added_tokens.json

@@ -0,0 +1,5 @@
+{
+  "<|endoftext|>": 151643,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644
+}

config.json

@@ -0,0 +1,27 @@
+{
+  "_name_or_path": "VAGOsolutions/SauerkrautLM-1.5b",
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "eos_token_id": 151645,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 8960,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 2,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 1000000.0,
+  "sliding_window": 32768,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.41.2",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "transformers_version": "4.41.2"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3add272ddc4261233deaa09ed85bf096026fafb674bfef664b1a8076599db498
+size 3087467144

special_tokens_map.json

@@ -0,0 +1,20 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,44 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}