87 lines
3.3 KiB
Markdown
87 lines
3.3 KiB
Markdown
---
|
|
base_model:
|
|
- Vortex5/Astral-Noctra-12B
|
|
- Vortex5/Azure-Starlight-12B
|
|
- Vortex5/Crimson-Constellation-12B
|
|
- Vortex5/Red-Synthesis-12B
|
|
- Vortex5/Shining-Seraph-12B
|
|
- Vortex5/Starlit-Shadow-12B
|
|
- Vortex5/Vermilion-Sage-12B
|
|
- Vortex5/Scarlet-Seraph-12B
|
|
- Vortex5/Maroon-Sunset-12B
|
|
- Vortex5/Amber-Starlight-12B
|
|
language:
|
|
- en
|
|
library_name: transformers
|
|
license: apache-2.0
|
|
tags:
|
|
- karcher
|
|
- merge
|
|
- mergekit
|
|
- mistral
|
|
- nemo
|
|
widget:
|
|
- text: "Riemannian-Redshift-12B-v1"
|
|
output:
|
|
url: https://cdn-uploads.huggingface.co/production/uploads/68e840caa318194c44ec2a04/-5OsIfrWlUxoJZP4U83uq.png
|
|
---
|
|
|
|
> [!NOTE]
|
|
> <span style="color:red; font-weight:bold">⚠️ Note:</span> This model requires **Mistral Tekken** chat template.
|
|
>
|
|
|
|
# 🌌 Riemannian Redshift 12B v1
|
|
|
|
This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
|
|
|
|
|
|
|
|
## Merge Details
|
|
### Merge Method
|
|
|
|
This is an experimental `karcher` merge of several high quality [Vortex5](https://huggingface.co/Vortex5) models. I used `float32` precision and `max_iter: 1000` to ensure the best bits were chosen for the Riemannian center. This merge took 5 hours using [graph_v18](https://huggingface.co/spaces/Naphula/model_tools/blob/main/graph_v18.py) as an accelerant with 8GB VRAM.
|
|
|
|
This model was merged using the [Karcher Mean](https://en.wikipedia.org/wiki/Karcher_mean) merge method.
|
|
|
|
### Models Merged
|
|
|
|
The following models were included in the merge:
|
|
* [Vortex5/Maroon-Sunset-12B](https://huggingface.co/Vortex5/Maroon-Sunset-12B)
|
|
* [Vortex5/Azure-Starlight-12B](https://huggingface.co/Vortex5/Azure-Starlight-12B)
|
|
* [Vortex5/Scarlet-Seraph-12B](https://huggingface.co/Vortex5/Scarlet-Seraph-12B)
|
|
* [Vortex5/Amber-Starlight-12B](https://huggingface.co/Vortex5/Amber-Starlight-12B)
|
|
* [Vortex5/Shining-Seraph-12B](https://huggingface.co/Vortex5/Shining-Seraph-12B)
|
|
* [Vortex5/Red-Synthesis-12B](https://huggingface.co/Vortex5/Red-Synthesis-12B)
|
|
* [Vortex5/Starlit-Shadow-12B](https://huggingface.co/Vortex5/Starlit-Shadow-12B)
|
|
* [Vortex5/Crimson-Constellation-12B](https://huggingface.co/Vortex5/Crimson-Constellation-12B)
|
|
* [Vortex5/Vermilion-Sage-12B](https://huggingface.co/Vortex5/Vermilion-Sage-12B)
|
|
* [Vortex5/Astral-Noctra-12B](https://huggingface.co/Vortex5/Astral-Noctra-12B)
|
|
|
|
### Configuration
|
|
|
|
The following YAML configuration was used to produce this model:
|
|
|
|
```yaml
|
|
models:
|
|
- model: B:/12B/models--Vortex5--Astral-Noctra-12B
|
|
- model: B:/12B/models--Vortex5--Azure-Starlight-12B
|
|
- model: B:/12B/models--Vortex5--Crimson-Constellation-12B
|
|
- model: B:/12B/models--Vortex5--Red-Synthesis-12B
|
|
- model: B:/12B/models--Vortex5--Shining-Seraph-12B
|
|
- model: B:/12B/models--Vortex5--Starlit-Shadow-12B
|
|
- model: B:/12B/models--Vortex5--Vermilion-Sage-12B
|
|
- model: B:/12B/models--Vortex5--Scarlet-Seraph-12B
|
|
- model: B:/12B/models--Vortex5--Maroon-Sunset-12B
|
|
- model: B:/12B/models--Vortex5--Amber-Starlight-12B
|
|
merge_method: karcher
|
|
parameters:
|
|
max_iter: 1000
|
|
tol: 1.0e-9
|
|
dtype: float32
|
|
out_dtype: bfloat16
|
|
tokenizer:
|
|
source: union
|
|
chat_template: auto
|
|
name: 🌌 Riemannian-Redshift-12B-v1
|
|
```
|