160 lines
3.6 KiB
Markdown
160 lines
3.6 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
tags:
|
||
|
|
- merge
|
||
|
|
- mergekit
|
||
|
|
- MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
---
|
||
|
|
|
||
|
|
# Calme-Instruct-Extended
|
||
|
|
|
||
|
|
Calme-Instruct-Extended is a merge of the following models using [mergekit](https://github.com/cg123/mergekit):
|
||
|
|
* [MaziyarPanahi/Calme-7B-Instruct-v0.1.1](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.1.1)
|
||
|
|
|
||
|
|
|
||
|
|
## 🧩 Configuration
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
slices:
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 0
|
||
|
|
- 4
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 3
|
||
|
|
- 4
|
||
|
|
parameters:
|
||
|
|
scale:
|
||
|
|
- filter: o_proj
|
||
|
|
value: 0
|
||
|
|
- filter: down_proj
|
||
|
|
value: 0
|
||
|
|
- value: 1
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 4
|
||
|
|
- 8
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 7
|
||
|
|
- 8
|
||
|
|
parameters:
|
||
|
|
scale:
|
||
|
|
- filter: o_proj
|
||
|
|
value: 0
|
||
|
|
- filter: down_proj
|
||
|
|
value: 0
|
||
|
|
- value: 1
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 8
|
||
|
|
- 12
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 11
|
||
|
|
- 12
|
||
|
|
parameters:
|
||
|
|
scale:
|
||
|
|
- filter: o_proj
|
||
|
|
value: 0
|
||
|
|
- filter: down_proj
|
||
|
|
value: 0
|
||
|
|
- value: 1
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 12
|
||
|
|
- 16
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 15
|
||
|
|
- 16
|
||
|
|
parameters:
|
||
|
|
scale:
|
||
|
|
- filter: o_proj
|
||
|
|
value: 0
|
||
|
|
- filter: down_proj
|
||
|
|
value: 0
|
||
|
|
- value: 1
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 16
|
||
|
|
- 20
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 19
|
||
|
|
- 20
|
||
|
|
parameters:
|
||
|
|
scale:
|
||
|
|
- filter: o_proj
|
||
|
|
value: 0
|
||
|
|
- filter: down_proj
|
||
|
|
value: 0
|
||
|
|
- value: 1
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 20
|
||
|
|
- 24
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 23
|
||
|
|
- 24
|
||
|
|
parameters:
|
||
|
|
scale:
|
||
|
|
- filter: o_proj
|
||
|
|
value: 0
|
||
|
|
- filter: down_proj
|
||
|
|
value: 0
|
||
|
|
- value: 1
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 24
|
||
|
|
- 28
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 27
|
||
|
|
- 28
|
||
|
|
parameters:
|
||
|
|
scale:
|
||
|
|
- filter: o_proj
|
||
|
|
value: 0
|
||
|
|
- filter: down_proj
|
||
|
|
value: 0
|
||
|
|
- value: 1
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 28
|
||
|
|
- 32
|
||
|
|
- sources:
|
||
|
|
- model: MaziyarPanahi/Calme-7B-Instruct-v0.1.1
|
||
|
|
layer_range:
|
||
|
|
- 31
|
||
|
|
- 32
|
||
|
|
parameters:
|
||
|
|
scale:
|
||
|
|
- filter: o_proj
|
||
|
|
value: 0
|
||
|
|
- filter: down_proj
|
||
|
|
value: 0
|
||
|
|
- value: 1
|
||
|
|
merge_method: passthrough
|
||
|
|
dtype: bfloat16
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
```
|