90 lines
1.9 KiB
Markdown
90 lines
1.9 KiB
Markdown
---
|
||
base_model:
|
||
- meta-llama/Meta-Llama-3-8B
|
||
library_name: transformers
|
||
tags:
|
||
- mergekit
|
||
- merge
|
||
|
||
---
|
||
# output
|
||
|
||
This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
|
||
|
||
## Merge Details
|
||
### Merge Method
|
||
|
||
This model was merged using the Passthrough merge method.
|
||
|
||
### Models Merged
|
||
|
||
The following models were included in the merge:
|
||
* [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)
|
||
|
||
### Configuration
|
||
|
||
The following YAML configuration was used to produce this model:
|
||
|
||
```yaml
|
||
dtype: bfloat16
|
||
merge_method: passthrough
|
||
|
||
slices:
|
||
# untouched intro
|
||
- sources:
|
||
- layer_range: [0, 8]
|
||
model: meta-llama/Meta-Llama-3-8B
|
||
|
||
- sources:
|
||
- layer_range: [8, 10]
|
||
model: meta-llama/Meta-Llama-3-8B
|
||
# 8–16 baseline
|
||
- sources:
|
||
- layer_range: [8, 12]
|
||
model: meta-llama/Meta-Llama-3-8B
|
||
# 8–16 duplicate with projections nulled
|
||
- sources:
|
||
- layer_range: [8, 16]
|
||
model: meta-llama/Meta-Llama-3-8B
|
||
parameters:
|
||
scale:
|
||
- filter: o_proj
|
||
value: 0.0
|
||
- filter: down_proj
|
||
value: 0.0
|
||
- value: 1.0
|
||
|
||
# 16–24 duplicate
|
||
- sources:
|
||
- layer_range: [16, 20]
|
||
model: meta-llama/Meta-Llama-3-8B
|
||
parameters:
|
||
scale:
|
||
- filter: o_proj
|
||
value: 0.0
|
||
- filter: down_proj
|
||
value: 0.0
|
||
- value: 1.0
|
||
# 16–24 baseline
|
||
- sources:
|
||
- layer_range: [16, 24]
|
||
model: meta-llama/Meta-Llama-3-8B
|
||
# 16–24 duplicate
|
||
- sources:
|
||
- layer_range: [16, 20]
|
||
model: meta-llama/Meta-Llama-3-8B
|
||
parameters:
|
||
scale:
|
||
- filter: o_proj
|
||
value: 0.0
|
||
- filter: down_proj
|
||
value: 0.0
|
||
- value: 1.0
|
||
|
||
# untouched tail
|
||
- sources:
|
||
- layer_range: [24, 32]
|
||
model: meta-llama/Meta-Llama-3-8B
|
||
|
||
```
|