43 lines
1.5 KiB
Markdown
43 lines
1.5 KiB
Markdown
---
|
||
base_model:
|
||
- Qwen/Qwen2.5-14B
|
||
- Qwen/Qwen2.5-14B-Instruct
|
||
library_name: transformers
|
||
tags:
|
||
- mergekit
|
||
- merge
|
||
|
||
---
|
||
# Arcee Fusion: The Art of Selective Merging
|
||
|
||
## Arcee Fusion works in three key stages:
|
||
***1.Importance Scoring:***
|
||
|
||
*Instead of blindly merging all parameters, Arcee Fusion calculates an importance score for each parameter, combining the absolute difference between model parameters with a divergence measure based on softmax distributions and KL divergence. This ensures that only meaningful changes are considered.*
|
||
|
||
***2.Dynamic Thresholding:***
|
||
|
||
*The algorithm analyzes the distribution of importance scores, calculating key quantiles (median, Q1, and Q3) and setting a dynamic threshold using median + 1.5 × IQR (a standard technique for outlier detection). This intelligently filters out less significant changes.*
|
||
|
||
***3.Selective Integration:***
|
||
|
||
*A fusion mask is created based on the importance scores and the threshold. Only the most significant elements are incorporated into the base model, ensuring that the merge process is adaptive and selective. This preserves the base model's stability while integrating the most valuable updates from the other model.*
|
||
|
||
### Configuration
|
||
|
||
The following YAML configuration was used to produce this model:
|
||
|
||
```yaml
|
||
models:
|
||
- model: Qwen/Qwen2.5-14B-Instruct
|
||
merge_method: arcee_fusion
|
||
base_model: Qwen/Qwen2.5-14B
|
||
parameters:
|
||
normalize: true
|
||
int8_mask: true
|
||
rescale: false
|
||
dtype: float16
|
||
out_dtype: float16
|
||
tokenizer_source: base
|
||
```
|