OpenChat-3.5-0106_8.99B_40Layers-Appended/README.md

license, library_name, tags, base_model, model-index

license	library_name	tags	base_model	model-index
apache-2.0	transformers	mergekit merge	openchat/openchat-3.5-0106	name results OpenChat-3.5-0106_8.99B_40Layers-Appended task dataset metrics source type name text-generation Text Generation name type args IFEval (0-Shot) HuggingFaceH4/ifeval num_few_shot 0 type value name inst_level_strict_acc and prompt_level_strict_acc 59.61 strict accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Appended Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args BBH (3-Shot) BBH num_few_shot 3 type value name acc_norm 24.06 normalized accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Appended Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MATH Lvl 5 (4-Shot) hendrycks/competition_math num_few_shot 4 type value name exact_match 6.8 exact match url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Appended Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args GPQA (0-shot) Idavidrein/gpqa num_few_shot 0 type value name acc_norm 7.61 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Appended Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type args MuSR (0-shot) TAUR-Lab/MuSR num_few_shot 0 type value name acc_norm 11.78 acc_norm url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Appended Open LLM Leaderboard task dataset metrics source type name text-generation Text Generation name type config split args MMLU-PRO (5-shot) TIGER-Lab/MMLU-Pro main test num_few_shot 5 type value name acc 25.44 accuracy url name https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/OpenChat-3.5-0106_8.99B_40Layers-Appended Open LLM Leaderboard

Buy me a Ko-Fi • Support my work using Patreon

OpenChat-3.5-0106_8.99B_40Layers-Appended

This is NOT your usual frankenmerge created using mergekit.

Merge Details

Merge Method

This model was merged using the passthrough merge method, but employing a variation of the Block Expansion method described in the paper LLaMA Pro: Progressive LLaMA with Block Expansion.

The authors of the paper added new layers interleaved in between the original layers of the model, setting the parameters of the o_proj and down_proj layers to zero. This effectively adds layers that will just output their input (as if they were "transparent") allowing the model to remain functional even without further training. These new layers can then be targeted during training or fine-tuning without risking catastrophic forgetting, if you follow the author's training method to freeze the original layers and only train the new layers.

I used the same method but added the new layers to the end of the model. My rationale is that the level of abstraction increases with each layer of the model. So, while new layers spread along the original layers will help the model to learn new tasks, adding layers to the end of the model and then re-training/fine-tuning the model on tasks it already performs well could improve the models understanding of those task and perform them better by employing more complex reasoning.

This model has not yet received additional training, so it should perform close to the original model.

Models Merged

The following models were included in the merge:

• openchat/openchat-3.5-0106

Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
    - model: openchat/openchat-3.5-0106
      layer_range: [0, 32]
  - sources:
    - model: openchat/openchat-3.5-0106
      layer_range: [31, 32]
      parameters:
        scale:
          - filter: o_proj
            value: 0.0
          - filter: down_proj
            value: 0.0
          - value: 1.0
  - sources:
    - model: openchat/openchat-3.5-0106
      layer_range: [31, 32]
      parameters:
        scale:
          - filter: o_proj
            value: 0.0
          - filter: down_proj
            value: 0.0
          - value: 1.0
  - sources:
    - model: openchat/openchat-3.5-0106
      layer_range: [31, 32]
      parameters:
        scale:
          - filter: o_proj
            value: 0.0
          - filter: down_proj
            value: 0.0
          - value: 1.0
  - sources:
    - model: openchat/openchat-3.5-0106
      layer_range: [31, 32]
      parameters:
        scale:
          - filter: o_proj
            value: 0.0
          - filter: down_proj
            value: 0.0
          - value: 1.0
  - sources:
    - model: openchat/openchat-3.5-0106
      layer_range: [31, 32]
      parameters:
        scale:
          - filter: o_proj
            value: 0.0
          - filter: down_proj
            value: 0.0
          - value: 1.0
  - sources:
    - model: openchat/openchat-3.5-0106
      layer_range: [31, 32]
      parameters:
        scale:
          - filter: o_proj
            value: 0.0
          - filter: down_proj
            value: 0.0
          - value: 1.0
  - sources:
    - model: openchat/openchat-3.5-0106
      layer_range: [31, 32]
      parameters:
        scale:
          - filter: o_proj
            value: 0.0
          - filter: down_proj
            value: 0.0
          - value: 1.0
  - sources:
    - model: openchat/openchat-3.5-0106
      layer_range: [31, 32]
      parameters:
        scale:
          - filter: o_proj
            value: 0.0
          - filter: down_proj
            value: 0.0
          - value: 1.0
merge_method: passthrough
dtype: bfloat16

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	22.55
IFEval (0-Shot)	59.61
BBH (3-Shot)	24.06
MATH Lvl 5 (4-Shot)	6.80
GPQA (0-shot)	7.61
MuSR (0-shot)	11.78
MMLU-PRO (5-shot)	25.44

Citation

@misc{wu2024llamaproprogressivellama,
      title={LLaMA Pro: Progressive LLaMA with Block Expansion}, 
      author={Chengyue Wu and Yukang Gan and Yixiao Ge and Zeyu Lu and Jiahao Wang and Ye Feng and Ying Shan and Ping Luo},
      year={2024},
      eprint={2401.02415},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2401.02415}, 
}