初始化项目，由ModelHub XC社区提供模型

Model: lemon07r/Gemma-2-Ataraxy-9B Source: Original Platform
2026-09-10 11:50:16 +08:00
commit 772027f3de
14 changed files with 2109 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,210 @@
+---
+license: gemma
+library_name: transformers
+tags:
+- mergekit
+- merge
+base_model:
+- princeton-nlp/gemma-2-9b-it-SimPO
+- nbeerbower/gemma2-gutenberg-9B
+model-index:
+- name: Gemma-2-Ataraxy-9B
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: IFEval (0-Shot)
+      type: HuggingFaceH4/ifeval
+      args:
+        num_few_shot: 0
+    metrics:
+    - type: inst_level_strict_acc and prompt_level_strict_acc
+      value: 30.09
+      name: strict accuracy
+    source:
+      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=lemon07r/Gemma-2-Ataraxy-9B
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: BBH (3-Shot)
+      type: BBH
+      args:
+        num_few_shot: 3
+    metrics:
+    - type: acc_norm
+      value: 42.03
+      name: normalized accuracy
+    source:
+      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=lemon07r/Gemma-2-Ataraxy-9B
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MATH Lvl 5 (4-Shot)
+      type: hendrycks/competition_math
+      args:
+        num_few_shot: 4
+    metrics:
+    - type: exact_match
+      value: 0.83
+      name: exact match
+    source:
+      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=lemon07r/Gemma-2-Ataraxy-9B
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: GPQA (0-shot)
+      type: Idavidrein/gpqa
+      args:
+        num_few_shot: 0
+    metrics:
+    - type: acc_norm
+      value: 11.3
+      name: acc_norm
+    source:
+      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=lemon07r/Gemma-2-Ataraxy-9B
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MuSR (0-shot)
+      type: TAUR-Lab/MuSR
+      args:
+        num_few_shot: 0
+    metrics:
+    - type: acc_norm
+      value: 14.47
+      name: acc_norm
+    source:
+      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=lemon07r/Gemma-2-Ataraxy-9B
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MMLU-PRO (5-shot)
+      type: TIGER-Lab/MMLU-Pro
+      config: main
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 35.85
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=lemon07r/Gemma-2-Ataraxy-9B
+      name: Open LLM Leaderboard
+---
+# Gemma-2-Ataraxy-9B
+
+![Ataraxy](https://i.imgur.com/aP03a5d.png)
+
+Working on some new versions. Sit tight. 
+
+## GGUF / EXL2 Quants
+
+Huge thanks to [@mradermacher](https://huggingface.co/mradermacher) and [@bartowski](https://huggingface.co/bartowski) for making these GGUF quants available to us.
+
+Bartowski quants (imatrix): [bartowski/Gemma-2-Ataraxy-9B-GGUF](https://huggingface.co/bartowski/Gemma-2-Ataraxy-9B-GGUF)
+
+Mradermacher quants (static): [mradermacher/Gemma-2-Ataraxy-9B-GGUF](https://huggingface.co/mradermacher/Gemma-2-Ataraxy-9B-GGUF)
+
+Mradermacher quants (imatrix): [mradermacher/Gemma-2-Ataraxy-9B-i1-GGUF](https://huggingface.co/mradermacher/Gemma-2-Ataraxy-9B-i1-GGUF)
+
+I think bartowski and mradermacher use different calibration data for imatrix quants, or maybe you prefer static quants. Pick your poison :).
+
+EXL2 quants from CameronRedmore: [CameronRedmore/Gemma-2-Ataraxy-9B-exl2](https://huggingface.co/CameronRedmore/Gemma-2-Ataraxy-9B-exl2)
+## Format
+
+Use Gemma 2 format.
+
+## Benchmarks and Leaderboard Rankings
+
+Creative Writing V2 - Score: 82.64 (Rank 1)
+
+https://eqbench.com/creative_writing.html
+
+That's right, much to everyone's surprise (mine included) this model has topped eqbench.com's creative writing benchmark. Big thanks to _sqrkl for taking the time to bother benchmarking this model and adding it ot his leaderboard, and [@nbeerbower](https://huggingface.co/nbeerbower) for entertaining my suggestions for his amazing gutenberg tunes.
+
+![Reddit](https://i.imgur.com/iL8BcYb.png)
+
+![Leaderboard](https://i.imgur.com/gJd9Pab.png)
+
+# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
+Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_lemon07r__Gemma-2-Ataraxy-9B)
+
+|      Metric       |Value|
+|-------------------|----:|
+|Avg.               |22.43|
+|IFEval (0-Shot)    |30.09|
+|BBH (3-Shot)       |42.03|
+|MATH Lvl 5 (4-Shot)| 0.83|
+|GPQA (0-shot)      |11.30|
+|MuSR (0-shot)      |14.47|
+|MMLU-PRO (5-shot)  |35.85|
+
+This makes this one of the smartest Gemma 2 9B finetunes/merges, only a few score slightly higher. This model scores above SimPO and SPPO iter2/3 by a whole point on average.
+
+## Preface and Rambling
+
+My favorite Gemma 2 9B models are the SPPO iter3 and SimPO finetunes, but I felt the slerp merge between the two (nephilim v3) wasn't as good for some reason. The Gutenberg Gemma 2 finetune by nbeerbower is another my favorites. It's trained on one of my favorite datasets, and actually improves the SPPO model's average openllm leaderboard 2 average score by a bit, on top of improving it's writing capabilities and making the LLM sound less AI-like. However I still liked the original SPPO finetune just a bit more. 
+
+Someone suggested that merging the base model on top of the gutenberg may help with tame it back down, which gave me a (possibly) better idea; slerp merging the SimPO finetune on top of the Gutenberg finetune, which is similar to the pretty popular Nephilim v3 recipe, using the Gutenberg finetune in place of the SPPO model, which I thought may give us better results since Gutenberg was trained on top of SPPO. 
+
+I wasn't entirely too sure, since if nephilim v3 is anything to go by, it was probably going to also end up worse than the parent models. Normally when I try merges like these, they dont go anywhere. I'm pretty picky, and very skeptical usually, so most times I find that the merge is usually just not better than the original models or only marginally better. Tried this merge anyways to see how it goes, and much to my surprise, this time, I feel like I got very good results. Figured I'd share, and hopefully this wont be just me introducing more useless slop into a world that already has way too many unnecessary merges. 
+
+If you're looking for a mistral nemo 12B model instead, I HIGHLY recommend Mistral Nemo Gutenberg v2 by nbeerbower. It's head and shoulders above the many other mistral nemo finetunes I've tried (the first version of mistral nemo gutenburg, romulus simpo, and magnum mini 1.1 being close second favorites).
+
+## Why Gutenberg?
+
+We use gutenberg 9b, which is finetuned over SPPO cause of how good the gutenberg dpo dataset is. It's not a german dataset like it sounds, it's a dataset based off of project gutenberg, a public domain collection of popular classic fictions, like Moby Dick, for example. This data set also uses LLM generated responses as the negative prompt, to train models to not sounds so much like AI, or your typical LLMs, and more like actual humans (based on the creative works from project gutenberg). This is quality writing, hence quality data, not just random RP logs, or synthetic data. This dataset, when trained for 3 epochs has shown to increase llama 3 average scores on the old openllm leaderboard, from 72 to 73 (nobody really got higher than 73 before the board revamp), and has already again, proven to increase average openllm 2 leaderboard scores, increasing the average score from 21.47 to 22.61, improving on sppo. Thats a huge improvement. That said, I didnt like gutenberg 9b more than original sppo in real use, felt a tiny bit overfit, so we tried this merge. Did not expect much, because of neph v3 turning out worse than either of its parents, but this surprisingly came out great.
+
+## Why is it 10b??
+
+See https://github.com/arcee-ai/mergekit/issues/390
+
+Model is not actually 10b, mergekit is randomly adding lm_head for some reason when doing SLERP merge with Gemma 2 models. I believe Nephilim v3 had a similar issue before they used some sort of workaround that I'm not aware of. Doesn't seem like this affects the GGUF quants, as they're the correct size, so I will leave it as is until mergekit gets a commit that addresses this issue.
+
+## Merge Details
+### Merge Method
+
+This model was merged using the SLERP merge method.
+
+### Models Merged
+
+The following models were included in the merge:
+* [princeton-nlp/gemma-2-9b-it-SimPO](https://huggingface.co/princeton-nlp/gemma-2-9b-it-SimPO)
+* [nbeerbower/gemma2-gutenberg-9B](https://huggingface.co/nbeerbower/gemma2-gutenberg-9B)
+
+### Configuration
+
+The following YAML configuration was used to produce this model:
+
+```yaml
+base_model: nbeerbower/gemma2-gutenberg-9B
+dtype: bfloat16
+merge_method: slerp
+parameters:
+  t:
+  - filter: self_attn
+    value: [0.0, 0.5, 0.3, 0.7, 1.0]
+  - filter: mlp
+    value: [1.0, 0.5, 0.7, 0.3, 0.0]
+  - value: 0.5
+slices:
+- sources:
+  - layer_range: [0, 42]
+    model: princeton-nlp/gemma-2-9b-it-SimPO
+  - layer_range: [0, 42]
+    model: nbeerbower/gemma2-gutenberg-9B
+```
+
+
--- a/config.json
+++ b/config.json
@@ -0,0 +1,34 @@
+{
+  "_name_or_path": "nbeerbower/gemma2-gutenberg-9B",
+  "architectures": [
+    "Gemma2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_logit_softcapping": 50.0,
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "eos_token_id": 1,
+  "final_logit_softcapping": 30.0,
+  "head_dim": 256,
+  "hidden_act": "gelu_pytorch_tanh",
+  "hidden_activation": "gelu_pytorch_tanh",
+  "hidden_size": 3584,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 8192,
+  "model_type": "gemma2",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 42,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "query_pre_attn_scalar": 224,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 10000.0,
+  "sliding_window": 4096,
+  "sliding_window_size": 4096,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.43.3",
+  "use_cache": true,
+  "vocab_size": 256000
+}
--- a/mergekit_config.yml
+++ b/mergekit_config.yml
@@ -0,0 +1,16 @@
+base_model: nbeerbower/gemma2-gutenberg-9B
+dtype: bfloat16
+merge_method: slerp
+parameters:
+  t:
+  - filter: self_attn
+    value: [0.0, 0.5, 0.3, 0.7, 1.0]
+  - filter: mlp
+    value: [1.0, 0.5, 0.7, 0.3, 0.0]
+  - value: 0.5
+slices:
+- sources:
+  - layer_range: [0, 42]
+    model: princeton-nlp/gemma-2-9b-it-SimPO
+  - layer_range: [0, 42]
+    model: nbeerbower/gemma2-gutenberg-9B
--- a/model-00001-of-00005.safetensors
+++ b/model-00001-of-00005.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b32a9e3a5842f233093e345518bdf0e83cda0880fc7b50113e8d95c2ebf5d5a
+size 4961959144
--- a/model-00002-of-00005.safetensors
+++ b/model-00002-of-00005.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a407b4a8ccdfd08fdeef72c5dc557fb3533ac96c8d1315e299b8511f6a4aa46
+size 4976923496
--- a/model-00003-of-00005.safetensors
+++ b/model-00003-of-00005.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:149b5498aa2d0dc91e93482bebfe3d262d629632b2624b820abccbd04f2ac92e
+size 4932868832
--- a/model-00004-of-00005.safetensors
+++ b/model-00004-of-00005.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:609df42f78f150d001f33dd2b833db7005740b949fc440bb0826e18c9f8b2882
+size 4976923456
--- a/model-00005-of-00005.safetensors
+++ b/model-00005-of-00005.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d6779c356e1f4b8d1c7a64fb93772e7206b3e0dbcdc7307b4c071e71cc57dce
+size 469799616
--- a/model.safetensors.index.json
+++ b/model.safetensors.index.json
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,34 @@
+{
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7da53ca29fb16f6b2489482fc0bc6a394162cdab14d12764a1755ebc583fea79
+size 17518525
--- a/tokenizer.model
+++ b/tokenizer.model
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61a7b147390c64585d6c3543dd6fc636906c9af3865a5548f27f31aee1d4c8e2
+size 4241003
--- a/tokenizer_config.json
+++ b/tokenizer_config.json