初始化项目，由ModelHub XC社区提供模型

Model: OzdowntheR/Qwen3-0.6B-SciGen-SLERP Source: Original Platform
2026-07-14 16:07:10 +08:00
commit f04e2840ab
9 changed files with 527 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,278 @@
+---
+license: apache-2.0
+language:
+- en
+base_model: Qwen/Qwen3-0.6B
+datasets:
+- camel-ai/physics
+- HuggingFaceH4/ultrachat_200k
+tags:
+- merge
+- mergekit
+- slerp
+- qwen3
+pipeline_tag: text-generation
+---
+
+# Qwen3-0.6B-SciGen-SLERP
+
+A SLERP merge of two domain-specialized full-SFT Qwen3-0.6B experts.
+
+**596M parameters** (same as base) · **bfloat16** · Architecture: Qwen3ForCausalLM
+
+## TL;DR
+
+Two Qwen3-0.6B models were full-fine-tuned on different domains (physics + general chat), then merged via SLERP at t=0.3. The result averages **+5.9%** over base on the HF Open LLM Leaderboard v1 task suite (ARC, HellaSwag, BoolQ, PIQA, Winogrande). On the harder Leaderboard v2 task suite (BBH, MMLU-Pro, MuSR, IFEval, MATH Hard) results are mixed: SLERP wins IFEval and MuSR, but loses BBH, MMLU-Pro, and MATH Hard. The merge also suppresses Qwen3's thinking behavior, making it faster and more direct but less suitable for tasks that benefit from explicit reasoning. See the v2 interpretation section for important caveats on the IFEval gain.
+
+Intended Use: Research and experimentation only. 
+
+## Training
+
+### Expert 1 — Science
+
+| Setting | Value |
+|---------|-------|
+| Base model | Qwen/Qwen3-0.6B |
+| Dataset | [camel-ai/physics](https://huggingface.co/datasets/camel-ai/physics) (20K examples) |
+| Training method | Full SFT (all 596M params trainable) |
+| Learning rate | 5e-5, cosine schedule, 100 step warmup |
+| Optimizer | paged_adamw_8bit |
+| Epochs | 1 |
+| Effective batch size | 32 (2 GPUs × 2 per_device × 8 grad_accum) |
+| Steps | 600 |
+| Hardware | 2× RTX 5060 Ti 16GB, FSDP full shard |
+| Wall time | ~77 minutes |
+| Weight divergence from base | 0.000427 (mean abs diff) |
+
+### Expert 2 — General
+
+| Setting | Value |
+|---------|-------|
+| Base model | Qwen/Qwen3-0.6B |
+| Dataset | [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) (25K subset) |
+| Training method | Full SFT (all 596M params trainable) |
+| Learning rate | 5e-5, cosine schedule, 100 step warmup |
+| Optimizer | paged_adamw_8bit |
+| Epochs | 1 |
+| Effective batch size | 32 (2 GPUs × 2 per_device × 8 grad_accum) |
+| Steps | 750 |
+| Hardware | 2× RTX 5060 Ti 16GB, FSDP full shard |
+| Weight divergence from base | 0.000607 (mean abs diff) |
+
+## Merge
+
+Tool: [mergekit](https://github.com/arcee-ai/mergekit) v0.1.4
+
+```yaml
+base_model: expert_science/checkpoint-600
+dtype: bfloat16
+merge_method: slerp
+modules:
+  default:
+    slices:
+    - sources:
+      - layer_range: [0, 28]
+        model: expert_science/checkpoint-600
+      - layer_range: [0, 28]
+        model: expert_general/checkpoint-750
+parameters:
+  t:
+  - value: 0.3
+```
+
+t=0.3 means 70% science expert, 30% general expert, applied uniformly across all 28 layers.
+
+**Note on t selection:** Two SLERP t values were tested manually (0.3 and 0.5). t=0.3 scored higher on the v1 task suite (0.5819 vs 0.5738 average). A per-layer CMA-ES search over t values was also attempted but did not beat the global t=0.3 baseline. No systematic sweep over additional t values was performed.
+
+## Evaluation settings (all tasks)
+
+| Setting | Value |
+|---------|-------|
+| Harness | [EleutherAI lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) v0.4.11 |
+| Model dtype | bfloat16 |
+| Decoding (v1 / v2 benchmarks) | Greedy (`do_sample=False`, `temperature=0.0`) |
+| Decoding (qualitative) | Sampling (`temperature=0.6, top_p=0.95, top_k=20`) — Qwen3 recommended |
+| Repetition penalty | None (1.0) |
+| Max generation tokens | 2048 (v2), 8192 (qualitative) |
+| Thinking mode | Enabled (default Qwen3 chat template behavior) |
+| Sample limit | None (full datasets) |
+| v1 chat template | Not applied (loglikelihood tasks) |
+| v2 chat template | Not applied — see note in v2 section below |
+| Qualitative chat template | Applied (real-user simulation) |
+
+## Evaluation — HF Open LLM Leaderboard v1 task suite
+
+(The "v1" / "v2" naming refers to two different HuggingFace benchmark suites with different task sets — not different versions of our evaluation.)
+
+
+Evaluated with [EleutherAI lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) v0.4.11. Full datasets, no sample limit, dtype=bfloat16. Each task uses HF Leaderboard v1 standard few-shot settings.
+
+| Task | n-shot | Metric | Base | SLERP | Delta |
+|------|--------|--------|------|-------|-------|
+| ARC Challenge | 25 | acc_norm | 0.4019 | **0.4181** | +0.0162 |
+| HellaSwag | 10 | acc_norm | 0.4678 | **0.4937** | +0.0259 |
+| BoolQ | 0 | acc | 0.6410 | **0.7419** | +0.1009 |
+| PIQA | 0 | acc_norm | 0.6746 | **0.6844** | +0.0098 |
+| Winogrande | 5 | acc | 0.5620 | **0.5714** | +0.0094 |
+| **Average** | | | **0.5495** | **0.5819** | **+0.0324 (+5.9%)** |
+
+All numbers were verified with an independent rerun matching the originals to within standard error. The BoolQ +10% gain is the largest single contributor and traces back to the science expert's factual Q&A training format.
+
+## Evaluation — HF Open LLM Leaderboard v2 task suite
+
+Evaluated with lm-evaluation-harness v0.4.11. **Note:** `--apply_chat_template` was NOT used because Qwen3-0.6B exhibits degenerate behavior (collapsing to a single answer choice on >94% of MMLU-Pro questions) when chat-templated for loglikelihood multiple-choice tasks. This was verified by inspecting per-sample answer distributions. Generation was run with `max_gen_toks=2048`, thinking enabled (default), greedy decoding.
+
+### Loglikelihood tasks (multiple choice)
+
+| Task | Metric | Base | SLERP | Delta | Stderr |
+|------|--------|------|-------|-------|--------|
+| BBH | acc_norm | **0.3687** | 0.3555 | -0.0132 | ±0.006 |
+| MMLU-Pro | acc | **0.2389** | 0.2149 | -0.0240 | ±0.004 |
+| MuSR | acc_norm | 0.3148 | **0.3399** | +0.0251 | ±0.016 |
+
+### Generation tasks
+
+| Task | Metric | Base | SLERP | Delta | Stderr |
+|------|--------|------|-------|-------|--------|
+| IFEval | prompt_level_strict | 0.1442 | **0.1774** | +0.0332 | ±0.016 |
+| IFEval | inst_level_strict | 0.2806 | **0.3082** | +0.0276 | — |
+| IFEval | prompt_level_loose | 0.1608 | **0.2033** | +0.0425 | ±0.017 |
+| IFEval | inst_level_loose | 0.2986 | **0.3333** | +0.0347 | — |
+| MATH Hard | exact_match | **0.1352** | 0.0755 | -0.0597 | ±0.009 |
+
+### v2 interpretation
+
+SLERP wins IFEval, MuSR, and loses BBH, MMLU-Pro, MATH Hard at the greedy `max_gen_toks=2048` setting these benchmarks were run at.
+
+**Important caveat on the IFEval gain:** these v2 numbers were all collected with greedy decoding and a 2048-token generation budget — the same settings that the qualitative comparison shows cause base Qwen3-0.6B to frequently run out of tokens mid-`<think>` and never produce a final answer. Because SLERP suppresses thinking, it almost always completes its response within the budget. The IFEval gain is therefore at least partly an artifact of "completes within token budget" rather than "follows instructions more precisely." When given proper sampling settings (see qualitative section, prompt 1), base actually follows strict format rules better than SLERP on the example tested.
+
+The MATH Hard regression is more straightforward: math benefits from explicit reasoning, and neither expert was trained on math. SLERP's thinking suppression makes the regression worse. Don't use this merge for math.
+
+**Net characterization:** SLERP is a faster, more direct variant of Qwen3-0.6B. Its benchmark advantages partly come from being more reliable about producing *some* answer within a tight token budget, not from being a stronger reasoner.
+
+## Qualitative comparison
+
+5 prompts run on both models using **Qwen3's recommended inference settings**: `temperature=0.6, top_p=0.95, top_k=20, do_sample=True, max_new_tokens=8192`. Chat template applied with thinking enabled. Designed to probe expected strengths (instruction following, science, helpfulness) and known weaknesses (math). Full outputs at `results/qualitative_comparison_proper.json`.
+
+(Note: this differs from the v1/v2 benchmark settings, which use greedy decoding for reproducibility. These qualitative settings reflect what a real user calling the model in chat mode would see.)
+
+### Prompt 1 — Strict instruction following
+
+> Write exactly 3 bullet points explaining why the sky is blue. Use no more than 15 words per bullet. End your response with "---END---"
+
+**Base** (305 tokens): Used thinking to plan, counted words, produced 3 properly-formatted bullets each under 15 words, ended with `---END---`. **Followed all instructions.**
+
+**SLERP** (135 tokens): Skipped thinking, used numbered points instead of bullets, exceeded the 15-word limit, **omitted the `---END---` marker.**
+
+### Prompt 2 — Science explanation
+
+> Explain the difference between conduction, convection, and radiation in heat transfer.
+
+**Base** (839 tokens): Used thinking, then produced a complete structured explanation with mechanism, medium, direction, and example for each — plus a summary.
+
+**SLERP** (466 tokens): Skipped thinking, gave decent definitions but truncated mid-radiation explanation.
+
+### Prompt 3 — Helpful explanation
+
+> I'm learning Python and struggling with list comprehensions. Can you explain them simply with one example?
+
+**Base** (591 tokens): Complete answer with code example, bullet-pointed explanation, and listed use cases.
+
+**SLERP** (170 tokens): Working code example with brief explanation. Concise but less detailed.
+
+### Prompt 4 — Arithmetic
+
+> A store sells notebooks for $2.75 each. If I buy 7 notebooks and pay with a $20 bill, how much change do I get?
+
+**Base** (843 tokens): Worked through the problem in thinking, double-checked using cents, produced final boxed answer `$0.75`. **Correct.**
+
+**SLERP** (111 tokens): Direct calculation, `$0.75`. **Correct, ~1/7 the tokens.**
+
+### Prompt 5 — Multi-step reasoning
+
+> A train leaves Station A at 9:00 AM traveling at 60 mph. Another train leaves Station B (180 miles away) at 10:00 AM traveling toward Station A at 90 mph. At what time do they meet?
+
+**Base** (7966 tokens): Set up the correct equation `60t + 90(t-1) = 180`, solved `t = 1.8 hours`. Then bungled the final clock conversion: wrote "9:00 AM + 1 hour + 48 minutes = 9:48 AM" (correct addition is 10:48 AM). **Reasoning correct, final arithmetic wrong.**
+
+**SLERP** (449 tokens): Used the wrong equation entirely (forgot Train B departs an hour later), then produced a confused conclusion claiming the trains meet at "10:12 AM and 12:00 PM" simultaneously. **Wrong.**
+
+**Both wrong, but base was structurally correct and SLERP wasn't.** The correct answer is ~10:48 AM.
+
+### Pattern observed
+
+- **The merge suppresses thinking behavior.** Across all 5 prompts, base Qwen3-0.6B opens with `<think>...</think>` and reasons through the problem. SLERP outputs an empty `<think>\n\n</think>` and goes straight to the answer on every prompt. This was not an intentional design choice — the ultrachat (general) expert was trained on conversational data without thinking traces, and merging it appears to have weighted the model away from using the thinking block.
+- **SLERP is consistently faster.** It produces 3-7× fewer tokens per response across all prompts. For latency-sensitive use cases that's a real practical benefit.
+- **Base is more thorough and more correct on complex prompts.** When given proper sampling settings (temp=0.6, top_p=0.95), base completes its responses, follows complex instructions accurately, and sets up multi-step reasoning correctly. SLERP's brevity costs accuracy on harder prompts.
+- **Both can solve simple arithmetic.** Both got prompt 4 right.
+- **Neither reliably solves multi-step word problems** at this scale, but base's structured reasoning at least produces a checkable workflow.
+- **Practical implication:** SLERP is positioned as a **faster, more direct** variant of Qwen3-0.6B. Use it when you want short answers without verbose chain-of-thought. Use the base model when you want full reasoning capability or strict instruction compliance.
+
+## Reproducing the evaluation
+
+### v1 (per task)
+
+Base:
+```bash
+lm_eval --model hf \
+    --model_args pretrained=Qwen/Qwen3-0.6B,dtype=bfloat16 \
+    --tasks arc_challenge --num_fewshot 25 --batch_size 4
+# Repeat for hellaswag (10), boolq (0), piqa (0), winogrande (5)
+```
+
+This merge:
+```bash
+lm_eval --model hf \
+    --model_args pretrained=OzdowntheR/Qwen3-0.6B-SciGen-SLERP,dtype=bfloat16 \
+    --tasks arc_challenge --num_fewshot 25 --batch_size 4
+# Repeat for hellaswag (10), boolq (0), piqa (0), winogrande (5)
+```
+
+### v2 (per task, no chat template)
+
+Base:
+```bash
+lm_eval --model hf \
+    --model_args pretrained=Qwen/Qwen3-0.6B,dtype=bfloat16 \
+    --tasks leaderboard_bbh \
+    --batch_size auto \
+    --gen_kwargs max_gen_toks=2048 \
+    --log_samples \
+    --output_path results/base_v2_clean_leaderboard_bbh
+# Repeat for leaderboard_mmlu_pro, leaderboard_musr,
+#            leaderboard_ifeval, leaderboard_math_hard
+```
+
+This merge:
+```bash
+lm_eval --model hf \
+    --model_args pretrained=OzdowntheR/Qwen3-0.6B-SciGen-SLERP,dtype=bfloat16 \
+    --tasks leaderboard_bbh \
+    --batch_size auto \
+    --gen_kwargs max_gen_toks=2048 \
+    --log_samples \
+    --output_path results/slerp_v2_clean_leaderboard_bbh
+# Repeat for leaderboard_mmlu_pro, leaderboard_musr,
+#            leaderboard_ifeval, leaderboard_math_hard
+```
+
+## Limitations
+
+- **Thinking suppression.** The merge produces empty `<think>` blocks and skips chain-of-thought reasoning entirely. This is an unintended side effect of the general expert's training data (ultrachat) not containing thinking traces. For tasks requiring explicit reasoning, use the base model.
+- **0.6B is small.** Absolute scores on hard reasoning benchmarks (MATH Hard, MMLU-Pro, BBH) are low for both base and merge.
+- **MATH Hard regression** (-6%) is the largest individual loss. If your use case involves math, prefer the base model.
+- **Trained only on English** synthetic data (GPT-4 generated). Quality ceiling is bounded by the teacher.
+- **Tested only with thinking enabled** at `max_gen_toks=2048`. Different generation budgets may produce different results.
+
+## Provenance
+
+- v1 results verified with independent rerun (April 9, 2026). All numbers reproduced within standard error.
+- v2 results were initially run with `--apply_chat_template`, then re-run without after we discovered the chat template causes Qwen3-0.6B to collapse to degenerate answer distributions on loglikelihood multiple-choice. Only the no-chat-template results are reported here.
+- Per-sample logs from all v2 runs are saved as JSONL files alongside the result JSONs.
+
+## Citations
+
+- Qwen3 base model: [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)
+- mergekit (Goddard et al.): https://github.com/arcee-ai/mergekit
+- lm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness
+- camel-ai/physics dataset: [camel-ai/physics](https://huggingface.co/datasets/camel-ai/physics)
+- ultrachat_200k dataset: [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k)
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,89 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,63 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 40960,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 1000000,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.0",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,12 @@
+{
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "temperature": 0.6,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "5.5.0"
+}
--- a/mergekit_config.yml
+++ b/mergekit_config.yml
@@ -0,0 +1,14 @@
+base_model: expert_science/checkpoint-600
+dtype: bfloat16
+merge_method: slerp
+modules:
+  default:
+    slices:
+    - sources:
+      - layer_range: [0, 28]
+        model: expert_science/checkpoint-600
+      - layer_range: [0, 28]
+        model: expert_general/checkpoint-750
+parameters:
+  t:
+  - value: 0.3
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df05d190832d4910e44c98da6bba52ec6cb25e4c1fbea32e52706c3c31c9ed43
+size 1192135096
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,29 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "model_max_length": 2048,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}