Qwen3-4B-Thinking-2507-20250813-033307-1/README.md

language, license, library_name, tags, datasets, metrics, base_model, base_model_relation, model-index

language	license	library_name	tags	datasets	metrics	base_model	base_model_relation	model-index
en	apache-2.0	transformers	merge model-merging mergekit lazymergekit qwen3 4b text-generation causal-lm	Idavidrein/gpqa	accuracy	Qwen/Qwen3-4B-Thinking-2507 Qwen/Qwen3-4B-Thinking-2507-FP8 unsloth/Qwen3-4B-Thinking-2507 ertghiu256/Qwen3-4b-tcomanr-merge-v2 huihui-ai/Huihui-Qwen3-4B-Thinking-2507-abliterated janhq/Jan-v1-4B BRlkl/BingoGuard-qwen3-4B-pt-grpo fireworks1231/agentic-4b-2607-sft sequelbox/Qwen3-4B-Thinking-2507-DAG-Reasoning ReallyFloppyPenguin/Mastermind-2x4b-thinking	merge	name results qwen3-4b-merged---configuration-1 task dataset metrics type name text-generation Text Generation type name config split args cais/mmlu MMLU (Massive Multitask Language Understanding) all test num_few_shot 5 type value name verified accuracy 68.37 MMLU (5-shot) false task dataset metrics type name text-generation Text Generation type name config split args Idavidrein/gpqa GPQA (Graduate-level Physics Q&A) gpqa_diamond test num_few_shot 0 type value name verified accuracy 43.43 GPQA Diamond (0-shot) false

Qwen3-4B Merged - Configuration 1

This is a Qwen3-4B based model created through layer-wise merging of multiple fine-tuned variants to optimize performance on GPQA Diamond.

Performance Metrics

Benchmark	Score	Description
MMLU (5-shot)	0.6837 (68.37%)	Massive Multitask Language Understanding
GPQA Diamond (0-shot)	0.4343 (43.43%)	Graduate-level Physics Q&A

Benchmark Details

• MMLU: Evaluated on the test set with 5-shot prompting across 57 subjects
• GPQA: Evaluated on the diamond subset with 0-shot prompting on graduate-level physics questions

Performance Visualizations

GPQA Diamond Performance Comparison

MMLU and GPQA Diamond Combined Performance

Model Information

• Run ID: 20250813_033307
• Optimization Task: GPQA (Graduate-level Physics Q&A)
• Number of Layers: 36
• Base Architecture: Qwen3-4B

Source Models

The following models were used in the layer-wise merge:

• Qwen/Qwen3-4B-Thinking-2507
• Qwen/Qwen3-4B-Thinking-2507-FP8
• unsloth/Qwen3-4B-Thinking-2507
• ertghiu256/Qwen3-4b-tcomanr-merge-v2
• huihui-ai/Huihui-Qwen3-4B-Thinking-2507-abliterated
• janhq/Jan-v1-4B
• BRlkl/BingoGuard-qwen3-4B-pt-grpo
• fireworks1231/agentic-4b-2607-sft
• sequelbox/Qwen3-4B-Thinking-2507-DAG-Reasoning
• ReallyFloppyPenguin/Mastermind-2x4b-thinking

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load the model
model = AutoModelForCausalLM.from_pretrained(
    "EganAI/Qwen3-4B-Thinking-2507-20250813-033307-1",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("EganAI/Qwen3-4B-Thinking-2507-20250813-033307-1")

# Example: MMLU-style question
prompt = '''Question: The study of the distribution and determinants of health and disease in populations is:
A) Epidemiology
B) Ecology  
C) Etiology
D) Endocrinology
Answer:'''

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_length=150,
    temperature=0.7,
    do_sample=True
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Inference with vLLM

from vllm import LLM, SamplingParams

llm = LLM(model="EganAI/Qwen3-4B-Thinking-2507-20250813-033307-1")
sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=256)

prompts = ["Question: Explain quantum entanglement in simple terms."]
outputs = llm.generate(prompts, sampling_params)

Technical Details

This model uses a layer-wise merging approach where each transformer layer is selected from different source models based on optimization criteria. This technique allows combining strengths from multiple fine-tuned models.

Merging Process

1. Layer Selection: Each layer (0-35 for this architecture) is independently selected from one of the source models
2. Non-layer Weights: Embeddings and final layers are taken from the base model
3. Optimization: The configuration was found through systematic optimization on the target benchmark

Limitations

• This is an experimental merge and performance may vary on tasks outside the optimization targets
• The model inherits limitations from its source models
• Performance on general tasks may differ from benchmark scores

Citation

If you use this model, please cite the original source models and egan.ai

Note

This model is provided for research purposes. Always validate performance on your specific use case before deployment.