109 lines
3.6 KiB
Markdown
109 lines
3.6 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- llm
|
|
- reasoning
|
|
- math
|
|
- eka
|
|
---
|
|
|
|
# Eka-4B
|
|
|
|
Eka-4B is a 4-billion parameter language model optimised for **mathematical reasoning** and **code**. Despite its compact size, Eka-4B delivers performance competitive with or superior to significantly larger open-source models across math and code benchmarks.
|
|
|
|
## Key Strengths
|
|
|
|
* **Strong Mathematical Reasoning** — Capable of solving complex, multi-step problems through sustained and coherent reasoning within a single forward pass. Achieves reliable results on challenging benchmarks including AIME 2026 I, HMMT, and IMO-Answer-Bench.
|
|
* **Strong Coding** — Achieves state-of-the-art results on LiveCodeBench-V6 and LiveCodeBench-Pro among models under 10B parameters.
|
|
* **Robust Preference Alignment** — Achieves solid alignment performance on Arena-Hard-v2 and Multi-Challenge, outperforming same-scale and substantially larger models.
|
|
|
|
## Benchmark Performance
|
|
|
|
### Math
|
|
|
|
| Benchmark | Qwen3-4B | Qwen3-8B | Qwen3-14B | Qwen3-32B | Qwen3-30B-A3B | **Eka-4B** |
|
|
|---|---|---|---|---|---|---|
|
|
| AIME 2026 I | 81.46 | 70.42 | 76.46 | 75.83 | 87.30 | **87.40** |
|
|
| HMMT Nov | 68.33 | 48.33 | 56.67 | 57.08 | 71.25 | **77.92** |
|
|
| IMO-Answer-Bench | 48.00 | 36.56 | 41.81 | 43.94 | **54.34** | 53.38 |
|
|
| GPQA | 65.8 | 62.0 | 63.38 | 68.4 | 73.4 | **83.8** |
|
|
| HLE (Text-only) | 6.72 | 5.28 | 7.00 | 9.31 | 11.77 | **12.60** |
|
|
|
|
### Code
|
|
|
|
| Benchmark | Qwen3-4B | Qwen3-8B | Qwen3-14B | Qwen3-32B | Qwen3-30B-A3B | **Eka-4B** |
|
|
|---|---|---|---|---|---|---|
|
|
| Live-Code-Bench-V6 | 57.4 | 49.4 | 55.9 | 55.7 | 66.0 | **76.9** |
|
|
| Live-Code-Bench-Pro-Easy | 40.2 | 41.2 | 33.0 | 42.3 | 60.8 | **81.4** |
|
|
| Live-Code-Bench-Pro-Medium | 5.3 | 3.5 | 1.8 | 3.5 | 3.5 | **28.1** |
|
|
|
|
### Alignment
|
|
|
|
| Benchmark | Qwen3-4B | Qwen3-8B | Qwen3-14B | Qwen3-32B | Qwen3-30B-A3B | **Eka-4B** |
|
|
|---|---|---|---|---|---|---|
|
|
| Arena-Hard-v2 | 34.9 | 26.3 | 36.9 | 56.0 | 60.2 | **73.2** |
|
|
| Multi-Challenge | 41.14 | 36.30 | 36.97 | 38.72 | 49.40 | **52.21** |
|
|
|
|
## Quickstart
|
|
|
|
### Recommended Inference Hyperparameters
|
|
|
|
| Parameter | Value |
|
|
|---|---|
|
|
| Temperature | 0.6 |
|
|
| Top-p | 0.95 |
|
|
| Repeat penalty | 1.0 |
|
|
| Max New Tokens | 131072 |
|
|
|
|
### Chat
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(
|
|
'yashmarathe/Eka-4B',
|
|
use_fast=False,
|
|
trust_remote_code=True,
|
|
)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
'yashmarathe/Eka-4B',
|
|
torch_dtype='auto',
|
|
device_map='auto',
|
|
trust_remote_code=True,
|
|
)
|
|
|
|
messages = [{'role': 'user', 'content': 'Solve: find all integer solutions to x^2 + y^2 = z^2 with x, y, z > 0 and x < 10.'}]
|
|
prompt = tokenizer.apply_chat_template(
|
|
messages,
|
|
add_generation_prompt=True,
|
|
tokenize=False,
|
|
)
|
|
input_ids = tokenizer(prompt, add_special_tokens=False, return_tensors='pt').input_ids
|
|
output_ids = model.generate(input_ids.to('cuda'), eos_token_id=166101)
|
|
response = tokenizer.decode(output_ids[0][len(input_ids[0]):], skip_special_tokens=True)
|
|
print(response)
|
|
```
|
|
|
|
### vLLM (Recommended for Production)
|
|
|
|
```bash
|
|
vllm serve yashmarathe/Eka-4B \
|
|
--port 8000 \
|
|
--max-model-len 131072 \
|
|
--gpu-memory-utilization 0.95 \
|
|
--enable-chunked-prefill \
|
|
--trust-remote-code
|
|
```
|
|
|
|
## Limitations
|
|
|
|
While significant effort has been made to align the model's outputs with ethical and legal requirements, the model may occasionally produce unexpected, biased, or otherwise problematic outputs due to its probabilistic nature. Users are responsible for evaluating outputs before deployment in production systems.
|
|
|
|
## License
|
|
|
|
Apache 2.0
|