license, datasets, library_name, tags, base_model, model-index

license	datasets	library_name	tags	base_model	model-index
apache-2.0	bigcode/the-stack bigcode/the-stack-v2 bigcode/starcoderdata bigcode/commitpack	transformers	code	JetBrains/Mellum-4b-base	name results Mellum-4b-sft-all task dataset metrics type text-generation type name tianyang/repobench_python_v1.1 RepoBench 1.1 (Python) name type value verified EM exact_match 0.2823 false name type value verified EM ≤ 8k exact_match 0.2870 false task dataset metrics type text-generation type name tianyang/repobench_python_v1.1 RepoBench 1.1 (Python, 2k) name type value verified EM exact_match 0.2638 false task dataset metrics type text-generation type name tianyang/repobench_python_v1.1 RepoBench 1.1 (Python, 4k) name type value verified EM exact_match 0.2930 false task dataset metrics type text-generation type name tianyang/repobench_python_v1.1 RepoBench 1.1 (Python, 8k) name type value verified EM exact_match 0.3042 false task dataset metrics type text-generation type name tianyang/repobench_python_v1.1 RepoBench 1.1 (Python, 12k) name type value verified EM exact_match 0.2685 false task dataset metrics type text-generation type name tianyang/repobench_python_v1.1 RepoBench 1.1 (Python, 16k) name type value verified EM exact_match 0.2818 false task dataset metrics type text-generation type name tianyang/repobench_java_v1.1 RepoBench 1.1 (Java) name type value verified EM exact_match 0.2867 false name type value verified EM ≤ 8k exact_match 0.3023 false task dataset metrics type text-generation type name tianyang/repobench_java_v1.1 RepoBench 1.1 (Java, 2k) name type value verified EM exact_match 0.2883 false task dataset metrics type text-generation type name tianyang/repobench_java_v1.1 RepoBench 1.1 (Java, 4k) name type value verified EM exact_match 0.3228 false task dataset metrics type text-generation type name tianyang/repobench_java_v1.1 RepoBench 1.1 (Java, 8k) name type value verified EM exact_match 0.2958 false task dataset metrics type text-generation type name tianyang/repobench_java_v1.1 RepoBench 1.1 (Java, 12k) name type value verified EM exact_match 0.2447 false task dataset metrics type text-generation type name tianyang/repobench_java_v1.1 RepoBench 1.1 (Java, 16k) name type value verified EM exact_match 0.2821 false task dataset metrics type text-generation type name gonglinyuan/safim SAFIM name type value verified pass@1 pass@1 0.5285 false task dataset metrics type text-generation type name gonglinyuan/safim SAFIM (API) name type value verified pass@1 pass@1 0.6548 false task dataset metrics type text-generation type name gonglinyuan/safim SAFIM (Block) name type value verified pass@1 pass@1 0.4005 false task dataset metrics type text-generation type name gonglinyuan/safim SAFIM (Control) name type value verified pass@1 pass@1 0.5303 false task dataset metrics type text-generation type name loubnabnl/humaneval_infilling HumanEval Infilling (Single-Line) name type value verified pass@1 pass@1 0.8083 false task dataset metrics type text-generation type name loubnabnl/humaneval_infilling HumanEval Infilling (Multi-Line) name type value verified pass@1 pass@1 0.4819 false task dataset metrics type text-generation type name loubnabnl/humaneval_infilling HumanEval Infilling (Random Span) name type value verified pass@1 pass@1 0.3720 false task dataset metrics type text-generation type name loubnabnl/humaneval_infilling HumanEval Infilling (Random Span Light) name type value verified pass@1 pass@1 0.4024 false

Model Description

Mellum-4b-sft-all is a fine-tuned version of JetBrains' first open-source large language model (LLM) optimized for code-related tasks.

Pre-trained on over 4 trillion tokens with a context window of 8192 tokens across multiple programming languages, and then fine-tuned, Mellum-4b-sft-all is tailored for context-aware code completion tasks. It was fine-tuned on a diverse set of languages, including Batchfile, C, C#, CMake, C++, CSS, Cython, Dockerfile, F#, Go, Groovy, HCL, HTML (and variants like Django, EEx, ERB, and PHP templates), Java, JSP, JavaScript, JSX, Kotlin, Less, Makefile, Objective-C++, PHP, PowerShell, Python, R, RHTML, Ruby, Rust, Sass, Scala, SCSS, Shell, SQL, Swift, TOML, TypeScript, Visual Basic, Vue, and YAML. The model follows a LLaMA-style architecture with 4 billion parameters, making it efficient for both cloud inference (e.g., via vLLM) and local deployment (e.g., using llama.cpp or Ollama).

Mellum was trained using Automatic Mixed Precision (AMP) with bf16 precision. The uploaded version on Hugging Face retains the bf16 format for public use.

Designed for integration into professional developer tooling (e.g., intelligent code suggestions in IDEs), AI-powered coding assistants, and research on code understanding and generation, Mellum is also well-suited for educational applications and fine-tuning experiments.

Limitations

• Biases: May reflect biases present in public codebases. For example it will likely produce code which is similar in style to the open-source repositories.
• Security: Code suggestions should not be assumed to be secure or free of vulnerabilities.

Sample Usage

Here is an example of how to run and sample from the model with additional files context and fill in the middle.

Fill in the middle with additional files as context generation

import json
from transformers import AutoTokenizer, AutoModelForCausalLM

example = """
<filename>Utils.kt
package utils

fun multiply(x: Int, y: Int): Int {
    return x * y
}

<filename>Config.kt
package config

object Config {
    const val DEBUG = true
    const val MAX_VALUE = 100
}

<filename>Example.kt
<fim_suffix>
fun main() {
    val result = calculateSum(5, 10)
    println(result)
}
<fim_prefix>fun calculateSum(a: Int, b: Int): Int {
<fim_middle>
"""

tokenizer = AutoTokenizer.from_pretrained('JetBrains/Mellum-4b-sft-all')
model = AutoModelForCausalLM.from_pretrained('JetBrains/Mellum-4b-sft-all')
encoded_input = tokenizer(example, return_tensors='pt', return_token_type_ids=False)
out = model.generate(
    **encoded_input,
    max_new_tokens=100,
)

Benchmarks

We are providing scores for Mellum‑4b‑sft‑all to give users an estimate of the model’s potential capabilities.

RepoBench 1.1

Type: single‑line  Languages: Python and Java  Metric: Exact Match (EM), %

Since Mellum has a maximum context window of 8 k, we report both the average over all evaluated context lengths (2 k, 4 k, 8 k, 12 k and 16 k) and the average over the lengths within its supported range (≤ 8 k).

Python subset

Model	2 k	4 k	8 k	12 k	16 k	Avg	Avg ≤ 8 k
Mellum‑4b‑sft‑all	26.38%	29.30%	30.42%	26.85%	28.18%	28.23%	28.70%

Java subset

Model	2 k	4 k	8 k	12 k	16 k	Avg	Avg ≤ 8 k
Mellum‑4b‑sft‑all	28.83%	32.28%	29.58%	24.47%	28.21%	28.67%	30.23%

Syntax‑Aware Fill‑in‑the‑Middle (SAFIM)

Type: mix of multi‑line and single‑line  Languages: multi‑language  Metric: pass@1, %

Model	Algorithmic	Control	API	Average
Mellum‑4b‑sft‑all	40.05%	53.03%	65.48%	52.85%

HumanEval Infilling

• Type: single‑line and multi‑line
• Languages: Python
• Metric: pass@1, %

Model	Single‑Line	Multi‑Line	Random Span
Mellum‑4b‑sft‑all	80.83%	48.19%	37.20%

Citation

If you use this model, please cite:

@misc{Mellum-4b-base,
  title     = {Mellum-4b-base},
  author    = {Pavlichenko, Nikita and Nazarov, Iurii and Dolgov, Ivan and Garanina, Ekaterina and Lasocki, Karol and Reshetnikova, Julia and Boitsov, Sergei and Bondyrev, Ivan and Karaeva, Dariia and Sheptyakov, Maksim and Ustalov, Dmitry and Mukhin, Artem and Proshev, Semyon and Abramov, Nikita and Kolomyttseva, Olga and Lysaniuk, Kseniia and Zavidnyi, Ilia and Semenkin, Anton and Tankov, Vladislav and Sazanovich, Uladzislau},
  year      = {2025},
}

Contact

For questions, collaborations and requests reach us out via mellum@jetbrains.com