Jan-v3-4B-base-instruct-GGUF/README.md

base_model, library_name, pipeline_tag, tags

base_model	library_name	pipeline_tag	tags
janhq/Jan-v3-4B-base-instruct	gguf	text-generation	gguf quantized llama-cpp

Jan-v3-4B-base-instruct - GGUF

This is a quantized GGUF version of janhq/Jan-v3-4B-base-instruct created using llama.cpp.

Available Quantizations

Filename	Quant Type	Description
Jan-v3-4B-base-instruct.Q2_K.gguf	Q2_K	Smallest, significant quality loss
Jan-v3-4B-base-instruct.Q3_K_S.gguf	Q3_K_S	Very small, low quality
Jan-v3-4B-base-instruct.Q3_K_M.gguf	Q3_K_M	Very small, medium quality
Jan-v3-4B-base-instruct.Q3_K_L.gguf	Q3_K_L	Small, better quality than Q3_K_M
Jan-v3-4B-base-instruct.Q4_0.gguf	Q4_0	Small, legacy format
Jan-v3-4B-base-instruct.Q4_1.gguf	Q4_1	Small, legacy format with better accuracy
Jan-v3-4B-base-instruct.Q4_K_S.gguf	Q4_K_S	Small, good quality
Jan-v3-4B-base-instruct.Q4_K_M.gguf	Q4_K_M	Medium, balanced quality - recommended
Jan-v3-4B-base-instruct.Q5_0.gguf	Q5_0	Medium, legacy format
Jan-v3-4B-base-instruct.Q5_1.gguf	Q5_1	Medium, legacy format with better accuracy
Jan-v3-4B-base-instruct.Q5_K_S.gguf	Q5_K_S	Medium, good quality
Jan-v3-4B-base-instruct.Q5_K_M.gguf	Q5_K_M	Medium, high quality - recommended
Jan-v3-4B-base-instruct.Q6_K.gguf	Q6_K	Large, very high quality
Jan-v3-4B-base-instruct.Q8_0.gguf	Q8_0	Large, near-lossless quality

Usage

With llama.cpp

./llama-cli -m Jan-v3-4B-base-instruct.Q4_K_M.gguf -p "Your prompt here"

With Ollama

ollama run hf.co/aashish1904/Jan-v3-4B-base-instruct-GGUF

Original Model

• Source: janhq/Jan-v3-4B-base-instruct
• Quantized by: GGUF Quantizer Space

---

Original Model Card

Jan-v3-4B-base-instruct: a 4B baseline model for fine-tuning

Overview

Jan-v3-4B-base-instruct is a 4B-parameter model obtained via post-training distillation from a larger teacher, transferring capabilities while preserving general-purpose performance on standard benchmarks. The result is a compact, ownable base that is straightforward to fine-tune, broadly applicable and minimizing the usual capacity–capability trade-offs.

Building on this base, Jan-Code, a code-tuned variant, will be released soon.

Model Overview

This repo contains the BF16 version of Jan-v3-4B-base-instruct, which has the following features:

• Type: Causal Language Models
• Training Stage: Pretraining & Post-training
• Number of Parameters: 4B in total
• Number of Layers: 36
• Number of Attention Heads (GQA): 32 for Q and 8 for KV
• Context Length: 262,144 natively.

Intended Use

• A better small base for downstream work: improved instruction following out of the box, strong starting point for fine-tuning, and effective lightweight coding assistance.

Performance

Quick Start

Integration with Jan Apps

Jan-v3 demo is hosted on Jan Browser at chat.jan.ai. It is also optimized for direct integration with Jan Desktop, select the model in the app to start using it.

Local Deployment

Using vLLM:

vllm serve janhq/Jan-v3-4B-base-instruct \
    --host 0.0.0.0 \
    --port 1234 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes 
    

Using llama.cpp:

llama-server --model Jan-v3-4B-base-instruct-Q8_0.gguf \
    --host 0.0.0.0 \
    --port 1234 \
    --jinja \
    --no-context-shift

Recommended Parameters

For optimal performance in agentic and general tasks, we recommend the following inference parameters:

temperature: 0.7
top_p: 0.8
top_k: 20

🤝 Community & Support

• Discussions: Hugging Face Community
• Jan App: Learn more about the Jan App at jan.ai

📄 Citation

Updated Soon