jan-nano-4b-gptqmodel-4bit/README.md

license, tags, inference, library_name, model_creator, base_model

license	tags	inference	library_name	model_creator	base_model
apache-2.0	gptq quantization vllm text-generation transformer	false	vllm	menlo	Menlo/Jan-nano

Jan-nano GPTQ 4bit (vLLM-ready)

This is a 4-bit GPTQ quantized version of Menlo/Jan-nano, optimized for fast inference with vLLM.

• Quantization: GPTQ (4-bit)
• Group size: 128
• Dtype: float16
• Backend: gptqmodel
• Max context length: 4096 tokens

---

🔧 Usage with vLLM

vllm serve ./jan-nano-4b-gptqmodel-4bit \
  --quantization gptq \
  --dtype half \
  --max-model-len 4096

---

📁 Files

• Sharded .safetensors model weights
• model.safetensors.index.json
• tokenizer.json, tokenizer_config.json
• config.json, generation_config.json, quantize_config.json (if available)

---

🙏 Credits

• Original model by Menlo
• Quantized and shared by ramgpt