---
license: apache-2.0
tags:
  - prismml
  - bonsai
  - awq
  - 4-bit
pipeline_tag: text-generation
base_model: prism-ml/Bonsai-8B-unpacked
library_name: transformers
---

# Bonsai-8B — AWQ 4-bit

## Summary

This repo provides an AWQ 4-bit checkpoint so you can run Bonsai-8B on [sglang](https://github.com/sgl-project/sglang) (or vLLM) until native 1-bit support lands in those engines. The repack from 1-bit to AWQ 4-bit is **lossless**. Both formats use group size 128, and Bonsai's binary weights (±d) fit exactly inside AWQ INT4 by an exact conversion formula.

> **For the best Bonsai experience on edge or consumer-grade hardware, use the native 1-bit releases.** The 1-bit format is where Bonsai's memory and energy wins come from.
>
> - **[Bonsai-8B MLX 1-bit](https://huggingface.co/prism-ml/Bonsai-8B-mlx-1bit)** — 1-bit MLX for Apple Silicon.
> - **[Bonsai-8B-gguf](https://huggingface.co/prism-ml/Bonsai-8B-gguf)** - 1-bit gguf supported by llama.cpp across many backends (GPU, Metal, CPU, Vulkan, etc)
> - **[Bonsai-8B FP16](https://huggingface.co/prism-ml/Bonsai-8B-unpacked)** — FP16 safetensors for stock HuggingFace tooling.

## How It Works

Bonsai weights are ±d (binary) with a shared scale across a group size of 128. INT4 can represent these values exactly. Embedding and `lm_head` stay FP16 due to sglang limitations.

AWQ INT4 dequantization: `weight = scale × (int4 − zero)`.

```
+d → scale=d, int4=9, zero=8 → d × (9-8) = +d
-d → scale=d, int4=7, zero=8 → d × (7-8) = -d
```

## Serve

```bash
pip install sglang

python -m sglang.launch_server \
    --model /path/to/Bonsai-8B-awq/ \
    --port 8000 \
    --dtype bfloat16
```

## Use

```bash
# Completion API
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Bonsai-8B","prompt":"The capital of France is","max_tokens":20}'

# Chat API
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Bonsai-8B","messages":[{"role":"user","content":"Who are you?"}],"max_tokens":100}'
```

## Multi-GPU (8× H100)

```bash
# Option A: DP=8 — 8 independent replicas, no inter-GPU comms
python -m sglang.launch_server \
    --model /path/to/Bonsai-8B-awq/ \
    --dp-size 8 \
    --load-balance-method total_tokens \
    --port 8000 --dtype bfloat16

# Option B: TP=2 DP=4 — 4 replicas, each split across 2 GPUs
python -m sglang.launch_server \
    --model /path/to/Bonsai-8B-awq/ \
    --tp-size 2 --dp-size 4 \
    --load-balance-method total_tokens \
    --port 8000 --dtype bfloat16

# Option C: TP=4 DP=2 — 2 replicas across 4 GPUs each
python -m sglang.launch_server \
    --model /path/to/Bonsai-8B-awq/ \
    --tp-size 4 --dp-size 2 \
    --load-balance-method total_tokens \
    --port 8000 --dtype bfloat16
```

## Appendix

### Launch time

With pre-built sgl-kernel wheels: 15 s. First launch on an arch without pre-built wheels (e.g. L40S / sm_89) takes 3–5 min while sglang JIT-compiles Marlin GEMM + FlashInfer kernels; artifacts cache to `~/.cache/tvm-ffi/` and `~/.cache/flashinfer/`, so subsequent launches drop back to 15 s.

### Known-good environment

Example of successful end-to-end serving environment:

- `sglang[all] == 0.5.9`
- `torch == 2.9.1`, `transformers == 4.57.1`, `triton == 3.5.1`
- `ninja == 1.13` on `PATH`
- `nvcc` from **CUDA 12.8** first on `PATH` (sglang's JIT Marlin needs `-std=c++20`; CUDA 11.x will fail)
- Python 3.12