112 lines
4.7 KiB
Markdown
112 lines
4.7 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: gguf
|
||
base_model:
|
||
- chromadb/context-1
|
||
pipeline_tag: text-generation
|
||
language: en
|
||
tags:
|
||
- gguf
|
||
- llama.cpp
|
||
- gpt-oss
|
||
- chromadb
|
||
- chroma
|
||
- moe
|
||
- text-generation
|
||
- quantized
|
||
---
|
||
|
||
# Chroma Context-1 — GGUF (llama.cpp)
|
||
|
||
**GGUF weights for [Chroma Context-1](https://huggingface.co/chromadb/context-1),** converted for **[llama.cpp](https://github.com/ggml-org/llama.cpp)** and any runtime that loads GGUF (LM Studio, Ollama with compatible import paths, local servers, etc.).
|
||
|
||
This repository exists because **the upstream model is distributed in PyTorch / safetensors form only**. These files are the same weights in **GGUF**, with a range of **llama-quantize** presets so you can trade quality for VRAM and disk.
|
||
|
||
---
|
||
|
||
## Upstream (source of truth)
|
||
|
||
| | Link |
|
||
|---|------|
|
||
| **Original weights & model card** | [**`chromadb/context-1`**](https://huggingface.co/chromadb/context-1) |
|
||
| **Architecture family** | gpt-oss MoE (see upstream card; base traceable to OpenAI **[`gpt-oss-20b`](https://huggingface.co/openai/gpt-oss-20b)**) |
|
||
| **License** | **Apache 2.0** (unchanged; you must comply with upstream terms) |
|
||
|
||
**Attribution:** All tensors are derived from **[chromadb/context-1](https://huggingface.co/chromadb/context-1)**. This repo is a **community conversion** and is **not** affiliated with or endorsed by Chroma. For behavior, safety, and intended use, read the **official** model card first.
|
||
|
||
---
|
||
|
||
## Quick start
|
||
|
||
**1. Install** a recent [llama.cpp](https://github.com/ggml-org/llama.cpp) build (or use a GUI that bundles it).
|
||
|
||
**2. Download** this repository:
|
||
|
||
```bash
|
||
huggingface-cli download ryancook/chromadb-context-1-gguf --local-dir ./chromadb-context-1-gguf
|
||
```
|
||
|
||
**3. Run** (example — adjust paths and context length to your hardware):
|
||
|
||
```bash
|
||
llama-cli -m ./chromadb-context-1-gguf/chromadb-context-1-Q4_0.gguf -cnv --color -ngl 99
|
||
```
|
||
|
||
Swap the filename for any published `chromadb-context-1-*.gguf` from the **Files** tab (for example `Q4_K_M` or `MXFP4_MOE` when available).
|
||
|
||
---
|
||
|
||
## Choosing a file
|
||
|
||
**Start here (good defaults for most people):**
|
||
|
||
| Priority | File pattern | When to use |
|
||
|----------|----------------|-------------|
|
||
| 1 | **`…-Q4_K_M.gguf`** or **`…-Q5_K_M.gguf`** | Best general-purpose balance of quality and size (if present in this repo). |
|
||
| 2 | **`…-MXFP4_MOE.gguf`** | Smaller MoE-oriented layout; strong choice when supported by your llama.cpp build/GPU stack. |
|
||
| 3 | **`…-Q4_0.gguf`** / **`…-Q5_0.gguf`** | Simpler legacy-style quants; predictable tradeoffs. |
|
||
| 4 | **`…-bf16.gguf`** | Full **BF16** fidelity (~40 GiB class); for reference or maximum quality when you have RAM/VRAM. |
|
||
|
||
**Other presets** (IQ*, TQ*, Q2_K, Q3_K*, Q6_K, Q8_0, F16, …) may appear in the **Files** tab as they are published. Lower-bit and ternary formats are **experimental** for quality; profile on your workload before relying on them.
|
||
|
||
> **Tip:** The **Files and versions** view on Hugging Face is authoritative for what is available in each commit. Filenames follow `chromadb-context-1-<PRESET>.gguf`.
|
||
|
||
---
|
||
|
||
## Conversion pipeline
|
||
|
||
Reproducible high-level steps:
|
||
|
||
1. **Obtain** weights from [**chromadb/context-1**](https://huggingface.co/chromadb/context-1) (Apache 2.0).
|
||
2. **Convert** to GGUF with llama.cpp **`convert_hf_to_gguf.py`** (BF16 output from upstream bf16 checkpoint).
|
||
3. **Quantize** with **`llama-quantize`** using the preset named in each filename (`Q4_0`, `Q4_K_M`, `MXFP4_MOE`, etc.).
|
||
|
||
### Reproducibility
|
||
|
||
Conversions for this collection were produced with **[ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp)** at commit **`07ba6d275`** (short SHA; matches upstream `convert_hf_to_gguf.py` / `llama-quantize` from that tree). Newer llama.cpp revisions are generally backward compatible for GGUF loading, but you may see small numerical differences if you re-quantize.
|
||
|
||
---
|
||
|
||
## Hardware & context
|
||
|
||
- **VRAM / RAM:** MoE models route only a subset of experts per token; still treat published sizes as a guide and monitor peak usage at your target context length.
|
||
- **Context length:** Upstream supports a very long context window; practical limits depend on **KV cache size** and quant. Start with a smaller **`-c`** / context setting and increase only after you confirm stability.
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
Same as upstream: **Apache 2.0**. Keep **[chromadb/context-1](https://huggingface.co/chromadb/context-1)** attribution visible when you redistribute or ship products built on these files.
|
||
|
||
---
|
||
|
||
## More from Chroma
|
||
|
||
- **Official model (safetensors):** [chromadb/context-1](https://huggingface.co/chromadb/context-1)
|
||
- **Chroma:** [trychroma.com](https://www.trychroma.com/)
|
||
|
||
</think>
|
||
|
||
|
||
<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>
|
||
Shell |