longchat-13b-16k-gguf/README.md

inference, language, license, model_creator, model_link, model_name, model_type, pipeline_tag, quantized_by, tasks, tags

inference	language	license	model_creator	model_link	model_name	model_type	pipeline_tag	quantized_by	tasks	tags
true	zh en	other	lmsys	https://huggingface.co/lmsys/longchat-13b-16k	longchat-13b-16k	longchat	text-generation	shaowenchen	text2text-generation	gguf longchat 13b 16k

Provided files

Name	Quant method	Size
longchat-13b-16k.Q2_K.gguf	Q2_K	5.1 GB
longchat-13b-16k.Q3_K.gguf	Q3_K	5.9 GB
longchat-13b-16k.Q3_K_L.gguf	Q3_K_L	6.5 GB
longchat-13b-16k.Q3_K_S.gguf	Q3_K_S	5.3 GB
longchat-13b-16k.Q4_0.gguf	Q4_0	6.9 GB
longchat-13b-16k.Q4_1.gguf	Q4_1	7.6 GB
longchat-13b-16k.Q4_K.gguf	Q4_K	7.3 GB
longchat-13b-16k.Q4_K_S.gguf	Q4_K_S	6.9 GB
longchat-13b-16k.Q5_0.gguf	Q5_0	8.4 GB
longchat-13b-16k.Q5_1.gguf	Q5_1	9.1 GB
longchat-13b-16k.Q5_K.gguf	Q5_K	8.6 GB
longchat-13b-16k.Q5_K_S.gguf	Q5_K_S	8.4 GB
longchat-13b-16k.Q6_K.gguf	Q6_K	9.9 GB
longchat-13b-16k.Q8_0.gguf	Q8_0	13 GB
longchat-13b-16k.gguf	full	24 GB

Usage:

docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/gguf-model-name.gguf hubimage/llama-cpp-python:latest

and you can view http://localhost:8000/docs to see the swagger UI.

Provided images

Name	Quant method	Compressed Size
shaowenchen/longchat-13b-16k-gguf:Q2_K	Q2_K	7.47 GB
shaowenchen/longchat-13b-16k-gguf:Q3_K	Q3_K	6.11 GB
shaowenchen/longchat-13b-16k-gguf:Q4_K	Q4_K	5.29 GB

Usage:

docker run --rm -p 8000:8000 shaowenchen/longchat-13b-16k-gguf:Q2_K

and you can view http://localhost:8000/docs to see the swagger UI.