chinese-llama-2-13b-16k-gguf/README.md

inference, language, license, model_creator, model_link, model_name, model_type, pipeline_tag, quantized_by, tasks, tags

inference	language	license	model_creator	model_link	model_name	model_type	pipeline_tag	quantized_by	tasks	tags
false	zh	other	ziqingyang	https://huggingface.co/ziqingyang/chinese-llama-2-13b-16k	chinese-llama-2-13b-16k	llama	text-generation	shaowenchen	text2text-generation	meta gguf llama llama-2 chinese

Provided files

Name	Quant method	Size
chinese-llama-2-13b-16k.Q2_K.gguf	Q2_K	5.2 G
chinese-llama-2-13b-16k.Q3_K.gguf	Q3_K	6.0 G
chinese-llama-2-13b-16k.Q3_K_L.gguf	Q3_K_L	6.6 G
chinese-llama-2-13b-16k.Q3_K_S.gguf	Q3_K_S	5.4 G
chinese-llama-2-13b-16k.Q4_0.gguf	Q4_0	7.0 G
chinese-llama-2-13b-16k.Q4_1.gguf	Q4_1	7.8 G
chinese-llama-2-13b-16k.Q4_K.gguf	Q4_K	7.5 G
chinese-llama-2-13b-16k.Q4_K_S.gguf	Q4_K_S	7.1 G
chinese-llama-2-13b-16k.Q5_0.gguf	Q5_0	8.5 G
chinese-llama-2-13b-16k.Q5_1.gguf	Q5_1	9.3 G
chinese-llama-2-13b-16k.Q5_K.gguf	Q5_K	8.8 G
chinese-llama-2-13b-16k.Q5_K_S.gguf	Q5_K_S	8.5 G
chinese-llama-2-13b-16k.Q6_K.gguf	Q6_K	10 G
chinese-llama-2-13b-16k.Q8_0.gguf	Q8_0	13 G
chinese-llama-2-13b-16k.gguf	full	25 G

Usage:

docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/gguf-model-name.gguf hubimage/llama-cpp-python:latest

and you can view http://localhost:8000/docs to see the swagger UI.