+
+TheBloke's LLM work is generously supported by a grant from andreessen horowitz (a16z)
如果您是本模型的贡献者，我们邀请您根据模型贡献文档，及时完善模型卡片内容。
\ No newline at end of file +Example Python code for interfacing with TGI (requires huggingface-hub 0.17.0 or later): + +```shell +pip3 install huggingface-hub +``` + +```python +from huggingface_hub import InferenceClient + +endpoint_url = "https://your-endpoint-url-here" + +prompt = "Tell me about AI" +prompt_template=f'''A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {prompt} ASSISTANT: + +''' + +client = InferenceClient(endpoint_url) +response = client.text_generation(prompt, + max_new_tokens=128, + do_sample=True, + temperature=0.7, + top_p=0.95, + top_k=40, + repetition_penalty=1.1) + +print(f"Model output: {response}") +``` + + + +## How to use this AWQ model from Python code + +### Install the necessary packages + +Requires: [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) 0.1.1 or later + +```shell +pip3 install autoawq +``` + +If you have problems installing [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) using the pre-built wheels, install it from source instead: + +```shell +pip3 uninstall -y autoawq +git clone https://github.com/casper-hansen/AutoAWQ +cd AutoAWQ +pip3 install . +``` + +### You can then try the following example code + +```python +from awq import AutoAWQForCausalLM +from transformers import AutoTokenizer + +model_name_or_path = "TheBloke/Xwin-LM-13B-v0.2-AWQ" + +# Load model +model = AutoAWQForCausalLM.from_quantized(model_name_or_path, fuse_layers=True, + trust_remote_code=False, safetensors=True) +tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=False) + +prompt = "Tell me about AI" +prompt_template=f'''A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {prompt} ASSISTANT: + +''' + +print("\n\n*** Generate:") + +tokens = tokenizer( + prompt_template, + return_tensors='pt' +).input_ids.cuda() + +# Generate output +generation_output = model.generate( + tokens, + do_sample=True, + temperature=0.7, + top_p=0.95, + top_k=40, + max_new_tokens=512 +) + +print("Output: ", tokenizer.decode(generation_output[0])) + +""" +# Inference should be possible with transformers pipeline as well in future +# But currently this is not yet supported by AutoAWQ (correct as of September 25th 2023) +from transformers import pipeline + +print("*** Pipeline:") +pipe = pipeline( + "text-generation", + model=model, + tokenizer=tokenizer, + max_new_tokens=512, + do_sample=True, + temperature=0.7, + top_p=0.95, + top_k=40, + repetition_penalty=1.1 +) + +print(pipe(prompt_template)[0]['generated_text']) +""" +``` + + + +## Compatibility + +The files provided are tested to work with: + +- [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) +- [vLLM](https://github.com/vllm-project/vllm) +- [Huggingface Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference) + +TGI merged AWQ support on September 25th, 2023: [TGI PR #1054](https://github.com/huggingface/text-generation-inference/pull/1054). Use the `:latest` Docker container until the next TGI release is made. + + + + + +## Discord + +For further support, and discussions on these models and AI in general, join us at: + +[TheBloke AI's Discord server](https://discord.gg/theblokeai) + +## Thanks, and how to contribute + +Thanks to the [chirper.ai](https://chirper.ai) team! + +Thanks to Clay from [gpus.llm-utils.org](llm-utils)! + +I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training. + +If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects. + +Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits. + +* Patreon: https://patreon.com/TheBlokeAI +* Ko-Fi: https://ko-fi.com/TheBlokeAI + +**Special thanks to**: Aemon Algiz. + +**Patreon special mentions**: Pierre Kircher, Stanislav Ovsiannikov, Michael Levine, Eugene Pentland, Andrey, 준교 김, Randy H, Fred von Graf, Artur Olbinski, Caitlyn Gatomon, terasurfer, Jeff Scroggin, James Bentley, Vadim, Gabriel Puliatti, Harry Royden McLaughlin, Sean Connelly, Dan Guido, Edmond Seymore, Alicia Loh, subjectnull, AzureBlack, Manuel Alberto Morcote, Thomas Belote, Lone Striker, Chris Smitley, Vitor Caleffi, Johann-Peter Hartmann, Clay Pascal, biorpg, Brandon Frisco, sidney chen, transmissions 11, Pedro Madruga, jinyuan sun, Ajan Kanaga, Emad Mostaque, Trenton Dambrowitz, Jonathan Leane, Iucharbius, usrbinkat, vamX, George Stoitzev, Luke Pendergrass, theTransient, Olakabola, Swaroop Kallakuri, Cap'n Zoog, Brandon Phillips, Michael Dempsey, Nikolai Manek, danny, Matthew Berman, Gabriel Tamborski, alfie_i, Raymond Fosdick, Tom X Nguyen, Raven Klaugh, LangChain4j, Magnesian, Illia Dulskyi, David Ziegler, Mano Prime, Luis Javier Navarrete Lozano, Erik Bjäreholt, 阿明, Nathan Dryer, Alex, Rainer Wilmers, zynix, TL, Joseph William Delisle, John Villwock, Nathan LeClaire, Willem Michiel, Joguhyik, GodLy, OG, Alps Aficionado, Jeffrey Morgan, ReadyPlayerEmma, Tiffany J. Kim, Sebastain Graf, Spencer Kim, Michael Davis, webtim, Talal Aujan, knownsqashed, John Detwiler, Imad Khwaja, Deo Leter, Jerry Meng, Elijah Stavena, Rooh Singh, Pieter, SuperWojo, Alexandros Triantafyllidis, Stephen Murray, Ai Maven, ya boyyy, Enrico Ros, Ken Nordquist, Deep Realms, Nicholas, Spiking Neurons AB, Elle, Will Dee, Jack West, RoA, Luke @flexchar, Viktor Bowallius, Derek Yates, Subspace Studios, jjj, Toran Billups, Asp the Wyvern, Fen Risland, Ilya, NimbleBox.ai, Chadd, Nitin Borwankar, Emre, Mandus, Leonard Tan, Kalila, K, Trailburnt, S_X, Cory Kujawski + + +Thank you to all my generous patrons and donaters! + +And thank you again to a16z for their generous grant. + + + +# Original model card: Xwin-LM's Xwin LM 13B v0.2 + + +
+ 
