121 lines
3.6 KiB
Markdown
121 lines
3.6 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
base_model: BSC-LT/salamandra-7b-instruct
|
||
pipeline_tag: text-generation
|
||
language:
|
||
- bg
|
||
- ca
|
||
- code
|
||
- cs
|
||
- cy
|
||
- da
|
||
- de
|
||
- el
|
||
- en
|
||
- es
|
||
- et
|
||
- eu
|
||
- fi
|
||
- fr
|
||
- ga
|
||
- gl
|
||
- hr
|
||
- hu
|
||
- it
|
||
- lt
|
||
- lv
|
||
- mt
|
||
- nl
|
||
- nn
|
||
- \no
|
||
- oc
|
||
- pl
|
||
- pt
|
||
- ro
|
||
- ru
|
||
- sh
|
||
- sk
|
||
- sl
|
||
- sr
|
||
- sv
|
||
- uk
|
||
---
|
||
|
||
|
||
|
||
# Salamandra-7b-instruct-gptq Model Card
|
||
|
||
This model is the gptq-quantized version of [Salamandra-7b-instruct](https://huggingface.co/BSC-LT/salamandra-7b-instruct) for speculative decoding.
|
||
|
||
The model weights are quantized from FP16 to W4A16 (4-bit weights and FP16 activations) using the [GPTQ](https://arxiv.org/abs/2210.17323) algorithm.
|
||
Inferencing with this model can be done using [VLLM](https://docs.vllm.ai/en/stable/models/engine_args.html).
|
||
|
||
Salamandra is a highly multilingual model pre-trained from scratch that comes in three different
|
||
sizes — 2B, 7B and 40B parameters — with their respective base and instruction-tuned variants,
|
||
promoted and financed by the Government of Catalonia through the [Aina Project](https://projecteaina.cat/)
|
||
and the _Ministerio para la Transformación Digital y de la Función Pública_ - Funded by EU – NextGenerationEU
|
||
within the framework of [ILENIA Project](https://proyectoilenia.es/) with reference 2022/TL22/00215337.
|
||
|
||
This model card corresponds to the gptq-quantized version of Salamandra-7b-instruct for speculative decoding.
|
||
|
||
The entire Salamandra family is released under a permissive [Apache 2.0 license]((https://www.apache.org/licenses/LICENSE-2.0)).
|
||
|
||
|
||
## How to Use
|
||
|
||
The following example code works under ``Python 3.9.16``, ``vllm==0.6.3.post1``, ``torch==2.4.0`` and ``torchvision==0.19.0``, though it should run on
|
||
any current version of the libraries. This is an example of a conversational chatbot using the model:
|
||
|
||
```
|
||
from vllm import LLM, SamplingParams
|
||
|
||
model_name = "BSC-LT/salamandra-7b-instruct-gptq"
|
||
llm = LLM(model=model_name)
|
||
|
||
messages = []
|
||
|
||
while True:
|
||
user_input = input("user >> ")
|
||
if user_input.lower() == "exit":
|
||
print("Chat ended.")
|
||
break
|
||
|
||
messages.append({'role': 'user', 'content': user_input})
|
||
|
||
outputs = llm.chat(messages,
|
||
sampling_params=SamplingParams(
|
||
temperature=0.5,
|
||
stop_token_ids=[5],
|
||
max_tokens=200)
|
||
)[0].outputs
|
||
|
||
model_output = outputs[0].text
|
||
print(f'assistant >> {model_output}')
|
||
messages.append({'role': 'assistant', 'content': model_output})
|
||
```
|
||
|
||
### Author
|
||
International Business Machines (IBM).
|
||
|
||
### Copyright
|
||
International Business Machines (IBM).
|
||
|
||
### Contact
|
||
For further information, please send an email to <langtech@bsc.es>.
|
||
|
||
### Acknowledgements
|
||
We appreciate the collaboration with IBM in this work.
|
||
Specifically, the IBM team created gptq-quantized version of the Salamandra-7b-instruct model for speculative decoding released here.
|
||
|
||
### Disclaimer
|
||
Be aware that the model may contain biases or other unintended distortions.
|
||
When third parties deploy systems or provide services based on this model, or use the model themselves,
|
||
they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable
|
||
regulations, including those governing the use of Artificial Intelligence.
|
||
|
||
Barcelona Supercomputing Center and International Business Machines shall
|
||
not be held liable for any outcomes resulting from third-party use.
|
||
|
||
### License
|
||
[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0) |