97 lines
4.3 KiB
Markdown
97 lines
4.3 KiB
Markdown
---
|
||
license: mit
|
||
base_model:
|
||
- meta-llama/Llama-3.3-70B-Instruct
|
||
language:
|
||
- ja
|
||
- en
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- llama-3
|
||
- pytorch
|
||
- llama-3.3
|
||
- autoawq
|
||
- meta
|
||
---
|
||
|
||
# kishizaki-sci/Llama-3.3-70B-Instruct-AWQ-4bit-JP-EN
|
||
|
||
## model information
|
||
[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)を[AutoAWQ](https://github.com/casper-hansen/AutoAWQ)で4bit 量子化したモデル。量子化の際のキャリブレーションデータに日本語と英語を含むデータを使用。
|
||
A model of Llama-3.3-70B-Instruct quantized to 4 bits using AutoAWQ. Calibration data containing Japanese and English was used during the quantization process.
|
||
|
||
## usage
|
||
### transformers
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained("kishizaki-sci/Llama-3.3-70B-Instruct-AWQ-4bit-JP-EN")
|
||
model = AutoModelForCausalLM.from_pretrained("kishizaki-sci/Llama-3.3-70B-Instruct-AWQ-4bit-JP-EN")
|
||
model.to("cuda")
|
||
|
||
chat = [
|
||
{"role": "system", "content": "あなたは日本語で応答するAIチャットボットです。ユーザをサポートしてください。"},
|
||
{"role": "user", "content": "plotly.graph_objectsを使って散布図を作るサンプルコードを書いてください。"}
|
||
]
|
||
prompt = tokenizer.apply_chat_template(
|
||
chat,
|
||
tokenize=False,
|
||
add_generation_prompt=True
|
||
)
|
||
inputs = tokenizer(prompt, return_tensors="pt")
|
||
inputs = inputs.to("cuda")
|
||
streamer = TextStreamer(tokenizer)
|
||
|
||
output = model.generate(**inputs, streamer=streamer, max_new_tokens=1024)
|
||
```
|
||
このコードはA100インスタンスの[Google Colab](https://colab.research.google.com/drive/1NSWV6MMisTlN5ELN42VPMLRH9ahAfNrJ?usp=sharing) でも動かせます。
|
||
This code can also run on Google Colab with an A100 instance.
|
||
|
||
### vLLM
|
||
```python
|
||
from vllm import LLM, SamplingParams
|
||
|
||
llm = LLM(
|
||
model="kishizaki-sci/Llama-3.3-70B-Instruct-AWQ-4bit-JP-EN",
|
||
tensor_parallel_size=1,
|
||
gpu_memory_utilization=0.97,
|
||
quantization="awq"
|
||
)
|
||
tokenizer = llm.get_tokenizer()
|
||
|
||
messages = [
|
||
{"role": "system", "content": "あなたは日本語で応答するAIチャットボットです。ユーザをサポートしてください。"},
|
||
{"role": "user", "content": "plotly.graph_objectsを使って散布図を作るサンプルコードを書いてください。"},
|
||
]
|
||
|
||
prompt = tokenizer.apply_chat_template(
|
||
messages,
|
||
tokenize=False,
|
||
add_generation_prompt=True
|
||
)
|
||
|
||
sampling_params = SamplingParams(
|
||
temperature=0.6,
|
||
top_p=0.9,
|
||
max_tokens=1024
|
||
)
|
||
|
||
outputs = llm.generate(prompt, sampling_params)
|
||
print(outputs[0].outputs[0].text)
|
||
```
|
||
H100 (94GB)を1基積んだインスタンスでの実行はこちらの[notebook](https://huggingface.co/kishizaki-sci/Llama-3.3-70B-Instruct-AWQ-4bit-JP-EN/blob/main/inference_vLLM.ipynb)をご覧ください。
|
||
Please refer to this notebook for execution on an instance equipped with a single H100 (94GB).
|
||
|
||
## calibration data
|
||
以下のデータセットから512個のデータ，プロンプトを抽出。1つのデータのトークン数は最大350制限。
|
||
Extract 512 data points and prompts from the following dataset. The maximum token limit per data point is 350.
|
||
- [TFMC/imatrix-dataset-for-japanese-llm](https://huggingface.co/datasets/TFMC/imatrix-dataset-for-japanese-llm)
|
||
- [meta-math/MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA)
|
||
- [m-a-p/CodeFeedback-Filtered-Instruction](https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction)
|
||
- [kunishou/databricks-dolly-15k-ja](https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja)
|
||
- その他日本語版・英語版のwikipedia記事から作成したオリジナルデータ，有害プロンプト回避のためのオリジナルデータを使用。 Original data created from Japanese and English Wikipedia articles, as well as original data for avoiding harmful prompts, is used.
|
||
|
||
## License
|
||
[MIT License](https://opensource.org/license/mit)を適用する。ただし量子化のベースモデルに適用されている[Llama 3.3 Community License Agreement](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/LICENSE)に従ってください。
|
||
The MIT License is applied. However, obey the Llama 3.3 Community License Agreement applied to the base model of quantization. |