97 lines
4.3 KiB
Markdown
97 lines
4.3 KiB
Markdown
|
---
|
|||
|
|
license: mit
|
|||
|
|
base_model:
|
|||
|
|
- meta-llama/Llama-3.3-70B-Instruct
|
|||
|
|
language:
|
|||
|
|
- ja
|
|||
|
|
- en
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
library_name: transformers
|
|||
|
|
tags:
|
|||
|
|
- llama-3
|
|||
|
|
- pytorch
|
|||
|
|
- llama-3.3
|
|||
|
|
- autoawq
|
|||
|
|
- meta
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# kishizaki-sci/Llama-3.3-70B-Instruct-AWQ-4bit-JP-EN
|
|||
|
|
|
|||
|
|
## model information
|
|||
|
|
[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)を[AutoAWQ](https://github.com/casper-hansen/AutoAWQ)で4bit 量子化したモデル。量子化の際のキャリブレーションデータに日本語と英語を含むデータを使用。
|
|||
|
|
A model of Llama-3.3-70B-Instruct quantized to 4 bits using AutoAWQ. Calibration data containing Japanese and English was used during the quantization process.
|
|||
|
|
|
|||
|
|
## usage
|
|||
|
|
### transformers
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
|
|||
|
|
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained("kishizaki-sci/Llama-3.3-70B-Instruct-AWQ-4bit-JP-EN")
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained("kishizaki-sci/Llama-3.3-70B-Instruct-AWQ-4bit-JP-EN")
|
|||
|
|
model.to("cuda")
|
|||
|
|
|
|||
|
|
chat = [
|
|||
|
|
{"role": "system", "content": "あなたは日本語で応答するAIチャットボットです。ユーザをサポートしてください。"},
|
|||
|
|
{"role": "user", "content": "plotly.graph_objectsを使って散布図を作るサンプルコードを書いてください。"}
|
|||
|
|
]
|
|||
|
|
prompt = tokenizer.apply_chat_template(
|
|||
|
|
chat,
|
|||
|
|
tokenize=False,
|
|||
|
|
add_generation_prompt=True
|
|||
|
|
)
|
|||
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
|||
|
|
inputs = inputs.to("cuda")
|
|||
|
|
streamer = TextStreamer(tokenizer)
|
|||
|
|
|
|||
|
|
output = model.generate(**inputs, streamer=streamer, max_new_tokens=1024)
|
|||
|
|
```
|
|||
|
|
このコードはA100インスタンスの[Google Colab](https://colab.research.google.com/drive/1NSWV6MMisTlN5ELN42VPMLRH9ahAfNrJ?usp=sharing) でも動かせます。
|
|||
|
|
This code can also run on Google Colab with an A100 instance.
|
|||
|
|
|
|||
|
|
### vLLM
|
|||
|
|
```python
|
|||
|
|
from vllm import LLM, SamplingParams
|
|||
|
|
|
|||
|
|
llm = LLM(
|
|||
|
|
model="kishizaki-sci/Llama-3.3-70B-Instruct-AWQ-4bit-JP-EN",
|
|||
|
|
tensor_parallel_size=1,
|
|||
|
|
gpu_memory_utilization=0.97,
|
|||
|
|
quantization="awq"
|
|||
|
|
)
|
|||
|
|
tokenizer = llm.get_tokenizer()
|
|||
|
|
|
|||
|
|
messages = [
|
|||
|
|
{"role": "system", "content": "あなたは日本語で応答するAIチャットボットです。ユーザをサポートしてください。"},
|
|||
|
|
{"role": "user", "content": "plotly.graph_objectsを使って散布図を作るサンプルコードを書いてください。"},
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
prompt = tokenizer.apply_chat_template(
|
|||
|
|
messages,
|
|||
|
|
tokenize=False,
|
|||
|
|
add_generation_prompt=True
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
sampling_params = SamplingParams(
|
|||
|
|
temperature=0.6,
|
|||
|
|
top_p=0.9,
|
|||
|
|
max_tokens=1024
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
outputs = llm.generate(prompt, sampling_params)
|
|||
|
|
print(outputs[0].outputs[0].text)
|
|||
|
|
```
|
|||
|
|
H100 (94GB)を1基積んだインスタンスでの実行はこちらの[notebook](https://huggingface.co/kishizaki-sci/Llama-3.3-70B-Instruct-AWQ-4bit-JP-EN/blob/main/inference_vLLM.ipynb)をご覧ください。
|
|||
|
|
Please refer to this notebook for execution on an instance equipped with a single H100 (94GB).
|
|||
|
|
|
|||
|
|
## calibration data
|
|||
|
|
以下のデータセットから512個のデータ，プロンプトを抽出。1つのデータのトークン数は最大350制限。
|
|||
|
|
Extract 512 data points and prompts from the following dataset. The maximum token limit per data point is 350.
|
|||
|
|
- [TFMC/imatrix-dataset-for-japanese-llm](https://huggingface.co/datasets/TFMC/imatrix-dataset-for-japanese-llm)
|
|||
|
|
- [meta-math/MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA)
|
|||
|
|
- [m-a-p/CodeFeedback-Filtered-Instruction](https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction)
|
|||
|
|
- [kunishou/databricks-dolly-15k-ja](https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja)
|
|||
|
|
- その他日本語版・英語版のwikipedia記事から作成したオリジナルデータ，有害プロンプト回避のためのオリジナルデータを使用。 Original data created from Japanese and English Wikipedia articles, as well as original data for avoiding harmful prompts, is used.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
[MIT License](https://opensource.org/license/mit)を適用する。ただし量子化のベースモデルに適用されている[Llama 3.3 Community License Agreement](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/LICENSE)に従ってください。
|
|||
|
|
The MIT License is applied. However, obey the Llama 3.3 Community License Agreement applied to the base model of quantization.
|