160 lines
4.0 KiB
Markdown
160 lines
4.0 KiB
Markdown
|
---
|
||
|
|
license: llama2
|
||
|
|
datasets:
|
||
|
|
- AlfredPros/smart-contracts-instructions
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
tags:
|
||
|
|
- code
|
||
|
|
- blockchain
|
||
|
|
- solidity
|
||
|
|
- smart contract
|
||
|
|
---
|
||
|
|
# Code LLaMA 7B Instruct Solidity
|
||
|
|
|
||
|
|
A finetuned 7 billion parameters Code LLaMA - Instruct model to generate Solidity smart contract using 4-bit QLoRA finetuning provided by PEFT library.
|
||
|
|
|
||
|
|
# Training Dataset
|
||
|
|
|
||
|
|
Dataset used to finetune the model is AlfredPros' Smart Contracts Instructions (https://huggingface.co/datasets/AlfredPros/smart-contracts-instructions).
|
||
|
|
A dataset containing 6,003 GPT-generated human instruction and Solidity source code data pairs. This dataset has been processed for training LLMs.
|
||
|
|
|
||
|
|
# Training Parameters
|
||
|
|
|
||
|
|
## Bitsandbytes quantization configurations
|
||
|
|
- Load in 4-bit: true
|
||
|
|
- 4-bit quantization type: NF4
|
||
|
|
- 4-bit compute dtype: float16
|
||
|
|
- 4-bit use double quantization: true
|
||
|
|
|
||
|
|
## Supervised finetuning trainer parameters
|
||
|
|
- Number of train epochs: 1
|
||
|
|
- FP16: true
|
||
|
|
- FP16 option level: O1
|
||
|
|
- BF16: false
|
||
|
|
- Per device train batch size: 1
|
||
|
|
- Gradient accumulation steps: 1
|
||
|
|
- Gradient checkpointing: true
|
||
|
|
- Max gradient normal: 0.3
|
||
|
|
- Learning rate: 2e-4
|
||
|
|
- Weight decay: 0.001
|
||
|
|
- Optimizer: paged AdamW 32-bit
|
||
|
|
- Learning rate scheduler type: cosine
|
||
|
|
- Warmup ratio: 0.03
|
||
|
|
|
||
|
|
# Training Details
|
||
|
|
- GPU used: 1x NVIDIA GeForce GTX 1080Ti
|
||
|
|
- Training time: 21 hours, 4 minutes, and 57 seconds
|
||
|
|
|
||
|
|
# Training Loss
|
||
|
|
```
|
||
|
|
Step Training Loss
|
||
|
|
100 0.330900
|
||
|
|
200 0.293000
|
||
|
|
300 0.276500
|
||
|
|
400 0.290900
|
||
|
|
500 0.306100
|
||
|
|
600 0.302600
|
||
|
|
700 0.337200
|
||
|
|
800 0.295000
|
||
|
|
900 0.297800
|
||
|
|
1000 0.299500
|
||
|
|
1100 0.268900
|
||
|
|
1200 0.257800
|
||
|
|
1300 0.264100
|
||
|
|
1400 0.294400
|
||
|
|
1500 0.293900
|
||
|
|
1600 0.287600
|
||
|
|
1700 0.281200
|
||
|
|
1800 0.273400
|
||
|
|
1900 0.266600
|
||
|
|
2000 0.227500
|
||
|
|
2100 0.261600
|
||
|
|
2200 0.275700
|
||
|
|
2300 0.290100
|
||
|
|
2400 0.290900
|
||
|
|
2500 0.316200
|
||
|
|
2600 0.296500
|
||
|
|
2700 0.291400
|
||
|
|
2800 0.253300
|
||
|
|
2900 0.321500
|
||
|
|
3000 0.269500
|
||
|
|
3100 0.295600
|
||
|
|
3200 0.265800
|
||
|
|
3300 0.262800
|
||
|
|
3400 0.274900
|
||
|
|
3500 0.259800
|
||
|
|
3600 0.226300
|
||
|
|
3700 0.325700
|
||
|
|
3800 0.249000
|
||
|
|
3900 0.237200
|
||
|
|
4000 0.251400
|
||
|
|
4100 0.247000
|
||
|
|
4200 0.278700
|
||
|
|
4300 0.264000
|
||
|
|
4400 0.245000
|
||
|
|
4500 0.235900
|
||
|
|
4600 0.240400
|
||
|
|
4700 0.235200
|
||
|
|
4800 0.220300
|
||
|
|
4900 0.202700
|
||
|
|
5000 0.240500
|
||
|
|
5100 0.258500
|
||
|
|
5200 0.236300
|
||
|
|
5300 0.267500
|
||
|
|
5400 0.236700
|
||
|
|
5500 0.265900
|
||
|
|
5600 0.244900
|
||
|
|
5700 0.297900
|
||
|
|
5800 0.281200
|
||
|
|
5900 0.313800
|
||
|
|
6000 0.249800
|
||
|
|
6003 0.271939
|
||
|
|
```
|
||
|
|
|
||
|
|
# Example Usage
|
||
|
|
```py
|
||
|
|
from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM
|
||
|
|
import torch
|
||
|
|
import accelerate
|
||
|
|
|
||
|
|
use_4bit = True
|
||
|
|
bnb_4bit_compute_dtype = "float16"
|
||
|
|
bnb_4bit_quant_type = "nf4"
|
||
|
|
use_double_nested_quant = True
|
||
|
|
compute_dtype = getattr(torch, bnb_4bit_compute_dtype)
|
||
|
|
|
||
|
|
# BitsAndBytesConfig 4-bit config
|
||
|
|
bnb_config = BitsAndBytesConfig(
|
||
|
|
load_in_4bit=use_4bit,
|
||
|
|
bnb_4bit_use_double_quant=use_double_nested_quant,
|
||
|
|
bnb_4bit_quant_type=bnb_4bit_quant_type,
|
||
|
|
bnb_4bit_compute_dtype=compute_dtype,
|
||
|
|
load_in_8bit_fp32_cpu_offload=True
|
||
|
|
)
|
||
|
|
|
||
|
|
# Load model in 4-bit
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("AlfredPros/CodeLlama-7b-Instruct-Solidity")
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("AlfredPros/CodeLlama-7b-Instruct-Solidity", quantization_config=bnb_config, device_map="balanced_low_0")
|
||
|
|
|
||
|
|
# Make input
|
||
|
|
input='Make a smart contract to create a whitelist of approved wallets. The purpose of this contract is to allow the DAO (Decentralized Autonomous Organization) to approve or revoke certain wallets, and also set a checker address for additional validation if needed. The current owner address can be changed by the current owner.'
|
||
|
|
|
||
|
|
# Make prompt template
|
||
|
|
prompt = f"""### Instruction:
|
||
|
|
Use the Task below and the Input given to write the Response, which is a programming code that can solve the following Task:
|
||
|
|
|
||
|
|
### Task:
|
||
|
|
{input}
|
||
|
|
|
||
|
|
### Solution:
|
||
|
|
"""
|
||
|
|
|
||
|
|
# Tokenize the input
|
||
|
|
input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda()
|
||
|
|
# Run the model to infere an output
|
||
|
|
outputs = model.generate(input_ids=input_ids, max_new_tokens=1024, do_sample=True, top_p=0.9, temperature=0.001, pad_token_id=1)
|
||
|
|
|
||
|
|
# Detokenize and display the generated output
|
||
|
|
print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0][len(prompt):])
|
||
|
|
```
|