52 lines
2.0 KiB
Markdown
52 lines
2.0 KiB
Markdown
---
|
||
license: llama2
|
||
datasets:
|
||
- huangyt/FINETUNE2
|
||
---
|
||
# Model Card for Model ID
|
||
|
||
<!-- Provide a quick summary of what the model is/does. -->
|
||
|
||
在llama-2-13b上使用huangyt/FINETUNE2資料集進行訓練，採用與platypus相近參數
|
||
|
||
# Fine-Tuning Information
|
||
- **GPU:** RTX4090 (single core / 24564MiB)
|
||
- **base model:** meta-llama/Llama-2-13b-hf
|
||
- **dataset:** huangyt/FINETUNE2 (共約3w筆訓練集)
|
||
- **peft_type:** LoRA
|
||
- **lora_rank:** 16
|
||
- **lora alpha:** 8
|
||
- **lora dropout:** 0.05
|
||
- **lora_target:** gate_proj, up_proj, down_proj
|
||
- **per_device_train_batch_size:** 8
|
||
- **gradient_accumulation_steps:** 8
|
||
- **learning_rate :** 4e-4
|
||
- **epoch:** 1
|
||
- **batch size:** 8
|
||
- **microbatch size:** 4
|
||
- **warmup steps:** 100
|
||
- **weight decay:** cosine
|
||
- **cutoff length:** 2048
|
||
- **precision:** bf16
|
||
- **group_by_length:** True
|
||
- **load_in_8bit:** True
|
||
|
||
# Fine-Tuning Detail
|
||
- **train_loss:** 0.0823
|
||
- **train_runtime:** 02:40:01
|
||
|
||
# Evaluation
|
||
- 評估結果來自**HuggingFaceH4/open_llm_leaderboard**
|
||
- 與Llama-2-13b比較4種Benchmark，包含**ARC**、**HellaSwag**、**MMLU**、**TruthfulQA**
|
||
|
||
| Model |Average| ARC |HellaSwag| MMLU |TruthfulQA|
|
||
|--------------------------------------------------------|-------|-------|---------|-------|----------|
|
||
|meta-llama/Llama-2-13b-hf | 56.9 | 58.11 | 80.97 | 54.34 | 34.17 |
|
||
|meta-llama/Llama-2-13b-chat-hf | 59.93 | 59.04 | 81.94 | 54.64 | 44.12 |
|
||
|CHIH-HUNG/llama-2-13b-FINETUNE2_3w | 58.24 | 58.62 | 82.32 | 54.25 | 38.17 |
|
||
|CHIH-HUNG/llama-2-13b-huangyt_Fintune_1_17w-q_k_v_o_proj| 58.49 | 59.73 | 81.06 | 54.53 | 38.64 |
|
||
|CHIH-HUNG/llama-2-13b-FINETUNE2_3w-q_k_v_o_proj | 58.21 | 58.53 | 82.47 | 53.9 | 37.92 |
|
||
|CHIH-HUNG/llama-2-13b-FINETUNE2_3w-gate_up_down_proj | 58.81 | 57.42 | 82.42 | 55.57 | 39.19 |
|
||
|wei123602/llama2-13b-fintune2 | wait | wait | wait | wait | wait |
|
||
|