90 lines
4.5 KiB
Markdown
90 lines
4.5 KiB
Markdown
|
---
|
||
|
|
license: llama3.2
|
||
|
|
datasets:
|
||
|
|
- CarrotAI/Magpie-Ko-Pro-AIR
|
||
|
|
- CarrotAI/Carrot
|
||
|
|
- CarrotAI/ko-instruction-dataset
|
||
|
|
language:
|
||
|
|
- ko
|
||
|
|
- en
|
||
|
|
base_model:
|
||
|
|
- meta-llama/Llama-3.2-3B-Instruct
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
new_version: CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct-2412
|
||
|
|
---
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## Model Description
|
||
|
|
|
||
|
|
### Model Details
|
||
|
|
- **Name**: Carrot Llama-3.2 Rabbit Ko
|
||
|
|
- **Version**: 3B Instruct
|
||
|
|
- **Base Model**: CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct
|
||
|
|
- **Languages**: Korean, English
|
||
|
|
- **Model Type**: Large Language Model (Instruction-tuned)
|
||
|
|
|
||
|
|
### Training Process
|
||
|
|
본 모델은 다음과 같은 주요 훈련 단계를 거쳤습니다:
|
||
|
|
|
||
|
|
1. **SFT (Supervised Fine-Tuning)**
|
||
|
|
- 고품질 한국어 및 영어 데이터셋을 사용하여 기본 모델을 세부 조정
|
||
|
|
|
||
|
|
### Limitations
|
||
|
|
- 3B 파라미터 규모로 인한 복잡한 작업에서의 제한적 성능
|
||
|
|
- 특정 도메인에 대한 깊이 있는 전문성 부족
|
||
|
|
- 편향성 및 환각 가능성
|
||
|
|
|
||
|
|
### Ethics Statement
|
||
|
|
모델 개발 과정에서 윤리적 고려사항을 최대한 반영하였으나, 사용자는 항상 결과를 비판적으로 검토해야 합니다.
|
||
|
|
|
||
|
|
### How to Use
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct")
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("CarrotAI/Llama-3.2-Rabbit-Ko-3B-Instruct")
|
||
|
|
```
|
||
|
|
|
||
|
|
## Score
|
||
|
|
|
||
|
|
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|
||
|
|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|
||
|
|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.6490|± |0.0131|
|
||
|
|
| | |strict-match | 5|exact_match|↑ |0.0023|± |0.0013|
|
||
|
|
|gsm8k-ko| 3|flexible-extract| 5|exact_match|↑ |0.3275|± |0.0134|
|
||
|
|
| | |strict-match | 5|exact_match|↑ |0.2737|± |0.0134|
|
||
|
|
|ifeval| 4|none | 5|inst_level_loose_acc |↑ |0.8058|± | N/A|
|
||
|
|
| | |none | 5|inst_level_strict_acc |↑ |0.7686|± | N/A|
|
||
|
|
| | |none | 5|prompt_level_loose_acc |↑ |0.7320|± |0.0191|
|
||
|
|
| | |none | 5|prompt_level_strict_acc|↑ |0.6858|± |0.0200|
|
||
|
|
|
||
|
|
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|
||
|
|
|-------------------------------|------:|------|-----:|--------|---|-----:|---|-----:|
|
||
|
|
|haerae | 1|none | |acc |↑ |0.4180|± |0.0148|
|
||
|
|
| | |none | |acc_norm|↑ |0.4180|± |0.0148|
|
||
|
|
| - haerae_general_knowledge | 1|none | 5|acc |↑ |0.3125|± |0.0350|
|
||
|
|
| | |none | 5|acc_norm|↑ |0.3125|± |0.0350|
|
||
|
|
| - haerae_history | 1|none | 5|acc |↑ |0.3404|± |0.0347|
|
||
|
|
| | |none | 5|acc_norm|↑ |0.3404|± |0.0347|
|
||
|
|
| - haerae_loan_word | 1|none | 5|acc |↑ |0.4083|± |0.0379|
|
||
|
|
| | |none | 5|acc_norm|↑ |0.4083|± |0.0379|
|
||
|
|
| - haerae_rare_word | 1|none | 5|acc |↑ |0.4815|± |0.0249|
|
||
|
|
| | |none | 5|acc_norm|↑ |0.4815|± |0.0249|
|
||
|
|
| - haerae_standard_nomenclature| 1|none | 5|acc |↑ |0.4771|± |0.0405|
|
||
|
|
| | |none | 5|acc_norm|↑ |0.4771|± |0.0405|
|
||
|
|
|
||
|
|
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|
||
|
|
|----------------|------:|------|-----:|--------|---|-----:|---|------|
|
||
|
|
|kobest_boolq | 1|none | 5|acc |↑ |0.7664|± |0.0113|
|
||
|
|
| | |none | 5|f1 |↑ |0.7662|± | N/A|
|
||
|
|
|kobest_copa | 1|none | 5|acc |↑ |0.5620|± |0.0157|
|
||
|
|
| | |none | 5|f1 |↑ |0.5612|± | N/A|
|
||
|
|
|kobest_hellaswag| 1|none | 5|acc |↑ |0.3840|± |0.0218|
|
||
|
|
| | |none | 5|acc_norm|↑ |0.4900|± |0.0224|
|
||
|
|
| | |none | 5|f1 |↑ |0.3807|± | N/A|
|
||
|
|
|kobest_sentineg | 1|none | 5|acc |↑ |0.5869|± |0.0247|
|
||
|
|
| | |none | 5|f1 |↑ |0.5545|± | N/A|
|
||
|
|
|kobest_wic | 1|none | 5|acc |↑ |0.4952|± |0.0141|
|
||
|
|
| | |none | 5|f1 |↑ |0.4000|± | N/A|
|