pathcosmos/frankenstallm
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: pathcosmos/frankenstallm
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-14 04:21:16 +08:00
commit d4abdb70fa
516 changed files with 5584477 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

308
source/eval/outputs/3b_reeval_20260305_1451/full_eval_report.md Normal file
View File

@@ -0,0 +1,308 @@
# FRANKENSTALLM 3B 종합 평가 리포트
- **모델**: FRANKENSTALLM 3B
- **체크포인트**: checkpoint-0057000
- **평가 일시**: 2026-03-05 16:40:32
- **총 소요 시간**: 256.6초
## Executive Summary
| 메트릭 | 값 |
|--------|-----|
| 주요 PPL (3b_val) | 5.2263 |
| MMLU-KO 평균 (6과목) | 22.75% |
| MMLU-EN 평균 | 25.81% |
| KoBEST 평균 (5태스크) | 43.69% |
| HAE-RAE | 19.71% |
| hellaswag (0-shot) | 26.15% |
| arc_easy (0-shot) | 25.63% |
| arc_challenge (0-shot) | 27.90% |
| winogrande (0-shot) | 50.59% |
| piqa (0-shot) | 52.50% |
| Top-1 정확도 (Calibration) | 0.6875 |
## 참고 모델 비교
| 모델 | 파라미터 | MMLU-KO | MMLU-EN | KoBEST 평균 | PPL |
|------|---------|---------|---------|------------|-----|
| **FRANKENSTALLM 3B** | 3B | 22.75% | 25.81% | 43.69% | 5.2263 |
| Llama-3.2-3B | 3B | ~42% | ~58% | ~55% | — |
| Qwen2.5-3B | 3B | ~48% | ~65% | ~60% | — |
| EXAONE-3.5-2.4B | 2.4B | ~35% | ~50% | ~50% | — |
---
# Perplexity 평가
| 데이터셋 | PPL | Bits/Token | 전체 토큰 | 평가 토큰 | 소요 시간 |
|---------|-----|-----------|---------|---------|---------|
| korean_namuwiki | 25.8814 | 4.6938 | 2,166,179 | 6,488,957 | 63.7s |
| cc100_ko | 21.7820 | 4.4451 | 4,532,995 | 13,585,262 | 133.2s |
| namuwiki_2023b | 18.9170 | 4.2416 | 2,554,574 | 7,654,022 | 75.1s |
| val | 18.3046 | 4.1941 | 3,040,344 | 9,110,737 | 89.4s |
| korean_wiki | 11.8359 | 3.5651 | 524,561 | 1,567,747 | 15.5s |
| wikipedia_ko | 10.7059 | 3.4203 | 590,691 | 1,765,762 | 17.4s |
| korean | 7.0155 | 2.8105 | 17,850,578 | 53,512,147 | 521.6s |
| open_web_math | 6.9264 | 2.7921 | 5,230,614 | 15,677,467 | 153.5s |
| korean_c4 | 5.7173 | 2.5153 | 15,159,838 | 45,445,722 | 443.1s |
| 3b | 5.2263 | 2.3858 | 75,681,623 | 226,891,932 | 2227.3s |
| cosmo_web_v2 | 4.1664 | 2.0588 | 2,875,418 | 8,616,467 | 84.6s |
| cosmo_stories | 3.9552 | 1.9837 | 6,299,229 | 18,881,012 | 185.2s |
| cosmo_openstax | 3.8673 | 1.9513 | 242,751 | 723,497 | 7.2s |
| cosmo_stanford | 3.3624 | 1.7495 | 2,217,375 | 6,642,457 | 65.3s |
| cosmo_wikihow | 3.3097 | 1.7267 | 395,586 | 1,180,927 | 11.8s |
| cosmo_auto_math_text | 3.1492 | 1.6550 | 2,632,946 | 7,888,877 | 77.3s |
| cosmo_khanacademy | 2.9322 | 1.5520 | 47,751 | 138,662 | 1.5s |
| mathpile | 2.7244 | 1.4459 | 2,379,696 | 7,129,052 | 69.9s |
| hplt_ko | 2.4028 | 1.2647 | 16,165,735 | 48,460,462 | 475.9s |
---
# Calibration 및 Token NLL 분석
## Calibration 결과
| 메트릭 | 값 |
|--------|-----|
| Top-1 Accuracy | 0.6875 |
| Top-5 Accuracy | 0.8164 |
| Top-10 Accuracy | 0.8593 |
| Mean Correct Prob | 0.6152 |
| Mean Entropy | 1.5682 |
## Token NLL 분포
| 통계 | 값 |
|------|-----|
| 평균 | 1.5561 |
| 표준편차 | 2.4926 |
| 중앙값 | 0.1221 |
| 최솟값 | N/A |
| 최댓값 | N/A |
### Percentiles
| Percentile | 값 |
|------------|-----|
| p5th | 0.0000 |
| p25th | 0.0017 |
| p75th | 2.3594 |
| p95th | 7.0312 |
| p99th | 10.3125 |
### 고손실 토큰 비율
| 임계값 | 비율 |
|--------|-----|
| NLL > 10 | 0.0118 |
| NLL > 5 | 0.1086 |
---
# 생성 품질 분석
## 요약 통계
| 메트릭 | 값 |
|--------|-----|
| Total Generations | 60.0000 |
| N Prompts | 15.0000 |
| Temperatures | [0.0, 0.5, 0.8, 1.0] |
| Greedy Avg 1Gram Rep | 0.7964 |
| Greedy Avg 2Gram Rep | 0.7586 |
| Greedy Avg 3Gram Rep | 0.7275 |
| Greedy Avg 4Gram Rep | 0.7078 |
| Greedy Eos Rate | 0.0000 |
| Greedy Avg Tokens | 256.0000 |
| Sampled Avg 3Gram Rep | 0.2427 |
| Sampled Eos Rate | 0.0000 |
| Sampled Avg Tokens | 256.0000 |
| Elapsed Sec | 141.8000 |
## 생성 샘플 (Greedy)
### 샘플 1
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 2
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 3
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 4
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 5
**Prompt**: 인공지능이란
**Generated**:
---
# 표준 벤치마크 결과
## 한국어 벤치마크
### KoBEST (0-shot)
| 태스크 | Accuracy | F1 |
|--------|----------|-----|
| kobest_boolq | 50.28% | 0.3457 |
| kobest_copa | 49.30% | 0.4921 |
| kobest_hellaswag | 21.60% | 0.2153 |
| kobest_sentineg | 48.61% | 0.4737 |
| kobest_wic | 48.65% | 0.3286 |
| **평균** | **43.69%** | |
### HAE-RAE (0-shot)
- Accuracy: 19.71%
| 서브태스크 | Accuracy |
|-----------|----------|
| haerae_general_knowledge | 21.59% |
| haerae_history | 23.40% |
| haerae_loan_word | 21.30% |
| haerae_rare_word | 18.77% |
| haerae_standard_nomenclature | 13.73% |
### MMLU-KO (0-shot)
평가된 과목 수: **6**
전체 평균: **22.75%**
**상위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| medical | 30.56% |
| humanities | 24.51% |
| business | 24.14% |
| social_sciences | 20.59% |
| other | 19.64% |
| stem | 19.57% |
**하위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| medical | 30.56% |
| humanities | 24.51% |
| business | 24.14% |
| social_sciences | 20.59% |
| other | 19.64% |
| stem | 19.57% |
## 영어 벤치마크
### 주요 벤치마크 (0-shot)
| 태스크 | Accuracy | Acc (norm) |
|--------|----------|-----------|
| hellaswag | 26.00% | 26.15% |
| arc_easy | 25.63% | 26.64% |
| arc_challenge | 21.67% | 27.90% |
| winogrande | 50.59% | N/A |
| piqa | 52.50% | 48.31% |
### MMLU-EN (0-shot)
평가된 과목 수: **61**
전체 평균: **25.81%**
**상위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| college_physics | 37.25% |
| college_computer_science | 34.00% |
| high_school_statistics | 33.80% |
| us_foreign_policy | 32.00% |
| security_studies | 31.43% |
| world_religions | 30.99% |
| professional_medicine | 30.88% |
| high_school_government_and_politics | 30.57% |
| jurisprudence | 30.56% |
| human_sexuality | 30.53% |
**하위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| moral_disputes | 22.54% |
| philosophy | 22.51% |
| college_chemistry | 22.00% |
| international_law | 21.49% |
| public_relations | 20.00% |
| human_aging | 19.73% |
| college_biology | 19.44% |
| anatomy | 17.04% |
| global_facts | 17.00% |
| abstract_algebra | 15.00% |
## 0-shot vs 5-shot 비교
| 태스크 | 0-shot Acc | 5-shot Acc | 변화 |
|--------|-----------|-----------|------|
| global_mmlu_ko | 22.75% | 26.75% | +4.00pp |
| global_mmlu_ko_business | 24.14% | 31.03% | +6.90pp |
| global_mmlu_ko_humanities | 24.51% | 28.43% | +3.92pp |
| global_mmlu_ko_medical | 30.56% | 36.11% | +5.56pp |
| global_mmlu_ko_other | 19.64% | 23.21% | +3.57pp |
| global_mmlu_ko_social_sciences | 20.59% | 23.53% | +2.94pp |
| global_mmlu_ko_stem | 19.57% | 21.74% | +2.17pp |
| haerae | 19.71% | 20.26% | +0.55pp |
| haerae_general_knowledge | 21.59% | 22.73% | +1.14pp |
| haerae_history | 23.40% | 14.89% | -8.51pp |
| haerae_loan_word | 21.30% | 24.26% | +2.96pp |
| haerae_rare_word | 18.77% | 18.02% | -0.74pp |
| haerae_standard_nomenclature | 13.73% | 25.49% | +11.76pp |
| kobest_boolq | 50.28% | 50.21% | -0.07pp |
| kobest_copa | 49.30% | 46.80% | -2.50pp |
| kobest_hellaswag | 21.60% | 20.80% | -0.80pp |
| kobest_sentineg | 48.61% | 47.86% | -0.76pp |
| kobest_wic | 48.65% | 48.97% | +0.32pp |
평균 변화: +1.80pp | 개선: 12 | 하락: 6 | 동일: 0
## Repetition 파라미터 검색
| 설정 | Temp | Rep Pen | 3-gram | 4-gram | EOS Rate | Avg Tokens |
|------|------|---------|--------|--------|----------|-----------|
| t0.7_rep1.3 | 0.70 | 1.30 | 0.0000 | 0.0000 | 0.0000 | 256.0 | **← best**
| t0.9_rep1.2 | 0.90 | 1.20 | 0.0000 | 0.0000 | 0.0000 | 256.0 |
| t0.7_rep1.2 | 0.70 | 1.20 | 0.0088 | 0.0000 | 0.0000 | 256.0 |
| t0.9_rep1.1 | 0.90 | 1.10 | 0.0094 | 0.0013 | 0.0000 | 256.0 |
| t1.0_rep1.1 | 1.00 | 1.10 | 0.0121 | 0.0048 | 0.0000 | 256.0 |
| t0.5_rep1.1 | 0.50 | 1.10 | 0.0192 | 0.0119 | 0.0000 | 256.0 |
| t1.0 | 1.00 | 1.00 | 0.0358 | 0.0281 | 0.0000 | 256.0 |
| t0.9 | 0.90 | 1.00 | 0.0839 | 0.0464 | 0.0000 | 256.0 |
| t0.7_rep1.1 | 0.70 | 1.10 | 0.0851 | 0.0551 | 0.0000 | 256.0 |
| t0.7 | 0.70 | 1.00 | 0.4769 | 0.4340 | 0.0000 | 256.0 |
| t0.5 | 0.50 | 1.00 | 0.6012 | 0.5868 | 0.0000 | 256.0 |
| greedy | 0.00 | 1.00 | 0.6099 | 0.5702 | 0.0000 | 256.0 |
---
*이 리포트는 자동으로 생성되었습니다.*

1370
source/eval/outputs/3b_reeval_20260305_1451/generation_samples.json Normal file
View File

File diff suppressed because it is too large Load Diff

1610
source/eval/outputs/3b_reeval_20260305_1451/phase1_results.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu0_pipeline_reeval.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ce757143b7dc33656b8dc8ec3ff97921e6bac01d3720f51b73cc510354ea1bb0
size 27954700

36
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu0_pipeline_reeval.log Normal file
View File

File diff suppressed because one or more lines are too long

208757
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu0_pipeline_reeval_0shot.json Normal file
View File

File diff suppressed because one or more lines are too long

3
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu0_pipeline_reeval_5shot.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c813f2a224de26df76c0f3eaeb9539b3fc005679b3d7f4fef588a990b90a9217
size 19229722

141154
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu1_pipeline_reeval.json Normal file
View File

File diff suppressed because one or more lines are too long

32
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu1_pipeline_reeval.log Normal file
View File

File diff suppressed because one or more lines are too long

70576
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu1_pipeline_reeval_0shot.json Normal file
View File

File diff suppressed because one or more lines are too long

70576
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu1_pipeline_reeval_5shot.json Normal file
View File

File diff suppressed because one or more lines are too long

3
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu2_pipeline_reeval.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0997779fe0ab7232bc9a9d643dd5cf9872a977e0dcba7757f61ea7af2e8052e5
size 17773351

44
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu2_pipeline_reeval.log Normal file
View File

File diff suppressed because one or more lines are too long

104166
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu2_pipeline_reeval_0shot.json Normal file
View File

File diff suppressed because one or more lines are too long

3
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu2_pipeline_reeval_5shot.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f14b8fed85a97c6ba794c66ddd89c96c927024580a4a21f73218de2985dd24f8
size 13059875

75466
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu3_pipeline_reeval.json Normal file
View File

File diff suppressed because one or more lines are too long

48
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu3_pipeline_reeval.log Normal file
View File

File diff suppressed because one or more lines are too long

37732
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu3_pipeline_reeval_0shot.json Normal file
View File

File diff suppressed because one or more lines are too long

37732
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu3_pipeline_reeval_5shot.json Normal file
View File

File diff suppressed because one or more lines are too long

3
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu4_pipeline_reeval.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:aa5b032fa1d3c3269433961fe1ab7d71e436c5d780bb597d6e7170a28b2dc110
size 64398794

30
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu4_pipeline_reeval.log Normal file
View File

File diff suppressed because one or more lines are too long

165351
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu5_pipeline_reeval.json Normal file
View File

File diff suppressed because one or more lines are too long

27
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu5_pipeline_reeval.log Normal file
View File

File diff suppressed because one or more lines are too long

3
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu6_pipeline_reeval.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:61a523f6cc0578573f86eacd9d80bdcc6bf9065522816ec9b188823a1a9ebe29
size 40316609

98
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu6_pipeline_reeval.log Normal file
View File

File diff suppressed because one or more lines are too long

3
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu7_pipeline_reeval.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4fb9848e05bfdb6d3d6f067b6fd2535add4fbc27e119d4dc3adb99eb527642cd
size 22470740

119
source/eval/outputs/3b_reeval_20260305_1451/phase2_gpu7_pipeline_reeval.log Normal file
View File

File diff suppressed because one or more lines are too long

3
source/eval/outputs/3b_reeval_20260305_1451/phase2_results.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4e587c1697b65cc3d5c4ebb06df53570e0a595a2267a19882980718985489975
size 202412580

31
source/eval/outputs/3b_reeval_20260305_1451/reports/00_executive_summary.md Normal file
View File

@@ -0,0 +1,31 @@
# FRANKENSTALLM 3B 종합 평가 리포트
- **모델**: FRANKENSTALLM 3B
- **체크포인트**: checkpoint-0057000
- **평가 일시**: 2026-03-05 16:40:32
- **총 소요 시간**: 256.6초
## Executive Summary
| 메트릭 | 값 |
|--------|-----|
| 주요 PPL (3b_val) | 5.2263 |
| MMLU-KO 평균 (6과목) | 22.75% |
| MMLU-EN 평균 | 25.81% |
| KoBEST 평균 (5태스크) | 43.69% |
| HAE-RAE | 19.71% |
| hellaswag (0-shot) | 26.15% |
| arc_easy (0-shot) | 25.63% |
| arc_challenge (0-shot) | 27.90% |
| winogrande (0-shot) | 50.59% |
| piqa (0-shot) | 52.50% |
| Top-1 정확도 (Calibration) | 0.6875 |
## 참고 모델 비교
| 모델 | 파라미터 | MMLU-KO | MMLU-EN | KoBEST 평균 | PPL |
|------|---------|---------|---------|------------|-----|
| **FRANKENSTALLM 3B** | 3B | 22.75% | 25.81% | 43.69% | 5.2263 |
| Llama-3.2-3B | 3B | ~42% | ~58% | ~55% | — |
| Qwen2.5-3B | 3B | ~48% | ~65% | ~60% | — |
| EXAONE-3.5-2.4B | 2.4B | ~35% | ~50% | ~50% | — |

23
source/eval/outputs/3b_reeval_20260305_1451/reports/01_perplexity_report.md Normal file
View File

@@ -0,0 +1,23 @@
# Perplexity 평가
| 데이터셋 | PPL | Bits/Token | 전체 토큰 | 평가 토큰 | 소요 시간 |
|---------|-----|-----------|---------|---------|---------|
| korean_namuwiki | 25.8814 | 4.6938 | 2,166,179 | 6,488,957 | 63.7s |
| cc100_ko | 21.7820 | 4.4451 | 4,532,995 | 13,585,262 | 133.2s |
| namuwiki_2023b | 18.9170 | 4.2416 | 2,554,574 | 7,654,022 | 75.1s |
| val | 18.3046 | 4.1941 | 3,040,344 | 9,110,737 | 89.4s |
| korean_wiki | 11.8359 | 3.5651 | 524,561 | 1,567,747 | 15.5s |
| wikipedia_ko | 10.7059 | 3.4203 | 590,691 | 1,765,762 | 17.4s |
| korean | 7.0155 | 2.8105 | 17,850,578 | 53,512,147 | 521.6s |
| open_web_math | 6.9264 | 2.7921 | 5,230,614 | 15,677,467 | 153.5s |
| korean_c4 | 5.7173 | 2.5153 | 15,159,838 | 45,445,722 | 443.1s |
| 3b | 5.2263 | 2.3858 | 75,681,623 | 226,891,932 | 2227.3s |
| cosmo_web_v2 | 4.1664 | 2.0588 | 2,875,418 | 8,616,467 | 84.6s |
| cosmo_stories | 3.9552 | 1.9837 | 6,299,229 | 18,881,012 | 185.2s |
| cosmo_openstax | 3.8673 | 1.9513 | 242,751 | 723,497 | 7.2s |
| cosmo_stanford | 3.3624 | 1.7495 | 2,217,375 | 6,642,457 | 65.3s |
| cosmo_wikihow | 3.3097 | 1.7267 | 395,586 | 1,180,927 | 11.8s |
| cosmo_auto_math_text | 3.1492 | 1.6550 | 2,632,946 | 7,888,877 | 77.3s |
| cosmo_khanacademy | 2.9322 | 1.5520 | 47,751 | 138,662 | 1.5s |
| mathpile | 2.7244 | 1.4459 | 2,379,696 | 7,129,052 | 69.9s |
| hplt_ko | 2.4028 | 1.2647 | 16,165,735 | 48,460,462 | 475.9s |

38
source/eval/outputs/3b_reeval_20260305_1451/reports/02_calibration_report.md Normal file
View File

@@ -0,0 +1,38 @@
# Calibration 및 Token NLL 분석
## Calibration 결과
| 메트릭 | 값 |
|--------|-----|
| Top-1 Accuracy | 0.6875 |
| Top-5 Accuracy | 0.8164 |
| Top-10 Accuracy | 0.8593 |
| Mean Correct Prob | 0.6152 |
| Mean Entropy | 1.5682 |
## Token NLL 분포
| 통계 | 값 |
|------|-----|
| 평균 | 1.5561 |
| 표준편차 | 2.4926 |
| 중앙값 | 0.1221 |
| 최솟값 | N/A |
| 최댓값 | N/A |
### Percentiles
| Percentile | 값 |
|------------|-----|
| p5th | 0.0000 |
| p25th | 0.0017 |
| p75th | 2.3594 |
| p95th | 7.0312 |
| p99th | 10.3125 |
### 고손실 토큰 비율
| 임계값 | 비율 |
|--------|-----|
| NLL > 10 | 0.0118 |
| NLL > 5 | 0.1086 |

52
source/eval/outputs/3b_reeval_20260305_1451/reports/03_generation_quality.md Normal file
View File

@@ -0,0 +1,52 @@
# 생성 품질 분석
## 요약 통계
| 메트릭 | 값 |
|--------|-----|
| Total Generations | 60.0000 |
| N Prompts | 15.0000 |
| Temperatures | [0.0, 0.5, 0.8, 1.0] |
| Greedy Avg 1Gram Rep | 0.7964 |
| Greedy Avg 2Gram Rep | 0.7586 |
| Greedy Avg 3Gram Rep | 0.7275 |
| Greedy Avg 4Gram Rep | 0.7078 |
| Greedy Eos Rate | 0.0000 |
| Greedy Avg Tokens | 256.0000 |
| Sampled Avg 3Gram Rep | 0.2427 |
| Sampled Eos Rate | 0.0000 |
| Sampled Avg Tokens | 256.0000 |
| Elapsed Sec | 141.8000 |
## 생성 샘플 (Greedy)
### 샘플 1
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 2
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 3
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 4
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 5
**Prompt**: 인공지능이란
**Generated**:

148
source/eval/outputs/3b_reeval_20260305_1451/reports/04_benchmark_report.md Normal file
View File

@@ -0,0 +1,148 @@
# 표준 벤치마크 결과
## 한국어 벤치마크
### KoBEST (0-shot)
| 태스크 | Accuracy | F1 |
|--------|----------|-----|
| kobest_boolq | 50.28% | 0.3457 |
| kobest_copa | 49.30% | 0.4921 |
| kobest_hellaswag | 21.60% | 0.2153 |
| kobest_sentineg | 48.61% | 0.4737 |
| kobest_wic | 48.65% | 0.3286 |
| **평균** | **43.69%** | |
### HAE-RAE (0-shot)
- Accuracy: 19.71%
| 서브태스크 | Accuracy |
|-----------|----------|
| haerae_general_knowledge | 21.59% |
| haerae_history | 23.40% |
| haerae_loan_word | 21.30% |
| haerae_rare_word | 18.77% |
| haerae_standard_nomenclature | 13.73% |
### MMLU-KO (0-shot)
평가된 과목 수: **6**
전체 평균: **22.75%**
**상위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| medical | 30.56% |
| humanities | 24.51% |
| business | 24.14% |
| social_sciences | 20.59% |
| other | 19.64% |
| stem | 19.57% |
**하위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| medical | 30.56% |
| humanities | 24.51% |
| business | 24.14% |
| social_sciences | 20.59% |
| other | 19.64% |
| stem | 19.57% |
## 영어 벤치마크
### 주요 벤치마크 (0-shot)
| 태스크 | Accuracy | Acc (norm) |
|--------|----------|-----------|
| hellaswag | 26.00% | 26.15% |
| arc_easy | 25.63% | 26.64% |
| arc_challenge | 21.67% | 27.90% |
| winogrande | 50.59% | N/A |
| piqa | 52.50% | 48.31% |
### MMLU-EN (0-shot)
평가된 과목 수: **61**
전체 평균: **25.81%**
**상위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| college_physics | 37.25% |
| college_computer_science | 34.00% |
| high_school_statistics | 33.80% |
| us_foreign_policy | 32.00% |
| security_studies | 31.43% |
| world_religions | 30.99% |
| professional_medicine | 30.88% |
| high_school_government_and_politics | 30.57% |
| jurisprudence | 30.56% |
| human_sexuality | 30.53% |
**하위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| moral_disputes | 22.54% |
| philosophy | 22.51% |
| college_chemistry | 22.00% |
| international_law | 21.49% |
| public_relations | 20.00% |
| human_aging | 19.73% |
| college_biology | 19.44% |
| anatomy | 17.04% |
| global_facts | 17.00% |
| abstract_algebra | 15.00% |
## 0-shot vs 5-shot 비교
| 태스크 | 0-shot Acc | 5-shot Acc | 변화 |
|--------|-----------|-----------|------|
| global_mmlu_ko | 22.75% | 26.75% | +4.00pp |
| global_mmlu_ko_business | 24.14% | 31.03% | +6.90pp |
| global_mmlu_ko_humanities | 24.51% | 28.43% | +3.92pp |
| global_mmlu_ko_medical | 30.56% | 36.11% | +5.56pp |
| global_mmlu_ko_other | 19.64% | 23.21% | +3.57pp |
| global_mmlu_ko_social_sciences | 20.59% | 23.53% | +2.94pp |
| global_mmlu_ko_stem | 19.57% | 21.74% | +2.17pp |
| haerae | 19.71% | 20.26% | +0.55pp |
| haerae_general_knowledge | 21.59% | 22.73% | +1.14pp |
| haerae_history | 23.40% | 14.89% | -8.51pp |
| haerae_loan_word | 21.30% | 24.26% | +2.96pp |
| haerae_rare_word | 18.77% | 18.02% | -0.74pp |
| haerae_standard_nomenclature | 13.73% | 25.49% | +11.76pp |
| kobest_boolq | 50.28% | 50.21% | -0.07pp |
| kobest_copa | 49.30% | 46.80% | -2.50pp |
| kobest_hellaswag | 21.60% | 20.80% | -0.80pp |
| kobest_sentineg | 48.61% | 47.86% | -0.76pp |
| kobest_wic | 48.65% | 48.97% | +0.32pp |
평균 변화: +1.80pp | 개선: 12 | 하락: 6 | 동일: 0
## Repetition 파라미터 검색
| 설정 | Temp | Rep Pen | 3-gram | 4-gram | EOS Rate | Avg Tokens |
|------|------|---------|--------|--------|----------|-----------|
| t0.7_rep1.3 | 0.70 | 1.30 | 0.0000 | 0.0000 | 0.0000 | 256.0 | **← best**
| t0.9_rep1.2 | 0.90 | 1.20 | 0.0000 | 0.0000 | 0.0000 | 256.0 |
| t0.7_rep1.2 | 0.70 | 1.20 | 0.0088 | 0.0000 | 0.0000 | 256.0 |
| t0.9_rep1.1 | 0.90 | 1.10 | 0.0094 | 0.0013 | 0.0000 | 256.0 |
| t1.0_rep1.1 | 1.00 | 1.10 | 0.0121 | 0.0048 | 0.0000 | 256.0 |
| t0.5_rep1.1 | 0.50 | 1.10 | 0.0192 | 0.0119 | 0.0000 | 256.0 |
| t1.0 | 1.00 | 1.00 | 0.0358 | 0.0281 | 0.0000 | 256.0 |
| t0.9 | 0.90 | 1.00 | 0.0839 | 0.0464 | 0.0000 | 256.0 |
| t0.7_rep1.1 | 0.70 | 1.10 | 0.0851 | 0.0551 | 0.0000 | 256.0 |
| t0.7 | 0.70 | 1.00 | 0.4769 | 0.4340 | 0.0000 | 256.0 |
| t0.5 | 0.50 | 1.00 | 0.6012 | 0.5868 | 0.0000 | 256.0 |
| greedy | 0.00 | 1.00 | 0.6099 | 0.5702 | 0.0000 | 256.0 |
---
*이 리포트는 자동으로 생성되었습니다.*