pathcosmos/frankenstallm
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: pathcosmos/frankenstallm
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-14 04:21:16 +08:00
commit d4abdb70fa
516 changed files with 5584477 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
source/eval/outputs/3b_reeval_20260305_1057/reports/00_executive_summary.md Normal file
View File

@@ -0,0 +1,31 @@
# FRANKENSTALLM 3B 종합 평가 리포트
- **모델**: FRANKENSTALLM 3B
- **체크포인트**: checkpoint-0057000
- **평가 일시**: 2026-03-05 11:03:00
- **총 소요 시간**: 306.9초
## Executive Summary
| 메트릭 | 값 |
|--------|-----|
| 주요 PPL (3b_val) | 5.2263 |
| MMLU-KO 평균 (6과목) | 25.30% |
| MMLU-EN 평균 | 25.82% |
| KoBEST 평균 (5태스크) | 43.58% |
| HAE-RAE | 19.98% |
| hellaswag (0-shot) | 26.15% |
| arc_easy (0-shot) | 25.63% |
| arc_challenge (0-shot) | 27.90% |
| winogrande (0-shot) | 50.59% |
| piqa (0-shot) | 52.50% |
| Top-1 정확도 (Calibration) | 0.6875 |
## 참고 모델 비교
| 모델 | 파라미터 | MMLU-KO | MMLU-EN | KoBEST 평균 | PPL |
|------|---------|---------|---------|------------|-----|
| **FRANKENSTALLM 3B** | 3B | 25.30% | 25.82% | 43.58% | 5.2263 |
| Llama-3.2-3B | 3B | ~42% | ~58% | ~55% | — |
| Qwen2.5-3B | 3B | ~48% | ~65% | ~60% | — |
| EXAONE-3.5-2.4B | 2.4B | ~35% | ~50% | ~50% | — |

23
source/eval/outputs/3b_reeval_20260305_1057/reports/01_perplexity_report.md Normal file
View File

@@ -0,0 +1,23 @@
# Perplexity 평가
| 데이터셋 | PPL | Bits/Token | 전체 토큰 | 평가 토큰 | 소요 시간 |
|---------|-----|-----------|---------|---------|---------|
| korean_namuwiki | 25.8814 | 4.6938 | 2,166,179 | 6,488,957 | 63.7s |
| cc100_ko | 21.7820 | 4.4451 | 4,532,995 | 13,585,262 | 133.2s |
| namuwiki_2023b | 18.9170 | 4.2416 | 2,554,574 | 7,654,022 | 75.1s |
| val | 18.3046 | 4.1941 | 3,040,344 | 9,110,737 | 89.4s |
| korean_wiki | 11.8359 | 3.5651 | 524,561 | 1,567,747 | 15.5s |
| wikipedia_ko | 10.7059 | 3.4203 | 590,691 | 1,765,762 | 17.4s |
| korean | 7.0155 | 2.8105 | 17,850,578 | 53,512,147 | 521.6s |
| open_web_math | 6.9264 | 2.7921 | 5,230,614 | 15,677,467 | 153.5s |
| korean_c4 | 5.7173 | 2.5153 | 15,159,838 | 45,445,722 | 443.1s |
| 3b | 5.2263 | 2.3858 | 75,681,623 | 226,891,932 | 2227.3s |
| cosmo_web_v2 | 4.1664 | 2.0588 | 2,875,418 | 8,616,467 | 84.6s |
| cosmo_stories | 3.9552 | 1.9837 | 6,299,229 | 18,881,012 | 185.2s |
| cosmo_openstax | 3.8673 | 1.9513 | 242,751 | 723,497 | 7.2s |
| cosmo_stanford | 3.3624 | 1.7495 | 2,217,375 | 6,642,457 | 65.3s |
| cosmo_wikihow | 3.3097 | 1.7267 | 395,586 | 1,180,927 | 11.8s |
| cosmo_auto_math_text | 3.1492 | 1.6550 | 2,632,946 | 7,888,877 | 77.3s |
| cosmo_khanacademy | 2.9322 | 1.5520 | 47,751 | 138,662 | 1.5s |
| mathpile | 2.7244 | 1.4459 | 2,379,696 | 7,129,052 | 69.9s |
| hplt_ko | 2.4028 | 1.2647 | 16,165,735 | 48,460,462 | 475.9s |

38
source/eval/outputs/3b_reeval_20260305_1057/reports/02_calibration_report.md Normal file
View File

@@ -0,0 +1,38 @@
# Calibration 및 Token NLL 분석
## Calibration 결과
| 메트릭 | 값 |
|--------|-----|
| Top-1 Accuracy | 0.6875 |
| Top-5 Accuracy | 0.8164 |
| Top-10 Accuracy | 0.8593 |
| Mean Correct Prob | 0.6152 |
| Mean Entropy | 1.5682 |
## Token NLL 분포
| 통계 | 값 |
|------|-----|
| 평균 | 1.5561 |
| 표준편차 | 2.4926 |
| 중앙값 | 0.1221 |
| 최솟값 | N/A |
| 최댓값 | N/A |
### Percentiles
| Percentile | 값 |
|------------|-----|
| p5th | 0.0000 |
| p25th | 0.0017 |
| p75th | 2.3594 |
| p95th | 7.0312 |
| p99th | 10.3125 |
### 고손실 토큰 비율
| 임계값 | 비율 |
|--------|-----|
| NLL > 10 | 0.0118 |
| NLL > 5 | 0.1086 |

52
source/eval/outputs/3b_reeval_20260305_1057/reports/03_generation_quality.md Normal file
View File

@@ -0,0 +1,52 @@
# 생성 품질 분석
## 요약 통계
| 메트릭 | 값 |
|--------|-----|
| Total Generations | 60.0000 |
| N Prompts | 15.0000 |
| Temperatures | [0.0, 0.5, 0.8, 1.0] |
| Greedy Avg 1Gram Rep | 0.7964 |
| Greedy Avg 2Gram Rep | 0.7586 |
| Greedy Avg 3Gram Rep | 0.7275 |
| Greedy Avg 4Gram Rep | 0.7078 |
| Greedy Eos Rate | 0.0000 |
| Greedy Avg Tokens | 256.0000 |
| Sampled Avg 3Gram Rep | 0.2427 |
| Sampled Eos Rate | 0.0000 |
| Sampled Avg Tokens | 256.0000 |
| Elapsed Sec | 141.8000 |
## 생성 샘플 (Greedy)
### 샘플 1
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 2
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 3
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 4
**Prompt**: 대한민국의 수도는
**Generated**:
### 샘플 5
**Prompt**: 인공지능이란
**Generated**:

146
source/eval/outputs/3b_reeval_20260305_1057/reports/04_benchmark_report.md Normal file
View File

@@ -0,0 +1,146 @@
# 표준 벤치마크 결과
## 한국어 벤치마크
### KoBEST (0-shot)
| 태스크 | Accuracy | F1 |
|--------|----------|-----|
| kobest_boolq | 50.14% | 0.3340 |
| kobest_copa | 49.40% | 0.4926 |
| kobest_hellaswag | 19.40% | 0.1927 |
| kobest_sentineg | 50.13% | 0.4667 |
| kobest_wic | 48.81% | 0.3294 |
| **평균** | **43.58%** | |
### HAE-RAE (0-shot)
- Accuracy: 19.98%
| 서브태스크 | Accuracy |
|-----------|----------|
| haerae_general_knowledge | 18.75% |
| haerae_history | 20.74% |
| haerae_loan_word | 14.79% |
| haerae_rare_word | 21.73% |
| haerae_standard_nomenclature | 21.57% |
### MMLU-KO (0-shot)
평가된 과목 수: **6**
전체 평균: **25.30%**
**상위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| medical | 36.11% |
| humanities | 26.47% |
| stem | 26.09% |
| business | 25.86% |
| other | 19.64% |
| social_sciences | 17.65% |
**하위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| medical | 36.11% |
| humanities | 26.47% |
| stem | 26.09% |
| business | 25.86% |
| other | 19.64% |
| social_sciences | 17.65% |
## 영어 벤치마크
### 주요 벤치마크 (0-shot)
| 태스크 | Accuracy | Acc (norm) |
|--------|----------|-----------|
| hellaswag | 26.00% | 26.15% |
| arc_easy | 25.63% | 26.64% |
| arc_challenge | 21.67% | 27.90% |
| winogrande | 50.59% | N/A |
| piqa | 52.50% | 48.31% |
### MMLU-EN (0-shot)
평가된 과목 수: **61**
전체 평균: **25.82%**
**상위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| college_physics | 37.25% |
| college_computer_science | 34.00% |
| high_school_statistics | 33.80% |
| us_foreign_policy | 32.00% |
| security_studies | 31.43% |
| world_religions | 30.99% |
| professional_medicine | 30.88% |
| high_school_government_and_politics | 30.57% |
| jurisprudence | 30.56% |
| human_sexuality | 30.53% |
**하위 10개 과목**:
| 과목 | Accuracy |
|------|----------|
| moral_disputes | 22.54% |
| philosophy | 22.51% |
| college_chemistry | 22.00% |
| international_law | 21.49% |
| public_relations | 20.00% |
| human_aging | 19.73% |
| college_biology | 19.44% |
| anatomy | 17.04% |
| global_facts | 17.00% |
| abstract_algebra | 15.00% |
## 0-shot vs 5-shot 비교
| 태스크 | 0-shot Acc | 5-shot Acc | 변화 |
|--------|-----------|-----------|------|
| global_mmlu_ko | 24.00% | 20.75% | -3.25pp |
| global_mmlu_ko_business | 25.86% | 18.97% | -6.90pp |
| global_mmlu_ko_humanities | 26.47% | 24.51% | -1.96pp |
| global_mmlu_ko_medical | 36.11% | 27.78% | -8.33pp |
| global_mmlu_ko_other | 19.64% | 19.64% | +0.00pp |
| global_mmlu_ko_social_sciences | 17.65% | 18.63% | +0.98pp |
| global_mmlu_ko_stem | 26.09% | 15.22% | -10.87pp |
| haerae | 19.98% | 20.81% | +0.82pp |
| haerae_general_knowledge | 18.75% | 17.05% | -1.70pp |
| haerae_history | 20.74% | 17.02% | -3.72pp |
| haerae_loan_word | 14.79% | 24.26% | +9.47pp |
| haerae_rare_word | 21.73% | 22.96% | +1.23pp |
| haerae_standard_nomenclature | 21.57% | 20.26% | -1.31pp |
| kobest_hellaswag | 19.40% | 21.60% | +2.20pp |
| kobest_sentineg | 50.13% | 49.87% | -0.25pp |
| kobest_wic | 48.81% | 48.81% | +0.00pp |
평균 변화: -1.47pp | 개선: 5 | 하락: 9 | 동일: 2
## Repetition 파라미터 검색
| 설정 | Temp | Rep Pen | 3-gram | 4-gram | EOS Rate | Avg Tokens |
|------|------|---------|--------|--------|----------|-----------|
| t0.7_rep1.3 | 0.70 | 1.30 | 0.0000 | 0.0000 | 0.0000 | 256.0 | **← best**
| t0.9_rep1.2 | 0.90 | 1.20 | 0.0000 | 0.0000 | 0.0000 | 256.0 |
| t0.7_rep1.2 | 0.70 | 1.20 | 0.0088 | 0.0000 | 0.0000 | 256.0 |
| t0.9_rep1.1 | 0.90 | 1.10 | 0.0094 | 0.0013 | 0.0000 | 256.0 |
| t1.0_rep1.1 | 1.00 | 1.10 | 0.0121 | 0.0048 | 0.0000 | 256.0 |
| t0.5_rep1.1 | 0.50 | 1.10 | 0.0192 | 0.0119 | 0.0000 | 256.0 |
| t1.0 | 1.00 | 1.00 | 0.0358 | 0.0281 | 0.0000 | 256.0 |
| t0.9 | 0.90 | 1.00 | 0.0839 | 0.0464 | 0.0000 | 256.0 |
| t0.7_rep1.1 | 0.70 | 1.10 | 0.0851 | 0.0551 | 0.0000 | 256.0 |
| t0.7 | 0.70 | 1.00 | 0.4769 | 0.4340 | 0.0000 | 256.0 |
| t0.5 | 0.50 | 1.00 | 0.6012 | 0.5868 | 0.0000 | 256.0 |
| greedy | 0.00 | 1.00 | 0.6099 | 0.5702 | 0.0000 | 256.0 |
---
*이 리포트는 자동으로 생성되었습니다.*