初始化项目，由ModelHub XC社区提供模型
Model: pathcosmos/frankenstallm Source: Original Platform
This commit is contained in:
ModelHub XC
@@ -0,0 +1,59 @@
|
||||
# FRANKENSTALLM 3B 종합 평가 리포트
|
||||
|
||||
- **모델**: FRANKENSTALLM 3B
|
||||
- **체크포인트**: checkpoint-0057000
|
||||
- **평가 일시**: 2026-03-05 03:27:39
|
||||
- **총 소요 시간**: 240.4초
|
||||
|
||||
## Executive Summary
|
||||
|
||||
| 메트릭 | 값 |
|
||||
|--------|-----|
|
||||
| 주요 PPL (3b_val) | 데이터 없음 |
|
||||
| KMMLU 평균 정확도 | 데이터 없음 |
|
||||
| KoBEST 평균 | 데이터 없음 |
|
||||
| Top-1 정확도 (Calibration) | 데이터 없음 |
|
||||
|
||||
## 3. Perplexity 평가
|
||||
|
||||
데이터 없음
|
||||
|
||||
## 4. Calibration 결과
|
||||
|
||||
데이터 없음
|
||||
|
||||
## 5. Token NLL 분포
|
||||
|
||||
데이터 없음
|
||||
|
||||
## 6. 생성 품질
|
||||
|
||||
데이터 없음
|
||||
|
||||
## 7. Repetition 파라미터 검색
|
||||
|
||||
데이터 없음
|
||||
|
||||
## 8. 표준 벤치마크
|
||||
|
||||
데이터 없음
|
||||
|
||||
## 9. 참고 모델 비교
|
||||
|
||||
| 모델 | 파라미터 | MMLU (ko) | KoBEST 평균 | PPL |
|
||||
|------|---------|-----------|------------|-----|
|
||||
| FRANKENSTALLM 3B | 3B | 데이터 없음 | 데이터 없음 | 데이터 없음 |
|
||||
| Llama-3.2-3B | 3B | ~42 | ~55 | — |
|
||||
| Qwen2.5-3B | 3B | ~48 | ~60 | — |
|
||||
| EXAONE-3.5-2.4B | 2.4B | ~35 | ~50 | — |
|
||||
|
||||
## 10. 컴퓨팅 자원 통계
|
||||
|
||||
| Phase | Task | 소요 시간(s) | 상태 |
|
||||
|-------|------|------------|------|
|
||||
| Phase 2 | Standard Benchmarks | - | 완료 |
|
||||
| **전체** | **모든 평가** | **240.4** | **완료** |
|
||||
|
||||
---
|
||||
|
||||
*이 리포트는 자동으로 생성되었습니다.*
|
||||
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"GPU 5 — Calibration + Token NLL": {
|
||||
"calibration": {
|
||||
"n_eval_tokens": 144802,
|
||||
"top1_accuracy": 0.6875,
|
||||
"top5_accuracy": 0.8164,
|
||||
"top10_accuracy": 0.8593,
|
||||
"mean_correct_prob": 0.6152,
|
||||
"mean_entropy": 1.5682,
|
||||
"elapsed_sec": 3.8
|
||||
},
|
||||
"token_nll": {
|
||||
"n_eval_tokens": 144802,
|
||||
"nll_mean": 1.5561,
|
||||
"nll_std": 2.4926,
|
||||
"nll_median": 0.1221,
|
||||
"nll_percentiles": {
|
||||
"p5": 0.0,
|
||||
"p25": 0.0017,
|
||||
"p75": 2.3594,
|
||||
"p95": 7.0312,
|
||||
"p99": 10.3125
|
||||
},
|
||||
"high_loss_fraction_5": 0.108617,
|
||||
"high_loss_fraction_10": 0.011823,
|
||||
"elapsed_sec": 4.8
|
||||
}
|
||||
},
|
||||
"GPU 6 — Generation (15 prompts × 4 temps)": {
|
||||
"error": "Traceback (most recent call last):\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\nconcurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.\n"
|
||||
},
|
||||
"GPU 7 — Repetition grid (12 × 5)": {
|
||||
"error": "Traceback (most recent call last):\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\nconcurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.\n"
|
||||
},
|
||||
"GPU 2 — PPL: 3b_val.bin": {
|
||||
"error": "Traceback (most recent call last):\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\nconcurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.\n"
|
||||
},
|
||||
"GPU 3 — PPL: korean_c4 + korean_val": {
|
||||
"error": "Traceback (most recent call last):\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\nconcurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.\n"
|
||||
},
|
||||
"GPU 4 — PPL: hplt_ko + cc100_ko + PPL: 7 cosmo files + PPL: 7 remaining files": {
|
||||
"error": "Traceback (most recent call last):\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\n File \"/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/full_eval_pipeline.py\", line 569, in run_phase1\n result = fut.result()\n ^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result\n raise self._exception\nconcurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.\n"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a27791a5c580c2434339c793c06a6fa64d79d3f000005641a071adefb752d1c7
|
||||
size 13711309
|
||||
Reference in New Issue
Block a user