初始化项目,由ModelHub XC社区提供模型
Model: Vilyam888/Broken_Code_Generation.1.0 Source: Original Platform
This commit is contained in:
35
metrics/evaluation_report.txt
Normal file
35
metrics/evaluation_report.txt
Normal file
@@ -0,0 +1,35 @@
|
||||
ОТЧЁТ ОБ ОЦЕНКЕ МОДЕЛИ Broken_Code_Generation.1.0
|
||||
Репозиторий: Vilyam888/Broken_Code_Generation.1.0
|
||||
Базовая модель: Qwen/Qwen2.5-Coder-3B-Instruct
|
||||
Дата оценки: 2026-06-11
|
||||
Выборка: test.json, N = 100 (hold-out test)
|
||||
Генерация: temperature = 0.2, max_new_tokens = 1200
|
||||
|
||||
1. Perplexity (validation, checkpoint-501):
|
||||
• train_loss_final: 0.1867
|
||||
• eval_loss_final: 0.2523
|
||||
• eval_mean_token_accuracy: 0.9323
|
||||
• perplexity_validation: 1.29
|
||||
• num_train_epochs: 3
|
||||
• global_step: 501
|
||||
|
||||
По эпохам:
|
||||
epoch 1.0: PPL=1.3247, eval_loss=0.2812, acc=0.9243
|
||||
epoch 2.0: PPL=1.2856, eval_loss=0.2512, acc=0.9317
|
||||
epoch 3.0: PPL=1.2869, eval_loss=0.2523, acc=0.9323
|
||||
|
||||
2. JSON validity (QLoRA vs baseline):
|
||||
• valid_json_rate: 0.94 (baseline 0.78, Δ 0.16)
|
||||
• required_fields_rate: 0.92 (baseline 0.74, Δ 0.18)
|
||||
• difficulty_match_rate: 0.96 (baseline 0.85, Δ 0.11)
|
||||
• topic_tag_key_match_rate: 0.97 (baseline 0.83, Δ 0.14)
|
||||
|
||||
3. BLEU / ROUGE (QLoRA vs baseline):
|
||||
• bleu4_corpus: 0.68 (baseline 0.52, Δ 0.16)
|
||||
• rouge1_f1: 0.73 (baseline 0.57, Δ 0.16)
|
||||
• rouge2_f1: 0.58 (baseline 0.41, Δ 0.17)
|
||||
• rougeL_f1: 0.71 (baseline 0.54, Δ 0.17)
|
||||
|
||||
4. Code metrics (поле broken_code):
|
||||
• broken_code_syntax_valid_rate: 0.91
|
||||
• codebleu_broken_code: 0.47
|
||||
Reference in New Issue
Block a user