35 lines
1.4 KiB
Plaintext
35 lines
1.4 KiB
Plaintext
ОТЧЁТ ОБ ОЦЕНКЕ МОДЕЛИ Broken_Code_Generation.1.0
|
|
Репозиторий: Vilyam888/Broken_Code_Generation.1.0
|
|
Базовая модель: Qwen/Qwen2.5-Coder-3B-Instruct
|
|
Дата оценки: 2026-06-11
|
|
Выборка: test.json, N = 100 (hold-out test)
|
|
Генерация: temperature = 0.2, max_new_tokens = 1200
|
|
|
|
1. Perplexity (validation, checkpoint-501):
|
|
• train_loss_final: 0.1867
|
|
• eval_loss_final: 0.2523
|
|
• eval_mean_token_accuracy: 0.9323
|
|
• perplexity_validation: 1.29
|
|
• num_train_epochs: 3
|
|
• global_step: 501
|
|
|
|
По эпохам:
|
|
epoch 1.0: PPL=1.3247, eval_loss=0.2812, acc=0.9243
|
|
epoch 2.0: PPL=1.2856, eval_loss=0.2512, acc=0.9317
|
|
epoch 3.0: PPL=1.2869, eval_loss=0.2523, acc=0.9323
|
|
|
|
2. JSON validity (QLoRA vs baseline):
|
|
• valid_json_rate: 0.94 (baseline 0.78, Δ 0.16)
|
|
• required_fields_rate: 0.92 (baseline 0.74, Δ 0.18)
|
|
• difficulty_match_rate: 0.96 (baseline 0.85, Δ 0.11)
|
|
• topic_tag_key_match_rate: 0.97 (baseline 0.83, Δ 0.14)
|
|
|
|
3. BLEU / ROUGE (QLoRA vs baseline):
|
|
• bleu4_corpus: 0.68 (baseline 0.52, Δ 0.16)
|
|
• rouge1_f1: 0.73 (baseline 0.57, Δ 0.16)
|
|
• rouge2_f1: 0.58 (baseline 0.41, Δ 0.17)
|
|
• rougeL_f1: 0.71 (baseline 0.54, Δ 0.17)
|
|
|
|
4. Code metrics (поле broken_code):
|
|
• broken_code_syntax_valid_rate: 0.91
|
|
• codebleu_broken_code: 0.47 |