ОТЧЁТ ОБ ОЦЕНКЕ МОДЕЛИ Broken_Code_Generation.1.0
Репозиторий: Vilyam888/Broken_Code_Generation.1.0
Базовая модель: Qwen/Qwen2.5-Coder-3B-Instruct
Дата оценки: 2026-06-11
Выборка: test.json, N = 100 (hold-out test)
Генерация: temperature = 0.2, max_new_tokens = 1200

1. Perplexity (validation, checkpoint-501):
  • train_loss_final: 0.1867
  • eval_loss_final: 0.2523
  • eval_mean_token_accuracy: 0.9323
  • perplexity_validation: 1.29
  • num_train_epochs: 3
  • global_step: 501

  По эпохам:
    epoch 1.0: PPL=1.3247, eval_loss=0.2812, acc=0.9243
    epoch 2.0: PPL=1.2856, eval_loss=0.2512, acc=0.9317
    epoch 3.0: PPL=1.2869, eval_loss=0.2523, acc=0.9323

2. JSON validity (QLoRA vs baseline):
  • valid_json_rate: 0.94 (baseline 0.78, Δ 0.16)
  • required_fields_rate: 0.92 (baseline 0.74, Δ 0.18)
  • difficulty_match_rate: 0.96 (baseline 0.85, Δ 0.11)
  • topic_tag_key_match_rate: 0.97 (baseline 0.83, Δ 0.14)

3. BLEU / ROUGE (QLoRA vs baseline):
  • bleu4_corpus: 0.68 (baseline 0.52, Δ 0.16)
  • rouge1_f1: 0.73 (baseline 0.57, Δ 0.16)
  • rouge2_f1: 0.58 (baseline 0.41, Δ 0.17)
  • rougeL_f1: 0.71 (baseline 0.54, Δ 0.17)

4. Code metrics (поле broken_code):
  • broken_code_syntax_valid_rate: 0.91
  • codebleu_broken_code: 0.47