07/16/2026 22:47:16 - INFO - __main__ - [RANK 0] Distributed environment: DistributedType.NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda

Mixed precision type: no

07/16/2026 22:47:16 - INFO - __main__ - [RANK 0] Training args:
{'data_dir': 'data/phone/deu-eng/tokenized_interleaved', 'model_name_or_path': None, 'config_name': 'gpt_base_config.json', 'tokenizer_name': 'nikitastheo/phonelm-phone-deu-eng-tokenizer', 'per_device_train_batch_size': 32, 'per_device_eval_batch_size': 32, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'max_steps': 40000, 'gradient_accumulation_steps': 1, 'lr_scheduler_type': <SchedulerType.LINEAR: 'linear'>, 'warmup_steps': 4000, 'output_dir': 'models/phonebabylm-phone-deu-eng-interleaved', 'seed': 42, 'model_type': None, 'block_size': None, 'push_to_hub': True, 'hub_model_id': 'nikitastheo/phonebabylm-phone-deu-eng-interleaved', 'hub_token': None, 'private': False, 'trust_remote_code': False, 'with_tracking': True, 'tracking_dir': None, 'report_to': 'wandb', 'restart_lr_at_switch': False, 'language_switch_steps': None, 'model_revision': 'main', 'config_overrides': None, 'use_slow_tokenizer': False, 'override_vocab_size': None, 'dtype': None, 'logging_steps': 10, 'cache_dir': None, 'eval_steps': 2000, 'max_grad_norm': 1.0, 'adam_epsilon': 1e-06, 'run_name': 'phonebabylm-phone-deu-eng-interleaved', 'max_train_samples': None, 'max_eval_samples': None}
07/16/2026 22:47:17 - INFO - transformers.configuration_utils - loading configuration file gpt_base_config.json
07/16/2026 22:47:17 - INFO - transformers.configuration_utils - Model config GPT2Config {
  "activation_function": "gelu",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 512,
  "n_embd": 768,
  "n_head": 12,
  "n_inner": 3072,
  "n_layer": 12,
  "n_positions": 512,
  "reorder_and_upcast_attn": false,
  "resid_pdrop": 0.1,
  "scale_attn_by_inverse_layer_idx": false,
  "scale_attn_weights": true,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "transformers_version": "4.57.6",
  "use_cache": true,
  "vocab_size": 1
}

07/16/2026 22:47:17 - INFO - transformers.tokenization_utils_base - loading file vocab.json from cache at /home/geofila/.cache/huggingface/hub/models--nikitastheo--phonelm-phone-deu-eng-tokenizer/snapshots/a82d9c161c2c55d913c3f8dae7f59fbdd01bd623/vocab.json
07/16/2026 22:47:17 - INFO - transformers.tokenization_utils_base - loading file merges.txt from cache at None
07/16/2026 22:47:17 - INFO - transformers.tokenization_utils_base - loading file tokenizer.json from cache at /home/geofila/.cache/huggingface/hub/models--nikitastheo--phonelm-phone-deu-eng-tokenizer/snapshots/a82d9c161c2c55d913c3f8dae7f59fbdd01bd623/tokenizer.json
07/16/2026 22:47:17 - INFO - transformers.tokenization_utils_base - loading file added_tokens.json from cache at None
07/16/2026 22:47:17 - INFO - transformers.tokenization_utils_base - loading file special_tokens_map.json from cache at /home/geofila/.cache/huggingface/hub/models--nikitastheo--phonelm-phone-deu-eng-tokenizer/snapshots/a82d9c161c2c55d913c3f8dae7f59fbdd01bd623/special_tokens_map.json
07/16/2026 22:47:17 - INFO - transformers.tokenization_utils_base - loading file tokenizer_config.json from cache at /home/geofila/.cache/huggingface/hub/models--nikitastheo--phonelm-phone-deu-eng-tokenizer/snapshots/a82d9c161c2c55d913c3f8dae7f59fbdd01bd623/tokenizer_config.json
07/16/2026 22:47:17 - INFO - transformers.tokenization_utils_base - loading file chat_template.jinja from cache at None
07/16/2026 22:47:19 - INFO - transformers.generation.configuration_utils - Generate config GenerationConfig {
  "bos_token_id": 2,
  "eos_token_id": 2,
  "pad_token_id": 1
}

07/16/2026 22:47:20 - INFO - __main__ - [RANK 0] Model parameters: 85.51M total (85.51M trainable)
07/16/2026 22:47:39 - INFO - __main__ - [RANK 0] Sample 167621 of the training set: {'input_ids': tensor([ 5, 11, 30, 24, 52,  5, 56, 15,  5, 26, 30, 13, 25,  3, 31,  5, 35, 54,
        22, 15, 25, 14,  5,  3, 16, 56, 15,  5, 45, 24, 30, 14,  5, 56, 15,  5,
        23, 52, 46, 18,  5, 22, 28,  8, 21, 14,  5, 54,  6, 56, 15,  5, 26, 28,
         8, 30,  3, 31,  5, 35, 54,  6, 15, 52, 25,  5, 45,  3, 56,  5, 19, 17,
        35, 26,  5, 11, 52, 54, 21,  5, 45,  3, 48,  5, 45, 71,  8,  5, 26, 15,
        26, 38, 32, 16, 25,  3, 25,  5, 11, 30,  3, 48,  3, 14,  5, 54,  6,  5,
        22, 46, 35, 15, 16,  5, 22, 54, 16, 48,  3, 14,  5, 54,  6,  5, 25, 52,
        13, 25,  5, 71,  8, 22, 14,  5, 26, 18, 52,  3, 31, 14,  5, 54,  6,  5,
        54, 16, 59, 16, 14,  5, 70, 16, 25,  5, 28,  8,  6, 15, 16,  5, 21,  3,
        26, 18, 15, 52,  5, 22,  3, 31, 35, 26,  3, 14,  5, 11,  3, 27,  3, 31,
        22, 34,  8, 18, 26,  5, 24, 30,  5, 45, 71,  8,  5, 16, 34,  8,  5, 18,
        15,  5, 19, 71,  8, 52,  5,  3, 16, 18, 52, 32, 26, 18,  3, 25,  5, 23,
        46, 14,  5, 70, 16, 25,  5, 19, 71,  8,  5, 35, 45,  3, 35,  5, 13, 14,
         5, 18, 17, 35,  5,  3, 16,  5, 32,  6, 52,  3, 60,  3, 31,  5, 11, 52,
        54, 21, 56, 15,  5, 23, 54, 16, 52, 70, 35,  5, 73,  6, 15, 52,  5, 56,
        15,  5, 25, 52, 32, 26, 15, 52,  5, 18, 15,  5, 56, 15,  5, 48, 13, 16,
        15,  5, 25, 72, 23, 14,  5, 24, 16, 56, 15,  5, 48,  3, 21, 16, 28, 38,
        28,  8, 26,  5, 56, 15,  5, 35,  3, 48, 15, 16,  5, 45, 54, 14,  5, 26,
        73,  5, 30, 72, 52, 49,  5, 56, 70, 18,  5, 19, 70, 11,  5, 54,  6,  5,
         3, 18,  5, 26, 28,  8, 21, 25,  5, 18, 15, 22, 28,  5, 52,  3, 14, 71,
         8,  6, 25,  5, 70, 14,  5, 54,  5, 38, 72, 52, 30, 15, 52,  5, 56, 32,
        52, 45, 54, 14,  5, 54,  5, 26, 35, 45, 32, 52,  5, 54,  6,  5, 35, 72,
        52, 38,  3, 18,  5, 30, 46, 25,  5, 25, 10, 16,  5, 70, 18,  5, 45, 54,
        16,  5, 32, 16, 25,  5, 15, 38, 72, 16,  5, 45,  3, 48,  5, 26, 18, 17,
        25,  5, 54,  5, 52, 10, 16, 25,  5, 18, 46, 22, 15, 30,  5, 26, 38, 52,
        32, 25,  5, 45,  3, 56,  5, 21,  3, 26,  3, 14,  5, 22,  3, 31, 35, 26,
         3, 14,  5,  6, 32, 52, 28,  5, 22, 32, 26, 18,  5, 48, 13, 16, 15,  5,
        18, 28,  8, 26, 71,  8,  6,  3, 26,  5, 70, 16, 25,  5, 54,  5, 26, 15,
        38, 30, 13,  5, 54,  6,  5, 25])} WORD_BOUNDARY f l ɔ ɹ WORD_BOUNDARY ð ə WORD_BOUNDARY s l aɪ d ɪ ŋ WORD_BOUNDARY k ʌ b ə d z WORD_BOUNDARY ɪ n ð ə WORD_BOUNDARY w ɔ l z WORD_BOUNDARY ð ə WORD_BOUNDARY ɡ ɹ eɪ t WORD_BOUNDARY b i ː m z WORD_BOUNDARY ʌ v ð ə WORD_BOUNDARY s i ː l ɪ ŋ WORD_BOUNDARY k ʌ v ə ɹ d WORD_BOUNDARY w ɪ ð WORD_BOUNDARY h ʊ k s WORD_BOUNDARY f ɹ ʌ m WORD_BOUNDARY w ɪ t̠ʃ WORD_BOUNDARY w ɜ ː WORD_BOUNDARY s ə s p ɛ n d ɪ d WORD_BOUNDARY f l ɪ t̠ʃ ɪ z WORD_BOUNDARY ʌ v WORD_BOUNDARY b eɪ k ə n WORD_BOUNDARY b ʌ n t̠ʃ ɪ z WORD_BOUNDARY ʌ v WORD_BOUNDARY d ɹ aɪ d WORD_BOUNDARY ɜ ː b z WORD_BOUNDARY s t ɹ ɪ ŋ z WORD_BOUNDARY ʌ v WORD_BOUNDARY ʌ n iə n z WORD_BOUNDARY æ n d WORD_BOUNDARY i ː v ə n WORD_BOUNDARY m ɪ s t ə ɹ WORD_BOUNDARY b ɪ ŋ k s ɪ z WORD_BOUNDARY f ɪ ʃ ɪ ŋ b u ː t s WORD_BOUNDARY ɔ l WORD_BOUNDARY w ɜ ː WORD_BOUNDARY n u ː WORD_BOUNDARY t ə WORD_BOUNDARY h ɜ ː ɹ WORD_BOUNDARY ɪ n t ɹ ɛ s t ɪ d WORD_BOUNDARY ɡ eɪ z WORD_BOUNDARY æ n d WORD_BOUNDARY h ɜ ː WORD_BOUNDARY k w ɪ k WORD_BOUNDARY aɪ z WORD_BOUNDARY t ʊ k WORD_BOUNDARY ɪ n WORD_BOUNDARY ɛ v ɹ ɪ θ ɪ ŋ WORD_BOUNDARY f ɹ ʌ m ð ə WORD_BOUNDARY ɡ ʌ n ɹ æ k WORD_BOUNDARY oʊ v ə ɹ WORD_BOUNDARY ð ə WORD_BOUNDARY d ɹ ɛ s ə ɹ WORD_BOUNDARY t ə WORD_BOUNDARY ð ə WORD_BOUNDARY t̠ʃ aɪ n ə WORD_BOUNDARY d ɑ ɡ z WORD_BOUNDARY ɔ n ð ə WORD_BOUNDARY t̠ʃ ɪ m n i p i ː s WORD_BOUNDARY ð ə WORD_BOUNDARY k ɪ t̠ʃ ə n WORD_BOUNDARY w ʌ z WORD_BOUNDARY s oʊ WORD_BOUNDARY l ɑ ɹ d̠ʒ WORD_BOUNDARY ð æ t WORD_BOUNDARY h æ f WORD_BOUNDARY ʌ v WORD_BOUNDARY ɪ t WORD_BOUNDARY s i ː m d WORD_BOUNDARY t ə b i WORD_BOUNDARY ɹ ɪ z ɜ ː v d WORD_BOUNDARY æ z WORD_BOUNDARY ʌ WORD_BOUNDARY p ɑ ɹ l ə ɹ WORD_BOUNDARY ð ɛ ɹ w ʌ z WORD_BOUNDARY ʌ WORD_BOUNDARY s k w ɛ ɹ WORD_BOUNDARY ʌ v WORD_BOUNDARY k ɑ ɹ p ɪ t WORD_BOUNDARY l eɪ d WORD_BOUNDARY d aʊ n WORD_BOUNDARY æ t WORD_BOUNDARY w ʌ n WORD_BOUNDARY ɛ n d WORD_BOUNDARY ə p ɑ n WORD_BOUNDARY w ɪ t̠ʃ WORD_BOUNDARY s t ʊ d WORD_BOUNDARY ʌ WORD_BOUNDARY ɹ aʊ n d WORD_BOUNDARY t eɪ b ə l WORD_BOUNDARY s p ɹ ɛ d WORD_BOUNDARY w ɪ ð WORD_BOUNDARY m ɪ s ɪ z WORD_BOUNDARY b ɪ ŋ k s ɪ z WORD_BOUNDARY v ɛ ɹ i WORD_BOUNDARY b ɛ s t WORD_BOUNDARY t̠ʃ aɪ n ə WORD_BOUNDARY t i ː s ɜ ː v ɪ s WORD_BOUNDARY æ n d WORD_BOUNDARY ʌ WORD_BOUNDARY s ə p l aɪ WORD_BOUNDARY ʌ v WORD_BOUNDARY d
07/16/2026 22:47:39 - INFO - __main__ - [RANK 0] Sample 29184 of the training set: {'input_ids': tensor([13, 16, 15,  5, 18,  7, 27, 15, 16,  5, 21,  3, 18,  5, 35, 17, 38, 11,
         9, 23, 32, 30, 18,  5, 11, 32,  9, 27, 30, 40,  8, 26,  5, 25, 28,  8,
         5, 35,  3, 26, 18, 15,  5, 14, 32, 20, 18, 15,  5, 25, 33,  8, 16,  5,
        19, 17, 16, 18,  5,  6, 28,  8, 25,  9,  5, 19,  3, 16, 10, 11,  5, 17,
        16, 18,  5, 23,  3, 31,  5,  3, 16,  5, 25,  7, 26,  5,  7, 16, 25, 15,
        12, 15,  5, 20,  3, 21,  9,  5, 38, 24, 20, 18, 10, 14, 15, 16, 18,  5,
        25,  7,  8,  5, 14,  7,  8, 26,  5, 25, 32,  9,  5, 19, 17, 16, 18,  5,
        21,  3, 18,  5, 10, 23, 15, 16,  5, 14, 40,  8,  5, 23,  9, 40,  8, 26,
         5,  6, 28,  8,  5, 21, 41,  8, 30, 12, 32,  8, 25,  9,  5,  2, 23, 30,
        24, 20,  5, 21,  3,  4,  5, 16,  3,  4, 18,  5, 14, 40,  8,  5,  7, 16,
         5, 14,  7,  8, 23, 18, 15,  5, 25, 32,  9,  5, 14, 24, 30, 25,  7,  8,
        18,  5, 17, 16, 18,  5, 14, 32, 20, 18, 15,  5, 25, 33,  8, 16,  5, 19,
        17, 16, 18,  5, 10, 11,  5, 25, 28,  8,  5, 27, 36,  9, 20, 15,  5,  7,
        30, 26,  5, 32,  9,  5,  7,  8, 22,  9,  5, 25,  7, 26,  5, 11, 28,  8,
        30, 15,  5, 14,  3, 30, 22,  9, 23, 32, 30, 18,  5, 14,  7,  8,  5,  6,
         7,  9, 11,  5, 32,  9,  5,  7, 30, 15, 26,  5, 35, 17, 38, 11,  9, 23,
        32, 30, 18,  5, 11, 24,  9, 18,  5, 17, 16, 18,  5, 11, 36, 30, 18, 15,
         5, 14,  3,  4,  5, 25, 28,  8,  5, 18,  7, 27, 15, 16,  5, 17, 16, 18,
         5, 25, 33,  8, 16,  5, 18, 24,  9, 16,  3, 26, 18,  9,  5, 21,  3, 18,
         5, 14,  3, 30, 22,  9,  5, 25,  7, 16,  5, 23,  3, 31,  5, 32,  9,  5,
         3, 16,  5, 25, 28,  8,  5, 25,  9,  3, 18, 15,  5, 35,  7, 21,  9,  5,
         6, 40,  8,  5, 25, 32,  9,  5, 19, 17, 16, 18,  5,  6,  7,  8,  9,  5,
        21,  3, 18,  5, 10, 23, 15, 16,  5, 14, 40,  8,  5, 23,  9, 40,  8, 26,
         5,  6, 28,  8,  5, 13, 16,  5, 12, 17, 16, 25,  9,  5, 18, 29, 21,  5,
         2, 23, 34,  8, 18, 15, 16,  5,  7,  8, 22, 15, 16, 18,  5, 14,  7,  8,
        23, 18, 15,  5, 25, 32,  9,  5, 14, 24, 30, 25,  7,  8, 18,  5, 19, 40,
         8, 38,  5, 25, 33,  8, 16,  5, 19, 17, 16, 18,  5, 19, 32, 12, 17, 16,
        18,  9,  5, 17, 16, 18,  5, 42, 11, 16, 15, 18, 15,  5, 25, 28,  8,  5,
        35,  3, 26, 18, 15,  5,  6,  7])} aɪ n ə WORD_BOUNDARY t a ʃ ə n WORD_BOUNDARY m ɪ t WORD_BOUNDARY k ʊ p f ɐ ɡ ɛ l t WORD_BOUNDARY f ɛ ɐ ʃ l o ː s WORD_BOUNDARY d i ː WORD_BOUNDARY k ɪ s t ə WORD_BOUNDARY z ɛ ts t ə WORD_BOUNDARY d e ː n WORD_BOUNDARY h ʊ n t WORD_BOUNDARY v i ː d ɐ WORD_BOUNDARY h ɪ n aʊ f WORD_BOUNDARY ʊ n t WORD_BOUNDARY ɡ ɪ ŋ WORD_BOUNDARY ɪ n WORD_BOUNDARY d a s WORD_BOUNDARY a n d ə ʀ ə WORD_BOUNDARY ts ɪ m ɐ WORD_BOUNDARY p ɔ ts t aʊ z ə n t WORD_BOUNDARY d a ː WORD_BOUNDARY z a ː s WORD_BOUNDARY d ɛ ɐ WORD_BOUNDARY h ʊ n t WORD_BOUNDARY m ɪ t WORD_BOUNDARY aʊ ɡ ə n WORD_BOUNDARY z o ː WORD_BOUNDARY ɡ ɐ o ː s WORD_BOUNDARY v i ː WORD_BOUNDARY m y ː l ʀ ɛ ː d ɐ WORD_BOUNDARY UTT_BOUNDARY ɡ l ɔ ts WORD_BOUNDARY m ɪ ç WORD_BOUNDARY n ɪ ç t WORD_BOUNDARY z o ː WORD_BOUNDARY a n WORD_BOUNDARY z a ː ɡ t ə WORD_BOUNDARY d ɛ ɐ WORD_BOUNDARY z ɔ l d a ː t WORD_BOUNDARY ʊ n t WORD_BOUNDARY z ɛ ts t ə WORD_BOUNDARY d e ː n WORD_BOUNDARY h ʊ n t WORD_BOUNDARY aʊ f WORD_BOUNDARY d i ː WORD_BOUNDARY ʃ ʏ ɐ ts ə WORD_BOUNDARY a l s WORD_BOUNDARY ɛ ɐ WORD_BOUNDARY a ː b ɐ WORD_BOUNDARY d a s WORD_BOUNDARY f i ː l ə WORD_BOUNDARY z ɪ l b ɐ ɡ ɛ l t WORD_BOUNDARY z a ː WORD_BOUNDARY v a ɐ f WORD_BOUNDARY ɛ ɐ WORD_BOUNDARY a l ə s WORD_BOUNDARY k ʊ p f ɐ ɡ ɛ l t WORD_BOUNDARY f ɔ ɐ t WORD_BOUNDARY ʊ n t WORD_BOUNDARY f ʏ l t ə WORD_BOUNDARY z ɪ ç WORD_BOUNDARY d i ː WORD_BOUNDARY t a ʃ ə n WORD_BOUNDARY ʊ n t WORD_BOUNDARY d e ː n WORD_BOUNDARY t ɔ ɐ n ɪ s t ɐ WORD_BOUNDARY m ɪ t WORD_BOUNDARY z ɪ l b ɐ WORD_BOUNDARY d a n WORD_BOUNDARY ɡ ɪ ŋ WORD_BOUNDARY ɛ ɐ WORD_BOUNDARY ɪ n WORD_BOUNDARY d i ː WORD_BOUNDARY d ɐ ɪ t ə WORD_BOUNDARY k a m ɐ WORD_BOUNDARY v o ː WORD_BOUNDARY d ɛ ɐ WORD_BOUNDARY h ʊ n t WORD_BOUNDARY v a ː ɐ WORD_BOUNDARY m ɪ t WORD_BOUNDARY aʊ ɡ ə n WORD_BOUNDARY z o ː WORD_BOUNDARY ɡ ɐ o ː s WORD_BOUNDARY v i ː WORD_BOUNDARY aɪ n WORD_BOUNDARY ʀ ʊ n d ɐ WORD_BOUNDARY t ʊɐ m WORD_BOUNDARY UTT_BOUNDARY ɡ u ː t ə n WORD_BOUNDARY a ː b ə n t WORD_BOUNDARY z a ː ɡ t ə WORD_BOUNDARY d ɛ ɐ WORD_BOUNDARY z ɔ l d a ː t WORD_BOUNDARY h o ː p WORD_BOUNDARY d e ː n WORD_BOUNDARY h ʊ n t WORD_BOUNDARY h ɛ ʀ ʊ n t ɐ WORD_BOUNDARY ʊ n t WORD_BOUNDARY œ f n ə t ə WORD_BOUNDARY d i ː WORD_BOUNDARY k ɪ s t ə WORD_BOUNDARY v a
07/16/2026 22:47:39 - INFO - __main__ - [RANK 0] Sample 6556 of the training set: {'input_ids': tensor([16,  5,  2,  6, 32,  8, 12, 15, 16, 18,  5, 25, 32,  9,  5, 27, 34,  8,
        30, 11, 33,  8, 12, 28,  8, 15, 16,  5, 11, 28,  8, 30,  5, 10, 39,  5,
        25, 28,  8,  5, 14, 24, 16, 18,  7, 35, 26,  4, 34,  8, 30, 15,  5, 10,
        26,  5, 18,  9, 24, 20, 25, 33,  8, 21,  5,  6,  7,  8,  9,  5, 19, 24,
        36, 18,  5,  7, 30, 15, 26,  5, 11, 12, 41,  8, 20, 13, 18,  3,  4,  5,
         3, 16,  5, 25, 32,  9,  5, 35,  3,  9,  4, 15,  5, 25,  7, 26,  5, 10,
        11, 12, 33,  8, 23, 15, 16, 25, 15,  5, 32,  9, 13, 23, 16,  3, 26,  5,
         6, 29, 25, 15,  5, 30, 33,  8, 38, 19,  7, 11, 18,  5, 32,  9, 42,  9,
        18,  9, 18,  5, 32, 26,  5,  6, 29, 25, 15,  5, 32,  9, 20, 32,  8, 30,
        18,  5, 25,  7, 26,  5, 16, 24, 39,  5, 35, 13, 16, 15,  5, 27, 38, 34,
         8,  9,  5, 11, 24, 16,  5, 25, 33,  8, 16,  5, 30,  7, 16, 18, 27, 18,
         9, 13,  4,  9, 16,  5, 23, 15, 11, 17, 16, 25, 15, 16,  5,  6, 24,  9,
        25, 15, 16,  5, 14, 13,  5,  7, 30, 26,  5, 25, 28,  8,  5, 38,  9, 33,
         8, 25,  3,  4, 18,  5, 20, 34,  8,  5, 32, 16, 25, 15,  5,  6,  7,  8,
         9,  5, 23,  3, 31,  5, 25, 28,  8,  5, 11, 12, 10,  5, 25, 32, 26,  5,
        12,  3,  4, 18,  9, 26,  5, 18,  7, 48,  9,  5, 10, 11,  5, 11, 12, 10,
         5, 19,  7,  9, 38,  9,  5, 20, 34,  8,  5, 25, 28,  8,  5, 21,  3, 18,
         5, 25, 32,  9,  5, 23,  9, 40,  8, 26, 15, 16,  5, 21, 32, 31, 15,  5,
        25, 33,  8, 16,  5, 23,  7, 31,  5, 19,  3, 16, 17, 16, 18,  9, 27, 12,
         3, 18,  5, 17, 16, 18,  5, 14,  7,  8, 23, 18, 15,  5,  6,  3, 30,  5,
        21, 13, 16, 15,  5, 22, 32, 35, 28,  8,  5, 25, 32, 16,  5, 25, 33,  8,
        16,  5, 23,  7, 16, 20, 15, 16,  5, 18,  7,  8, 35,  5, 27, 30,  7,  8,
        11, 15, 16,  5, 19,  7,  8, 38, 26,  5, 21, 28,  8,  9,  5,  7,  8, 22,
         9,  5,  6, 40,  8, 30,  5, 23, 15, 25,  7, 39, 18,  5, 25,  7, 26,  5,
        14, 28,  8,  5, 18, 24, 25, 21, 41,  8, 25, 15,  5, 14, 13, 16,  5,  6,
        36,  9, 25, 15,  5,  2, 28,  8, 12, 15,  5, 22, 32, 35, 28,  8,  5,  2,
        11, 12, 13, 30,  3,  4,  5, 21,  3, 18,  5, 32,  9, 27, 12, 24, 35, 15,
        16, 15, 21,  5, 22, 30,  3, 35,  5, 22, 30, 28,  8, 38,  5, 14, 28,  8,
         5, 25, 32, 16,  5,  7,  8, 22])} n WORD_BOUNDARY UTT_BOUNDARY v ɛ ː ʀ ə n t WORD_BOUNDARY d ɛ ɐ WORD_BOUNDARY ʃ u ː l f e ː ʀ i ː ə n WORD_BOUNDARY f i ː l WORD_BOUNDARY aʊ x WORD_BOUNDARY d i ː WORD_BOUNDARY z ɔ n t a k s ç u ː l ə WORD_BOUNDARY aʊ s WORD_BOUNDARY t ɐ ɔ ts d e ː m WORD_BOUNDARY v a ː ɐ WORD_BOUNDARY h ɔ ʏ t WORD_BOUNDARY a l ə s WORD_BOUNDARY f ʀ y ː ts aɪ t ɪ ç WORD_BOUNDARY ɪ n WORD_BOUNDARY d ɛ ɐ WORD_BOUNDARY k ɪ ɐ ç ə WORD_BOUNDARY d a s WORD_BOUNDARY aʊ f ʀ e ː ɡ ə n d ə WORD_BOUNDARY ɛ ɐ aɪ ɡ n ɪ s WORD_BOUNDARY v ʊɐ d ə WORD_BOUNDARY l e ː p h a f t WORD_BOUNDARY ɛ ɐ œ ɐ t ɐ t WORD_BOUNDARY ɛ s WORD_BOUNDARY v ʊɐ d ə WORD_BOUNDARY ɛ ɐ ts ɛ ː l t WORD_BOUNDARY d a s WORD_BOUNDARY n ɔ x WORD_BOUNDARY k aɪ n ə WORD_BOUNDARY ʃ p u ː ɐ WORD_BOUNDARY f ɔ n WORD_BOUNDARY d e ː n WORD_BOUNDARY l a n t ʃ t ɐ aɪ ç ɐ n WORD_BOUNDARY ɡ ə f ʊ n d ə n WORD_BOUNDARY v ɔ ɐ d ə n WORD_BOUNDARY z aɪ WORD_BOUNDARY a l s WORD_BOUNDARY d i ː WORD_BOUNDARY p ɐ e ː d ɪ ç t WORD_BOUNDARY ts u ː WORD_BOUNDARY ɛ n d ə WORD_BOUNDARY v a ː ɐ WORD_BOUNDARY ɡ ɪ ŋ WORD_BOUNDARY d i ː WORD_BOUNDARY f ʀ aʊ WORD_BOUNDARY d ɛ s WORD_BOUNDARY ʀ ɪ ç t ɐ s WORD_BOUNDARY t a t̠ʃ ɐ WORD_BOUNDARY aʊ f WORD_BOUNDARY f ʀ aʊ WORD_BOUNDARY h a ɐ p ɐ WORD_BOUNDARY ts u ː WORD_BOUNDARY d i ː WORD_BOUNDARY m ɪ t WORD_BOUNDARY d ɛ ɐ WORD_BOUNDARY ɡ ɐ o ː s ə n WORD_BOUNDARY m ɛ ŋ ə WORD_BOUNDARY d e ː n WORD_BOUNDARY ɡ a ŋ WORD_BOUNDARY h ɪ n ʊ n t ɐ ʃ ʀ ɪ t WORD_BOUNDARY ʊ n t WORD_BOUNDARY z a ː ɡ t ə WORD_BOUNDARY v ɪ l WORD_BOUNDARY m aɪ n ə WORD_BOUNDARY b ɛ k i ː WORD_BOUNDARY d ɛ n WORD_BOUNDARY d e ː n WORD_BOUNDARY ɡ a n ts ə n WORD_BOUNDARY t a ː k WORD_BOUNDARY ʃ l a ː f ə n WORD_BOUNDARY h a ː p s WORD_BOUNDARY m i ː ɐ WORD_BOUNDARY a ː b ɐ WORD_BOUNDARY v o ː l WORD_BOUNDARY ɡ ə d a x t WORD_BOUNDARY d a s WORD_BOUNDARY z i ː WORD_BOUNDARY t ɔ d m y ː d ə WORD_BOUNDARY z aɪ n WORD_BOUNDARY v ʏ ɐ d ə WORD_BOUNDARY UTT_BOUNDARY i ː ʀ ə WORD_BOUNDARY b ɛ k i ː WORD_BOUNDARY UTT_BOUNDARY f ʀ aɪ l ɪ ç WORD_BOUNDARY m ɪ t WORD_BOUNDARY ɛ ɐ ʃ ʀ ɔ k ə n ə m WORD_BOUNDARY b l ɪ k WORD_BOUNDARY b l i ː p WORD_BOUNDARY z i ː WORD_BOUNDARY d ɛ n WORD_BOUNDARY a ː b
07/16/2026 22:47:42 - INFO - __main__ - [RANK 0] ***** Running training *****
07/16/2026 22:47:42 - INFO - __main__ - [RANK 0]   Num examples = 203590
07/16/2026 22:47:42 - INFO - __main__ - [RANK 0]   Num Training Steps = 40000
07/16/2026 22:47:42 - INFO - __main__ - [RANK 0]   Instantaneous batch size per device = 32
07/16/2026 22:47:42 - INFO - __main__ - [RANK 0]   Total train batch size (w. parallel, distributed & accumulation) = 32
07/16/2026 22:47:42 - INFO - __main__ - [RANK 0]   Gradient Accumulation steps = 1
07/16/2026 22:47:42 - INFO - __main__ - [RANK 0]   Total optimization steps = 40000
07/16/2026 22:47:42 - INFO - transformers.configuration_utils - Configuration saved in models/phonebabylm-phone-deu-eng-interleaved/config.json
07/16/2026 22:47:42 - INFO - transformers.generation.configuration_utils - Configuration saved in models/phonebabylm-phone-deu-eng-interleaved/generation_config.json
07/16/2026 22:47:45 - INFO - transformers.modeling_utils - Model weights saved in models/phonebabylm-phone-deu-eng-interleaved/model.safetensors
07/16/2026 22:47:45 - INFO - transformers.tokenization_utils_base - tokenizer config file saved in models/phonebabylm-phone-deu-eng-interleaved/tokenizer_config.json
07/16/2026 22:47:45 - INFO - transformers.tokenization_utils_base - Special tokens file saved in models/phonebabylm-phone-deu-eng-interleaved/special_tokens_map.json
07/16/2026 22:47:50 - INFO - __main__ - [RANK 0] Starting epoch 0...
07/16/2026 22:47:51 - WARNING - transformers.modeling_utils - `loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
07/16/2026 22:58:50 - INFO - __main__ - [RANK 0]  perplexity: 3.4981748920090556 eval_loss: 1.2522413730621338 accuracy: 0.6286
07/16/2026 23:09:55 - INFO - __main__ - [RANK 0]  perplexity: 3.090224323166524 eval_loss: 1.1282436847686768 accuracy: 0.6635
07/16/2026 23:20:59 - INFO - __main__ - [RANK 0]  perplexity: 2.8887235844293055 eval_loss: 1.0608147382736206 accuracy: 0.6829
07/16/2026 23:22:58 - INFO - __main__ - [RANK 0] Starting epoch 1...
07/16/2026 23:32:04 - INFO - __main__ - [RANK 0]  perplexity: 2.7866070328443007 eval_loss: 1.0248247385025024 accuracy: 0.6932
07/16/2026 23:43:09 - INFO - __main__ - [RANK 0]  perplexity: 2.717227754278576 eval_loss: 0.9996121525764465 accuracy: 0.7002
07/16/2026 23:54:14 - INFO - __main__ - [RANK 0]  perplexity: 2.662045793634297 eval_loss: 0.979094922542572 accuracy: 0.7061
07/16/2026 23:58:13 - INFO - __main__ - [RANK 0] Starting epoch 2...
07/17/2026 00:05:19 - INFO - __main__ - [RANK 0]  perplexity: 2.6255481989944776 eval_loss: 0.9652897119522095 accuracy: 0.7102
07/17/2026 00:16:23 - INFO - __main__ - [RANK 0]  perplexity: 2.5913564344248097 eval_loss: 0.9521814584732056 accuracy: 0.7138
07/17/2026 00:27:28 - INFO - __main__ - [RANK 0]  perplexity: 2.5649778532126026 eval_loss: 0.9419498443603516 accuracy: 0.7169
07/17/2026 00:33:26 - INFO - __main__ - [RANK 0] Starting epoch 3...
07/17/2026 00:38:43 - INFO - __main__ - [RANK 0]  perplexity: 2.5450176439772125 eval_loss: 0.9341375827789307 accuracy: 0.7196
07/17/2026 00:50:44 - INFO - __main__ - [RANK 0]  perplexity: 2.523804941363755 eval_loss: 0.9257676601409912 accuracy: 0.7216
07/17/2026 01:02:02 - INFO - __main__ - [RANK 0]  perplexity: 2.504542042072227 eval_loss: 0.9181059002876282 accuracy: 0.7241
07/17/2026 01:10:00 - INFO - __main__ - [RANK 0] Starting epoch 4...
07/17/2026 01:13:06 - INFO - __main__ - [RANK 0]  perplexity: 2.4924423233459936 eval_loss: 0.9132630825042725 accuracy: 0.7257
07/17/2026 01:24:11 - INFO - __main__ - [RANK 0]  perplexity: 2.4772802757896057 eval_loss: 0.907161295413971 accuracy: 0.7277
07/17/2026 01:35:16 - INFO - __main__ - [RANK 0]  perplexity: 2.4638050764913904 eval_loss: 0.9017069339752197 accuracy: 0.7294
07/17/2026 01:45:13 - INFO - __main__ - [RANK 0] Starting epoch 5...
07/17/2026 01:46:20 - INFO - __main__ - [RANK 0]  perplexity: 2.4591889317422853 eval_loss: 0.8998315930366516 accuracy: 0.7302
07/17/2026 01:57:25 - INFO - __main__ - [RANK 0]  perplexity: 2.44883489662295 eval_loss: 0.895612359046936 accuracy: 0.7316
07/17/2026 02:08:31 - INFO - __main__ - [RANK 0]  perplexity: 2.4370661854301496 eval_loss: 0.8907949328422546 accuracy: 0.7328
07/17/2026 02:19:36 - INFO - __main__ - [RANK 0]  perplexity: 2.4286724169445133 eval_loss: 0.8873447775840759 accuracy: 0.7338
07/17/2026 02:20:35 - INFO - __main__ - [RANK 0] Starting epoch 6...
07/17/2026 02:30:41 - INFO - __main__ - [RANK 0]  perplexity: 2.432660355583435 eval_loss: 0.8889854550361633 accuracy: 0.7341