--- library_name: transformers license: llama3.1 base_model: meta-llama/Llama-3.1-8B-Instruct tags: - generated_from_trainer datasets: - cfierro/othello-snake-llama3-fixed-prefix - cfierro/othello-snake-llama3-fixed-prefix - cfierro/othello-snake-llama3-fixed-prefix - cfierro/othello-snake-llama3-fixed-prefix model-index: - name: scratch/project/eu-26-55/knowledge-ft/axolotl/models/llama-3.1-8b-fft-othello-snake-fixed-prefix-2e-5 results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.10.0` ```yaml base_model: meta-llama/Llama-3.1-8B-Instruct model_type: LlamaForCausalLM tokenizer_type: AutoTokenizer load_in_8bit: false load_in_4bit: false # --- Dataset: continued pre-training with a masked constant prefix --- # type: input_output gives per-segment loss masking (template-free). The dataset # stores `segments`: the prefix "<|begin_of_text|>Game of bjk\n\n" is label:false # (attended to as context, but NOT in the loss); the space-separated moves + trailing # <|end_of_text|> are label:true (trained), like a Llama-3 pre-training document. # Built by data/push_input_output_dataset.py. # Trains on ALL four cumulative shards (full 50k-game scale). Listing multiple # dataset entries is the reliable way to combine them — Axolotl concatenates them. # To train at a smaller scale, delete trailing entries (shards are cumulative): # 25k -> keep first three; 10k -> keep first two; 5k -> keep only the first. # (Concise alternative, if your Axolotl forwards split arithmetic to load_dataset: # a single entry with split: games_0_5k+games_5k_10k+games_10k_25k+games_25k_50k) datasets: - path: cfierro/othello-snake-llama3-fixed-prefix type: input_output split: games_0_5k - path: cfierro/othello-snake-llama3-fixed-prefix type: input_output split: games_5k_10k - path: cfierro/othello-snake-llama3-fixed-prefix type: input_output split: games_10k_25k - path: cfierro/othello-snake-llama3-fixed-prefix type: input_output split: games_25k_50k train_on_inputs: false # REQUIRED for input_output masking to take effect dataset_prepared_path: /scratch/project/eu-26-55/knowledge-ft/axolotl/datasets/llama-3.1-8b/othello-snake-llama3-fixed-prefix val_set_size: 0.02 output_dir: /scratch/project/eu-26-55/knowledge-ft/axolotl/models/llama-3.1-8b-fft-othello-snake-fixed-prefix-2e-5 sequence_len: 1024 # games are ~200 tokens max; long context is unused here sample_packing: true # pack many short games per sequence (block-diagonal attn) eval_sample_packing: false # No LoRA — full fine-tuning wandb_project: othello-snake-ft wandb_entity: cfierro wandb_watch: wandb_name: llama-3.1-8b-fft-fixed-prefix-2e-5 wandb_log_model: "false" # --- Multi-GPU: 4x A40 (48GB) per node --- # Full FT of an 8B model needs ZeRO-3 to shard weights+grads+optimizer. # Effective batch = 4 GPUs * micro 4 * grad_accum 1 = 8 packed sequences. # At seq_len 1024 the 50k games pack into ~5k sequences -> ~320 steps/epoch # (~960 steps over 3 epochs). gradient_accumulation_steps: 1 micro_batch_size: 4 num_epochs: 3 #max_steps: 500 # Checkpoint + evaluate once per epoch. (Axolotl-idiomatic alternative if your # version prefers it: saves_per_epoch: 1 / evals_per_epoch: 1.) save_strategy: epoch eval_strategy: epoch # uses val_set_size 0.02 (else the val split is never evaluated) optimizer: adamw_bnb_8bit lr_scheduler: cosine learning_rate: 2e-5 bf16: auto tf32: false gradient_checkpointing: true resume_from_checkpoint: logging_steps: 1 flash_attention: true warmup_ratio: 0.03 save_total_limit: 3 # keep all per-epoch checkpoints weight_decay: 0.0 special_tokens: pad_token: <|end_of_text|> # DeepSpeed ZeRO Stage 3 — shards weights, gradients, and optimizer across GPUs deepspeed: deepspeed_configs/zero3.json # Verify the loss mask before training: # axolotl preprocess axolotl_configs/fullft-othello-snake-8b.yaml --debug # Confirm the prefix tokens show -100 (masked) and the move tokens + <|end_of_text|> are trained. ```

# scratch/project/eu-26-55/knowledge-ft/axolotl/models/llama-3.1-8b-fft-othello-snake-fixed-prefix-2e-5 This model is a fine-tuned version of [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) on the cfierro/othello-snake-llama3-fixed-prefix, the cfierro/othello-snake-llama3-fixed-prefix, the cfierro/othello-snake-llama3-fixed-prefix and the cfierro/othello-snake-llama3-fixed-prefix datasets. It achieves the following results on the evaluation set: - Loss: 0.2668 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 4 - eval_batch_size: 4 - seed: 42 - distributed_type: multi-GPU - num_devices: 4 - total_train_batch_size: 16 - total_eval_batch_size: 16 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 28 - training_steps: 951 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | No log | 0 | 0 | 1.4982 | | 0.2808 | 1.0 | 317 | 0.2806 | | 0.2629 | 2.0 | 634 | 0.2679 | | 0.2543 | 3.0 | 951 | 0.2668 | ### Framework versions - Transformers 4.57.3 - Pytorch 2.9.0+cu128 - Datasets 3.5.0 - Tokenizers 0.22.2