--- library_name: transformers license: mit base_model: unsloth/phi-4 tags: - generated_from_trainer datasets: - shisa-ai/shisa-v2-best-of-n-athenev2-tulu70b-llama33-only-no-sysprompt - shisa-ai/shisa-v2-roleplaying-sft - shisa-ai/translation_expanded_master_set_filtered - shisa-ai/rewild-set - shisa-ai/magpie-ultra-set - shisa-ai/magpie-advanced-questions-set - shisa-ai/japan-magpie-set model-index: - name: outputs/ablation-114-bestofn.atlo.rp.tlx.newmix2.unphi-shisa-v2-unphi-4-14b results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.8.0.dev0` ```yaml base_model: unsloth/phi-4 model_type: AutoModelForCausalLM tokenizer_type: AutoTokenizer load_in_8bit: false load_in_4bit: false strict: false # User Liger plugins: - axolotl.integrations.liger.LigerPlugin liger_rope: true liger_rms_norm: true liger_glu_activation: true liger_fused_linear_cross_entropy: true chat_template: tokenizer_default datasets: - path: shisa-ai/shisa-v2-best-of-n-athenev2-tulu70b-llama33-only-no-sysprompt type: chat_template field_messages: conversations message_field_role: from message_field_content: value - path: shisa-ai/shisa-v2-roleplaying-sft type: chat_template field_messages: conversations message_property_mappings: role: role content: content roles: system: - system assistant: - gpt - model - assistant user: - human - user roles_to_train: ["assistant"] - path: shisa-ai/translation_expanded_master_set_filtered split: train[:25%] type: chat_template field_messages: conversations message_property_mappings: role: role content: content roles: system: - system assistant: - gpt - model - assistant user: - human - user roles_to_train: ["assistant"] - path: shisa-ai/rewild-set split: train[:5%] type: chat_template field_messages: conversations message_property_mappings: role: role content: content roles: system: - system assistant: - gpt - model - assistant user: - human - user roles_to_train: ["assistant"] - path: shisa-ai/magpie-ultra-set split: train[:8%] type: chat_template field_messages: conversations message_property_mappings: role: role content: content roles: system: - system assistant: - gpt - model - assistant user: - human - user roles_to_train: ["assistant"] - path: shisa-ai/magpie-advanced-questions-set split: train[:8%] type: chat_template field_messages: conversations message_property_mappings: role: role content: content roles: system: - system assistant: - gpt - model - assistant user: - human - user roles_to_train: ["assistant"] - path: shisa-ai/japan-magpie-set split: train type: chat_template field_messages: conversations message_property_mappings: role: role content: content roles: system: - system assistant: - gpt - model - assistant user: - human - user roles_to_train: ["assistant"] dataset_prepared_path: last_run_prepared val_set_size: 0.05 output_dir: ./outputs/ablation-114-bestofn.atlo.rp.tlx.newmix2.unphi-shisa-v2-unphi-4-14b sequence_len: 8192 sample_packing: true pad_to_sequence_len: true # marginal difference neftune_noise_alpha: 5 use_wandb: true wandb_project: shisa-v2 wandb_entity: augmxnt wandb_name: ablation-114-bestofn.atlo.rp.tlx.newmix2.unphi-shisa-v2-unphi-4-14b gradient_accumulation_steps: 1 micro_batch_size: 4 num_epochs: 3 optimizer: paged_adamw_8bit lr_scheduler: linear learning_rate: 5e-6 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: false early_stopping_patience: resume_from_checkpoint: logging_steps: 1 xformers_attention: flash_attention: true warmup_ratio: 0.1 evals_per_epoch: 4 eval_table_size: saves_per_epoch: 0 save_total_limit: 1 # Only store a single checkpoint debug: deepspeed: zero3_bf16.json weight_decay: 0.0001 fsdp: fsdp_config: special_tokens: # pad_token: "<|dummy_87|>" ```

# outputs/ablation-114-bestofn.atlo.rp.tlx.newmix2.unphi-shisa-v2-unphi-4-14b This model is a fine-tuned version of [unsloth/phi-4](https://huggingface.co/unsloth/phi-4) on the shisa-ai/shisa-v2-best-of-n-athenev2-tulu70b-llama33-only-no-sysprompt, the shisa-ai/shisa-v2-roleplaying-sft, the shisa-ai/translation_expanded_master_set_filtered, the shisa-ai/rewild-set, the shisa-ai/magpie-ultra-set, the shisa-ai/magpie-advanced-questions-set and the shisa-ai/japan-magpie-set datasets. It achieves the following results on the evaluation set: - Loss: 0.4720 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-06 - train_batch_size: 4 - eval_batch_size: 4 - seed: 42 - distributed_type: multi-GPU - num_devices: 16 - total_train_batch_size: 64 - total_eval_batch_size: 64 - optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 296 - num_epochs: 3.0 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 0.7211 | 0.0010 | 1 | 0.7194 | | 0.542 | 0.2508 | 248 | 0.5293 | | 0.4985 | 0.5015 | 496 | 0.5020 | | 0.4755 | 0.7523 | 744 | 0.4898 | | 0.4368 | 1.0030 | 992 | 0.4833 | | 0.4605 | 1.2538 | 1240 | 0.4796 | | 0.4388 | 1.5046 | 1488 | 0.4758 | | 0.4494 | 1.7553 | 1736 | 0.4723 | | 0.3682 | 2.0061 | 1984 | 0.4753 | | 0.4104 | 2.2568 | 2232 | 0.4742 | | 0.3903 | 2.5076 | 2480 | 0.4732 | | 0.3818 | 2.7583 | 2728 | 0.4720 | ### Framework versions - Transformers 4.50.0 - Pytorch 2.6.0+cu124 - Datasets 3.4.1 - Tokenizers 0.21.1