2025-04-11 12:07:59,921:amlt-code-runner:INFO - SINGULARITY_LOCATION: centralus
2025-04-11 12:07:59,921:amlt-code-runner:INFO - AISC_INSTANCE_TYPE: Singularity.ND96_v4
2025-04-11 12:08:03,058:amlt-code-runner:INFO - Not removing AzureML's cd commands from /etc/profile due to an error: [Errno 13] Permission denied: '/etc/profile'
2025-04-11 12:08:03,058:amlt-code-runner:WARNING - Environment variable 'NCCL_SOCKET_IFNAME' already set to '=eth0', not changing to '^docker0,lo'
2025-04-11 12:08:03,058:amlt-code-runner:INFO - RANK = 0
2025-04-11 12:08:03,058:amlt-code-runner:INFO - LOCAL_RANK = None
2025-04-11 12:08:03,058:amlt-code-runner:INFO - WORLD_SIZE = 1
2025-04-11 12:08:03,058:amlt-code-runner:INFO - MASTER_ADDR = node-0
2025-04-11 12:08:03,058:amlt-code-runner:INFO - MASTER_PORT = 9500
2025-04-11 12:08:03,059:amlt-code-runner:WARNING - Installing amlt runtime dependencies: ['wrapt', 'azure-identity', 'python-dateutil', 'pytz'] into /tmp/amlt-user-base
2025-04-11 12:08:04,628:amlt-code-runner:INFO - Setting WANDB_RUN_ID to 'sincere_feast_96p6v4qzfp_178'
2025-04-11 12:08:04,628:amlt-code-runner:INFO - Expanding HyperDrive arguments into /tmp/amlt_run_hd.sh
2025-04-11 12:08:04,975:amlt-code-runner:INFO - Parsing tracking uri /mlflow/v1.0/subscriptions/bbd59374-b76d-4cb5-88b6-2be35debc7cf/resourceGroups/jordan/providers/Microsoft.MachineLearningServices/workspaces/jordanws
2025-04-11 12:08:04,975:amlt-code-runner:INFO - Tracking uri /mlflow/v1.0/subscriptions/bbd59374-b76d-4cb5-88b6-2be35debc7cf/resourceGroups/jordan/providers/Microsoft.MachineLearningServices/workspaces/jordanws has sub id bbd59374-b76d-4cb5-88b6-2be35debc7cf, resource group jordan, and workspace jordanws
2025-04-11 12:08:04,975:aml_token_auth:WARNING - The AzureMLTokenAuthentication created will not be updated due to missing params. The token expires on 2025-04-30 23:33:23.

2025-04-11 12:08:04,977:urllib3.connectionpool:DEBUG - Starting new HTTPS connection (1): eastus.api.azureml.ms:443
2025-04-11 12:08:05,516:urllib3.connectionpool:DEBUG - https://eastus.api.azureml.ms:443 "POST /mlflow/v1.0/subscriptions/bbd59374-b76d-4cb5-88b6-2be35debc7cf/resourceGroups/jordan/providers/Microsoft.MachineLearningServices/workspaces/jordanws/api/2.0/mlflow/runs/set-tag HTTP/11" 200 3
2025-04-11 12:08:05,539:amlt-code-runner:INFO - Executing ./amlt_setup.sh, /tmp/amlt_run_hd.sh
2025-04-11 12:08:05,608:background_dirsync:INFO - Starting directory syncer from '/scratch/amlt_code/outputs' to '/mnt/output/projects/amlt_project/amlt-results/sincere_feast_96p6v4qzfp_178', every 30.000000s
2025-04-11 12:08:05,611:background_dirsync:INFO - Starting directory syncer from '/scratch/azureml/cr/j/7969abcf6d8f41c9ad8062370e0065b9/exe/wd/logs' to '/scratch/amlt_code/outputs/logs', every 30.000000s
