2025-03-02 06:12:28,166:amlt-code-runner:INFO - SINGULARITY_LOCATION: eastus
2025-03-02 06:12:28,166:amlt-code-runner:INFO - AISC_INSTANCE_TYPE: Singularity.NC96ad_A100_v4
2025-03-02 06:12:29,442:amlt-code-runner:INFO - Not removing AzureML's cd commands from /etc/profile due to an error: [Errno 13] Permission denied: '/etc/profile'
2025-03-02 06:12:29,442:amlt-code-runner:WARNING - Environment variable 'NCCL_SOCKET_IFNAME' already set to 'eth0', not changing to '^docker0,lo'
2025-03-02 06:12:29,442:amlt-code-runner:INFO - RANK = 0
2025-03-02 06:12:29,442:amlt-code-runner:INFO - LOCAL_RANK = None
2025-03-02 06:12:29,442:amlt-code-runner:INFO - WORLD_SIZE = 1
2025-03-02 06:12:29,442:amlt-code-runner:INFO - MASTER_ADDR = node-0
2025-03-02 06:12:29,442:amlt-code-runner:INFO - MASTER_PORT = 9500
2025-03-02 06:12:29,444:amlt-code-runner:WARNING - Installing amlt runtime dependencies: ['wrapt', 'azure-identity', 'python-dateutil', 'pytz'] into /tmp/amlt-user-base
2025-03-02 06:12:30,543:amlt-code-runner:INFO - Setting WANDB_RUN_ID to 'kind_onion_8sfvnlnwfk_11'
2025-03-02 06:12:30,543:amlt-code-runner:INFO - Expanding HyperDrive arguments into /tmp/amlt_run_hd.sh
2025-03-02 06:12:30,878:amlt-code-runner:INFO - Parsing tracking uri /mlflow/v1.0/subscriptions/2aac527a-de5a-4fe3-95e9-5c8b9d48ed62/resourceGroups/cyrilzhang/providers/Microsoft.MachineLearningServices/workspaces/cyrilzhangws
2025-03-02 06:12:30,879:amlt-code-runner:INFO - Tracking uri /mlflow/v1.0/subscriptions/2aac527a-de5a-4fe3-95e9-5c8b9d48ed62/resourceGroups/cyrilzhang/providers/Microsoft.MachineLearningServices/workspaces/cyrilzhangws has sub id 2aac527a-de5a-4fe3-95e9-5c8b9d48ed62, resource group cyrilzhang, and workspace cyrilzhangws
2025-03-02 06:12:30,879:aml_token_auth:WARNING - The AzureMLTokenAuthentication created will not be updated due to missing params. The token expires on 2025-03-20 19:29:14.

2025-03-02 06:12:30,881:urllib3.connectionpool:DEBUG - Starting new HTTPS connection (1): eastus.api.azureml.ms:443
2025-03-02 06:12:31,056:urllib3.connectionpool:DEBUG - https://eastus.api.azureml.ms:443 "POST /mlflow/v1.0/subscriptions/2aac527a-de5a-4fe3-95e9-5c8b9d48ed62/resourceGroups/cyrilzhang/providers/Microsoft.MachineLearningServices/workspaces/cyrilzhangws/api/2.0/mlflow/runs/set-tag HTTP/11" 403 0
2025-03-02 06:12:31,057:amlt-code-runner:WARNING - Failed to rename job according to the amulet job name template. Run 'amlt list' client side to set the display name according to the amulet job template name. The error we encountered was: Failed to update display name: 
2025-03-02 06:12:31,079:amlt-code-runner:INFO - Executing ./amlt_setup.sh, /tmp/amlt_run_hd.sh
2025-03-02 06:12:31,149:background_dirsync:INFO - Starting directory syncer from '/scratch/amlt_code/outputs' to '/mnt/output/projects/lmpref/amlt-results/kind_onion_8sfvnlnwfk_11', every 30.000000s
2025-03-02 06:12:31,153:background_dirsync:INFO - Starting directory syncer from '/scratch/azureml/cr/j/6e2dd4f9c91946289a7d190e71b9ae5a/exe/wd/logs' to '/scratch/amlt_code/outputs/logs', every 30.000000s
