==== STARTING EXPERIMENT: qwen3_1.7b_coding_fft ====
Log File:  saves/qwen3_1.7b/coding/fft/qwen3_1.7b_coding_fft_20260429_153648.log
HF Hub:    https://huggingface.co/KKHYA/qwen3-1.7b-fft-coding
Timestamp: 2026-04-29 15:36:48
=====================================
[INFO|2026-04-29 15:37:02] llamafactory.launcher:144 >> Initializing 8 distributed tasks at: 127.0.0.1:42841
W0429 15:37:04.319000 212778 site-packages/torch/distributed/run.py:852] 
W0429 15:37:04.319000 212778 site-packages/torch/distributed/run.py:852] *****************************************
W0429 15:37:04.319000 212778 site-packages/torch/distributed/run.py:852] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0429 15:37:04.319000 212778 site-packages/torch/distributed/run.py:852] *****************************************
warmup_ratio is deprecated and will be removed in v5.2. Use `warmup_steps` instead.
warmup_ratio is deprecated and will be removed in v5.2. Use `warmup_steps` instead.
warmup_ratio is deprecated and will be removed in v5.2. Use `warmup_steps` instead.
df: /root/.triton/autotune: No such file or directory
warmup_ratio is deprecated and will be removed in v5.2. Use `warmup_steps` instead.
warmup_ratio is deprecated and will be removed in v5.2. Use `warmup_steps` instead.
warmup_ratio is deprecated and will be removed in v5.2. Use `warmup_steps` instead.
warmup_ratio is deprecated and will be removed in v5.2. Use `warmup_steps` instead.
warmup_ratio is deprecated and will be removed in v5.2. Use `warmup_steps` instead.
[W429 15:37:18.649985073 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[W429 15:37:18.746117814 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[W429 15:37:18.778768535 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[W429 15:37:18.828110308 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[W429 15:37:18.868414499 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[W429 15:37:18.947969051 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ^docker0,lo
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO Bootstrap: Using eth0:10.200.170.96<0>
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO cudaDriverVersion 13000
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO cudaDriverVersion 13000
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ^docker0,lo
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO Bootstrap: Using eth0:10.200.170.96<0>
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NCCL version 2.28.9+cuda13.0
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NCCL version 2.28.9+cuda13.0
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO Comm config Blocking set to 1
[W429 15:37:18.174754892 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO cudaDriverVersion 13000
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ^docker0,lo
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO Bootstrap: Using eth0:10.200.170.96<0>
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NCCL version 2.28.9+cuda13.0
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO Comm config Blocking set to 1
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO cudaDriverVersion 13000
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO Successfully loaded external network plugin /opt/amazon/ofi-nccl/lib/libnccl-net.so
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NET/OFI Using Libfabric version 2.3
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NET/OFI Configuring AWS-specific options
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NET/OFI Setting provider_filter to efa
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NET/OFI Internode latency set at 75.0 us
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ^docker0,lo
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO Bootstrap: Using eth0:10.200.170.96<0>
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NCCL version 2.28.9+cuda13.0

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212862:212862 [1] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212862:212862 [1] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212862:212862 [1] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO plugin/net/net_v10.cc:81 -> 2
ywang29-p4d-debug-1-worker-0:212862:212862 [1] NCCL INFO Failed to initialize NET plugin Libfabric
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO Comm config Blocking set to 1
[W429 15:37:18.265962460 ProcessGroupNCCL.cpp:929] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO cudaDriverVersion 13000
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ^docker0,lo
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO Bootstrap: Using eth0:10.200.170.96<0>
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NCCL version 2.28.9+cuda13.0
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO Comm config Blocking set to 1
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO cudaDriverVersion 13000
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ^docker0,lo
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO Bootstrap: Using eth0:10.200.170.96<0>
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NCCL version 2.28.9+cuda13.0
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO Comm config Blocking set to 1
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO Comm config Blocking set to 1
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO Successfully loaded external network plugin /opt/amazon/ofi-nccl/lib/libnccl-net.so
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NET/OFI Using Libfabric version 2.3
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NET/OFI Configuring AWS-specific options
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NET/OFI Setting provider_filter to efa
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NET/OFI Internode latency set at 75.0 us
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO cudaDriverVersion 13000
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ^docker0,lo

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212863:212863 [2] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO Bootstrap: Using eth0:10.200.170.96<0>

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212863:212863 [2] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212863:212863 [2] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO plugin/net/net_v10.cc:81 -> 2
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NCCL version 2.28.9+cuda13.0
ywang29-p4d-debug-1-worker-0:212863:212863 [2] NCCL INFO Failed to initialize NET plugin Libfabric
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO Successfully loaded external network plugin /opt/amazon/ofi-nccl/lib/libnccl-net.so
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NET/OFI Using Libfabric version 2.3
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO Comm config Blocking set to 1
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NET/OFI Configuring AWS-specific options
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NET/OFI Setting provider_filter to efa
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NET/OFI Internode latency set at 75.0 us
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212867:212867 [6] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212867:212867 [6] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212867:212867 [6] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO plugin/net/net_v10.cc:81 -> 2
ywang29-p4d-debug-1-worker-0:212867:212867 [6] NCCL INFO Failed to initialize NET plugin Libfabric
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO cudaDriverVersion 13000
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ^docker0,lo
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO Bootstrap: Using eth0:10.200.170.96<0>
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NCCL version 2.28.9+cuda13.0
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO Comm config Blocking set to 1
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO Successfully loaded external network plugin /opt/amazon/ofi-nccl/lib/libnccl-net.so
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NET/OFI Using Libfabric version 2.3
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NET/OFI Configuring AWS-specific options
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NET/OFI Setting provider_filter to efa
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NET/OFI Internode latency set at 75.0 us
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212865:212865 [4] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212865:212865 [4] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212865:212865 [4] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO plugin/net/net_v10.cc:81 -> 2
ywang29-p4d-debug-1-worker-0:212865:212865 [4] NCCL INFO Failed to initialize NET plugin Libfabric
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO Successfully loaded external network plugin /opt/amazon/ofi-nccl/lib/libnccl-net.so
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NET/OFI Using Libfabric version 2.3
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NET/OFI Configuring AWS-specific options
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NET/OFI Setting provider_filter to efa
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NET/OFI Internode latency set at 75.0 us
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212868:212868 [7] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212868:212868 [7] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed

[2026-04-29 15:37:18] ywang29-p4d-debug-1-worker-0:212868:212868 [7] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO plugin/net/net_v10.cc:81 -> 2
ywang29-p4d-debug-1-worker-0:212868:212868 [7] NCCL INFO Failed to initialize NET plugin Libfabric
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO Successfully loaded external network plugin /opt/amazon/ofi-nccl/lib/libnccl-net.so
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NET/OFI Using Libfabric version 2.3
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NET/OFI Configuring AWS-specific options
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NET/OFI Setting provider_filter to efa
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NET/OFI Internode latency set at 75.0 us
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)

[2026-04-29 15:37:19] ywang29-p4d-debug-1-worker-0:212861:212861 [0] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol

[2026-04-29 15:37:19] ywang29-p4d-debug-1-worker-0:212861:212861 [0] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed

[2026-04-29 15:37:19] ywang29-p4d-debug-1-worker-0:212861:212861 [0] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO plugin/net/net_v10.cc:81 -> 2
ywang29-p4d-debug-1-worker-0:212861:212861 [0] NCCL INFO Failed to initialize NET plugin Libfabric
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO Successfully loaded external network plugin /opt/amazon/ofi-nccl/lib/libnccl-net.so
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NET/OFI Using Libfabric version 2.3
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NET/OFI Configuring AWS-specific options
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NET/OFI Setting provider_filter to efa
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NET/OFI Internode latency set at 75.0 us
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)

[2026-04-29 15:37:19] ywang29-p4d-debug-1-worker-0:212864:212864 [3] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol

[2026-04-29 15:37:19] ywang29-p4d-debug-1-worker-0:212864:212864 [3] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed

[2026-04-29 15:37:19] ywang29-p4d-debug-1-worker-0:212864:212864 [3] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO plugin/net/net_v10.cc:81 -> 2
ywang29-p4d-debug-1-worker-0:212864:212864 [3] NCCL INFO Failed to initialize NET plugin Libfabric
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO Successfully loaded external network plugin /opt/amazon/ofi-nccl/lib/libnccl-net.so
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NET/OFI Using Libfabric version 2.3
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NET/OFI Configuring AWS-specific options
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NET/OFI Setting provider_filter to efa
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NET/OFI Internode latency set at 75.0 us
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)

[2026-04-29 15:37:19] ywang29-p4d-debug-1-worker-0:212866:212866 [5] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol

[2026-04-29 15:37:19] ywang29-p4d-debug-1-worker-0:212866:212866 [5] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed

[2026-04-29 15:37:19] ywang29-p4d-debug-1-worker-0:212866:212866 [5] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO plugin/net/net_v10.cc:81 -> 2
ywang29-p4d-debug-1-worker-0:212866:212866 [5] NCCL INFO Failed to initialize NET plugin Libfabric