1188 lines
1.3 MiB
Plaintext
1188 lines
1.3 MiB
Plaintext
|
The following values were not passed to `accelerate launch` and had defaults used instead:
|
||
|
|
More than one GPU was found, enabling multi-GPU training.
|
||
|
|
If this was unintended please pass in `--num_processes=1`.
|
||
|
|
`--dynamo_backend` was set to a value of `'no'`
|
||
|
|
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
|
||
|
|
2026-04-30:10:53:43 INFO [_cli.run:358] [rank=4] non-main process detected, suppressing wandb logging.
|
||
|
|
2026-04-30:10:53:43 INFO [_cli.run:358] [rank=6] non-main process detected, suppressing wandb logging.
|
||
|
|
2026-04-30:10:53:43 INFO [_cli.run:358] [rank=1] non-main process detected, suppressing wandb logging.
|
||
|
|
2026-04-30:10:53:43 INFO [_cli.run:358] [rank=3] non-main process detected, suppressing wandb logging.
|
||
|
|
2026-04-30:10:53:43 INFO [_cli.run:358] [rank=5] non-main process detected, suppressing wandb logging.
|
||
|
|
2026-04-30:10:53:43 INFO [_cli.run:358] [rank=7] non-main process detected, suppressing wandb logging.
|
||
|
|
2026-04-30:10:53:43 INFO [_cli.run:358] [rank=2] non-main process detected, suppressing wandb logging.
|
||
|
|
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
|
||
|
|
wandb: Currently logged in as: kkhya (maskmoe) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
||
|
|
wandb: Tracking run with wandb version 0.26.1
|
||
|
|
wandb: Run data is saved locally in /nfs/ywang29/lm-factory/wandb/run-20260430_105344-jm4o6xvg
|
||
|
|
wandb: Run `wandb offline` to turn off syncing.
|
||
|
|
wandb: Syncing run fft_coding
|
||
|
|
wandb: ⭐️ View project at https://wandb.ai/maskmoe/MFT-LM-eval
|
||
|
|
wandb: 🚀 View run at https://wandb.ai/maskmoe/MFT-LM-eval/runs/jm4o6xvg
|
||
|
|
2026-04-30:10:54:46 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'humaneval_plus', 'mbpp', 'mbpp_plus']
|
||
|
|
2026-04-30:10:54:46 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'humaneval_plus', 'mbpp', 'mbpp_plus']
|
||
|
|
2026-04-30:10:54:46 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'humaneval_plus', 'mbpp', 'mbpp_plus']
|
||
|
|
2026-04-30:10:54:46 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'humaneval_plus', 'mbpp', 'mbpp_plus']
|
||
|
|
2026-04-30:10:54:46 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'humaneval_plus', 'mbpp', 'mbpp_plus']
|
||
|
|
2026-04-30:10:54:46 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'humaneval_plus', 'mbpp', 'mbpp_plus']
|
||
|
|
2026-04-30:10:54:46 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'humaneval_plus', 'mbpp', 'mbpp_plus']
|
||
|
|
2026-04-30:10:54:46 INFO [_cli.run:388] Selected Tasks: ['humaneval', 'humaneval_plus', 'mbpp', 'mbpp_plus']
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:213] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:238] Initializing hf model, with arguments: {'pretrained': 'saves/qwen3_14b/coding/fft', 'dtype': 'bfloat16', 'parallelize': False, 'trust_remote_code': True, 'max_length': 4096}
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:213] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:238] Initializing hf model, with arguments: {'pretrained': 'saves/qwen3_14b/coding/fft', 'dtype': 'bfloat16', 'parallelize': False, 'trust_remote_code': True, 'max_length': 4096}
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:213] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:238] Initializing hf model, with arguments: {'pretrained': 'saves/qwen3_14b/coding/fft', 'dtype': 'bfloat16', 'parallelize': False, 'trust_remote_code': True, 'max_length': 4096}
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:213] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:238] Initializing hf model, with arguments: {'pretrained': 'saves/qwen3_14b/coding/fft', 'dtype': 'bfloat16', 'parallelize': False, 'trust_remote_code': True, 'max_length': 4096}
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:213] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:238] Initializing hf model, with arguments: {'pretrained': 'saves/qwen3_14b/coding/fft', 'dtype': 'bfloat16', 'parallelize': False, 'trust_remote_code': True, 'max_length': 4096}
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:213] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:238] Initializing hf model, with arguments: {'pretrained': 'saves/qwen3_14b/coding/fft', 'dtype': 'bfloat16', 'parallelize': False, 'trust_remote_code': True, 'max_length': 4096}
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:213] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:238] Initializing hf model, with arguments: {'pretrained': 'saves/qwen3_14b/coding/fft', 'dtype': 'bfloat16', 'parallelize': False, 'trust_remote_code': True, 'max_length': 4096}
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:213] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
|
|
2026-04-30:10:54:47 INFO [evaluator:238] Initializing hf model, with arguments: {'pretrained': 'saves/qwen3_14b/coding/fft', 'dtype': 'bfloat16', 'parallelize': False, 'trust_remote_code': True, 'max_length': 4096}
|
||
|
|
2026-04-30:10:55:00 INFO [models.huggingface:273] Using `accelerate launch` or `parallelize=True`, device 'cuda:0' will be overridden when placing model.
|
||
|
|
2026-04-30:10:55:00 INFO [models.huggingface:273] Using `accelerate launch` or `parallelize=True`, device 'cuda:0' will be overridden when placing model.
|
||
|
|
2026-04-30:10:55:00 INFO [models.huggingface:273] Using `accelerate launch` or `parallelize=True`, device 'cuda:0' will be overridden when placing model.
|
||
|
|
2026-04-30:10:55:00 INFO [models.huggingface:273] Using `accelerate launch` or `parallelize=True`, device 'cuda:0' will be overridden when placing model.
|
||
|
|
2026-04-30:10:55:00 INFO [models.huggingface:273] Using `accelerate launch` or `parallelize=True`, device 'cuda:0' will be overridden when placing model.
|
||
|
|
2026-04-30:10:55:00 INFO [models.huggingface:273] Using `accelerate launch` or `parallelize=True`, device 'cuda:0' will be overridden when placing model.
|
||
|
|
2026-04-30:10:55:00 INFO [models.huggingface:273] Using `accelerate launch` or `parallelize=True`, device 'cuda:0' will be overridden when placing model.
|
||
|
|
2026-04-30:10:55:00 INFO [models.huggingface:273] Using `accelerate launch` or `parallelize=True`, device 'cuda:0' will be overridden when placing model.
|
||
|
|
2026-04-30:10:55:01 INFO [models.huggingface:524] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:2'}
|
||
|
|
2026-04-30:10:55:02 INFO [models.huggingface:524] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:3'}
|
||
|
|
2026-04-30:10:55:02 INFO [models.huggingface:524] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:7'}
|
||
|
|
2026-04-30:10:55:02 INFO [models.huggingface:524] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:4'}
|
||
|
|
2026-04-30:10:55:02 INFO [models.huggingface:524] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:1'}
|
||
|
|
2026-04-30:10:55:02 INFO [models.huggingface:524] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:6'}
|
||
|
|
2026-04-30:10:55:02 INFO [models.huggingface:524] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:5'}
|
||
|
|
2026-04-30:10:55:02 INFO [models.huggingface:524] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
|
||
|
|
Loading weights: 0%| | 0/443 [00:00<?, ?it/s]
Loading weights: 0%| | 0/443 [00:00<?, ?it/s]
Loading weights: 0%| | 0/443 [00:00<?, ?it/s]
Loading weights: 0%| | 0/443 [00:00<?, ?it/s]
Loading weights: 0%| | 1/443 [00:00<00:00, 8289.14it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 5282.50it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 8473.34it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 6159.04it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 5391.14it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 4275.54it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 0/443 [00:00<?, ?it/s]
Loading weights: 0%| | 1/443 [00:00<00:00, 13443.28it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 8886.24it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 5562.74it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 0/443 [00:00<?, ?it/s]
Loading weights: 0%| | 1/443 [00:00<00:00, 3134.76it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 9383.23it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 5874.38it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 0/443 [00:00<?, ?it/s]
Loading weights: 0%| | 1/443 [00:00<00:00, 8035.07it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 5184.55it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 0/443 [00:00<?, ?it/s]
Loading weights: 0%| | 1/443 [00:00<00:00, 8065.97it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 1/443 [00:00<00:00, 4718.00it/s, Materializing param=lm_head.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=lm_head.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=lm_head.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=lm_head.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=lm_head.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=lm_head.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=lm_head.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=lm_head.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 2/443 [00:19<1:10:14, 9.56s/it, Materializing param=model.embed_tokens.weight]
Loading weigh
|
||
|
|
Loading weights: 100%|██████████| 443/443 [03:31<00:00, 2.09it/s, Materializing param=model.norm.weight]
|
||
|
|
Loading weights: 100%|██████████| 443/443 [03:31<00:00, 2.09it/s, Materializing param=model.norm.weight]
|
||
|
|
Loading weights: 100%|██████████| 443/443 [03:31<00:00, 2.09it/s, Materializing param=model.norm.weight]
|
||
|
|
Loading weights: 100%|██████████| 443/443 [03:31<00:00, 2.09it/s, Materializing param=model.norm.weight]
|
||
|
|
Loading weights: 100%|██████████| 443/443 [03:31<00:00, 2.09it/s, Materializing param=model.norm.weight]
|
||
|
|
Loading weights: 100%|██████████| 443/443 [03:31<00:00, 2.09it/s, Materializing param=model.norm.weight]
|
||
|
|
Loading weights: 100%|██████████| 443/443 [03:31<00:00, 2.09it/s, Materializing param=model.norm.weight]
|
||
|
|
Downloading builder script: 0.00B [00:00, ?B/s]
Downloading builder script: 9.18kB [00:00, 27.9MB/s]
|
||
|
|
Downloading extra modules: 0.00B [00:00, ?B/s]
Downloading extra modules: 6.10kB [00:00, 31.6MB/s]
|
||
|
|
Generating test split: 0%| | 0/164 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 164/164 [00:00<00:00, 36447.09 examples/s]
|
||
|
|
Generating test split: 0%| | 0/164 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 164/164 [00:00<00:00, 3259.33 examples/s]
|
||
|
|
Generating train split: 0%| | 0/374 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 374/374 [00:00<00:00, 55912.09 examples/s]
|
||
|
|
Generating test split: 0%| | 0/500 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 500/500 [00:00<00:00, 170264.84 examples/s]
|
||
|
|
Generating validation split: 0%| | 0/90 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 90/90 [00:00<00:00, 47554.47 examples/s]
|
||
|
|
Generating prompt split: 0%| | 0/10 [00:00<?, ? examples/s]
Generating prompt split: 100%|██████████| 10/10 [00:00<00:00, 5596.14 examples/s]
|
||
|
|
Generating test split: 0%| | 0/378 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 378/378 [00:00<00:00, 18723.24 examples/s]
|
||
|
|
2026-04-30:10:58:47 INFO [evaluator_utils:446] Selected tasks:
|
||
|
|
2026-04-30:10:58:47 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
|
||
|
|
2026-04-30:10:58:47 INFO [evaluator_utils:480] Task: humaneval_plus (humaneval/humaneval_plus.yaml)
|
||
|
|
2026-04-30:10:58:47 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
|
||
|
|
2026-04-30:10:58:47 INFO [evaluator_utils:480] Task: mbpp_plus (mbpp/mbpp_plus.yaml)
|
||
|
|
2026-04-30:10:58:47 INFO [evaluator:307] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:47 INFO [evaluator:307] humaneval_plus: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:47 INFO [evaluator:307] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:47 INFO [evaluator:307] mbpp_plus: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:47 INFO [api.task:312] Building contexts for humaneval on rank 5...
|
||
|
|
0%| | 0/20 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 1323.69it/s]
|
||
|
|
2026-04-30:10:58:50 INFO [evaluator_utils:446] Selected tasks:
|
||
|
|
2026-04-30:10:58:50 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
|
||
|
|
2026-04-30:10:58:50 INFO [evaluator_utils:480] Task: humaneval_plus (humaneval/humaneval_plus.yaml)
|
||
|
|
2026-04-30:10:58:50 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
|
||
|
|
2026-04-30:10:58:50 INFO [evaluator_utils:480] Task: mbpp_plus (mbpp/mbpp_plus.yaml)
|
||
|
|
2026-04-30:10:58:50 INFO [evaluator:307] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:50 INFO [evaluator:307] humaneval_plus: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:50 INFO [evaluator:307] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:50 INFO [evaluator:307] mbpp_plus: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:50 INFO [api.task:312] Building contexts for humaneval on rank 0...
|
||
|
|
0%| | 0/21 [00:00<?, ?it/s]
100%|██████████| 21/21 [00:00<00:00, 1198.91it/s]
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:39120 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:39120 [0] NCCL INFO Bootstrap: Using eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:39120 [0] NCCL INFO cudaDriverVersion 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:39125 [5] NCCL INFO cudaDriverVersion 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:39120 [0] NCCL INFO NCCL version 2.27.7+cuda13.0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:39125 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:39125 [5] NCCL INFO Bootstrap: Using eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:39125 [5] NCCL INFO NCCL version 2.27.7+cuda13.0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:39120 [0] NCCL INFO Comm config Blocking set to 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:39125 [5] NCCL INFO Comm config Blocking set to 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v10 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v7 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Successfully loaded external plugin libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/OFI Using Libfabric version 2.3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v10 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v7 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO Successfully loaded external plugin libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/OFI Using Libfabric version 2.3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/OFI Configuring AWS-specific options
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/OFI Setting provider_filter to efa
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/OFI Internode latency set at 75.0 us
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/OFI Configuring AWS-specific options
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/OFI Setting provider_filter to efa
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/OFI Internode latency set at 75.0 us
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)
|
||
|
|
|
||
|
|
[2026-04-30 10:58:50] ywang29-p4d-debug-2-worker-0:39120:40089 [0] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol
|
||
|
|
|
||
|
|
[2026-04-30 10:58:50] ywang29-p4d-debug-2-worker-0:39120:40089 [0] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed
|
||
|
|
|
||
|
|
[2026-04-30 10:58:50] ywang29-p4d-debug-2-worker-0:39120:40089 [0] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
|
||
|
|
|
||
|
|
[2026-04-30 10:58:50] ywang29-p4d-debug-2-worker-0:39125:40091 [5] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol
|
||
|
|
|
||
|
|
[2026-04-30 10:58:50] ywang29-p4d-debug-2-worker-0:39125:40091 [5] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed
|
||
|
|
|
||
|
|
[2026-04-30 10:58:50] ywang29-p4d-debug-2-worker-0:39125:40091 [5] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/IB : No device found.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/IB : No device found.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/Socket : Using [0]eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/Socket : Using [0]eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Initialized NET plugin Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO Initialized NET plugin Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO Assigned NET plugin Socket to comm
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Assigned NET plugin Socket to comm
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO Using network Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Using network Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO ncclCommInitRankConfig comm 0x55cf3325aa10 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 101c0 commId 0xdd4996e97ba7d4db - Init START
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO ncclCommInitRankConfig comm 0x5584194aa180 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 901d0 commId 0xdd4996e97ba7d4db - Init START
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:446] Selected tasks:
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval_plus (humaneval/humaneval_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp_plus (mbpp/mbpp_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval_plus: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp_plus: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [api.task:312] Building contexts for humaneval on rank 7...
|
||
|
|
0%| | 0/20 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 1294.30it/s]
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:39127 [7] NCCL INFO cudaDriverVersion 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:39127 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:39127 [7] NCCL INFO Bootstrap: Using eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:39127 [7] NCCL INFO NCCL version 2.27.7+cuda13.0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:39127 [7] NCCL INFO Comm config Blocking set to 1
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:446] Selected tasks:
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval_plus (humaneval/humaneval_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp_plus (mbpp/mbpp_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval_plus: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp_plus: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [api.task:312] Building contexts for humaneval on rank 3...
|
||
|
|
0%| | 0/21 [00:00<?, ?it/s]
100%|██████████| 21/21 [00:00<00:00, 1287.93it/s]
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:39123 [3] NCCL INFO cudaDriverVersion 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:39123 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:39123 [3] NCCL INFO Bootstrap: Using eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:39123 [3] NCCL INFO NCCL version 2.27.7+cuda13.0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:39123 [3] NCCL INFO Comm config Blocking set to 1
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:446] Selected tasks:
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval_plus (humaneval/humaneval_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp_plus (mbpp/mbpp_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval_plus: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp_plus: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [api.task:312] Building contexts for humaneval on rank 4...
|
||
|
|
0%| | 0/20 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 1297.48it/s]
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:39124 [4] NCCL INFO cudaDriverVersion 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:39124 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:39124 [4] NCCL INFO Bootstrap: Using eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:39124 [4] NCCL INFO NCCL version 2.27.7+cuda13.0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:39124 [4] NCCL INFO Comm config Blocking set to 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v10 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v7 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO Successfully loaded external plugin libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/OFI Using Libfabric version 2.3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/OFI Configuring AWS-specific options
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/OFI Setting provider_filter to efa
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/OFI Internode latency set at 75.0 us
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39127:40099 [7] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39127:40099 [7] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39127:40099 [7] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/IB : No device found.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/Socket : Using [0]eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO Initialized NET plugin Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO Assigned NET plugin Socket to comm
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO Using network Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO ncclCommInitRankConfig comm 0x5570936d9f40 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId a01d0 commId 0xdd4996e97ba7d4db - Init START
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v10 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v7 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO Successfully loaded external plugin libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/OFI Using Libfabric version 2.3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/OFI Configuring AWS-specific options
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/OFI Setting provider_filter to efa
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/OFI Internode latency set at 75.0 us
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39123:40101 [3] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39123:40101 [3] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39123:40101 [3] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/IB : No device found.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/Socket : Using [0]eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO Initialized NET plugin Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO Assigned NET plugin Socket to comm
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO Using network Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO ncclCommInitRankConfig comm 0x55ea64d13050 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 201d0 commId 0xdd4996e97ba7d4db - Init START
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v10 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v7 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO Successfully loaded external plugin libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/OFI Using Libfabric version 2.3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/OFI Configuring AWS-specific options
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/OFI Setting provider_filter to efa
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/OFI Internode latency set at 75.0 us
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39124:40105 [4] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39124:40105 [4] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39124:40105 [4] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/IB : No device found.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/Socket : Using [0]eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO Initialized NET plugin Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO Assigned NET plugin Socket to comm
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO Using network Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO ncclCommInitRankConfig comm 0x557c784ec2c0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 901c0 commId 0xdd4996e97ba7d4db - Init START
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO RAS client listening socket at ::1<28028>
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:446] Selected tasks:
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval_plus (humaneval/humaneval_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp_plus (mbpp/mbpp_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval_plus: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp_plus: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [api.task:312] Building contexts for humaneval on rank 2...
|
||
|
|
0%| | 0/21 [00:00<?, ?it/s]
100%|██████████| 21/21 [00:00<00:00, 1314.48it/s]
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:39122 [2] NCCL INFO cudaDriverVersion 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:39122 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:39122 [2] NCCL INFO Bootstrap: Using eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:39122 [2] NCCL INFO NCCL version 2.27.7+cuda13.0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:39122 [2] NCCL INFO Comm config Blocking set to 1
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:446] Selected tasks:
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval_plus (humaneval/humaneval_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp_plus (mbpp/mbpp_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval_plus: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp_plus: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [api.task:312] Building contexts for humaneval on rank 1...
|
||
|
|
0%| | 0/21 [00:00<?, ?it/s]
100%|██████████| 21/21 [00:00<00:00, 1289.42it/s]
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:39121 [1] NCCL INFO cudaDriverVersion 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:39121 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:39121 [1] NCCL INFO Bootstrap: Using eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:39121 [1] NCCL INFO NCCL version 2.27.7+cuda13.0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:39121 [1] NCCL INFO Comm config Blocking set to 1
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:446] Selected tasks:
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval (humaneval/humaneval.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: humaneval_plus (humaneval/humaneval_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp (mbpp/mbpp.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator_utils:480] Task: mbpp_plus (mbpp/mbpp_plus.yaml)
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] humaneval_plus: Using gen_kwargs: {'until': ['\nclass', '\ndef', '\n#', '\nif', '\nprint'], 'max_gen_toks': 1024, 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [evaluator:307] mbpp_plus: Using gen_kwargs: {'until': ['[DONE]'], 'do_sample': False}
|
||
|
|
2026-04-30:10:58:51 INFO [api.task:312] Building contexts for humaneval on rank 6...
|
||
|
|
0%| | 0/20 [00:00<?, ?it/s]ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v10 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v7 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO Successfully loaded external plugin libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/OFI Using Libfabric version 2.3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/OFI Configuring AWS-specific options
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/OFI Setting provider_filter to efa
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/OFI Internode latency set at 75.0 us
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)
|
||
|
|
100%|██████████| 20/20 [00:00<00:00, 1191.07it/s]
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39122:40113 [2] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39122:40113 [2] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39122:40113 [2] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/IB : No device found.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:39126 [6] NCCL INFO cudaDriverVersion 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/Socket : Using [0]eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO Initialized NET plugin Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO Assigned NET plugin Socket to comm
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO Using network Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:39126 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:39126 [6] NCCL INFO Bootstrap: Using eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:39126 [6] NCCL INFO NCCL version 2.27.7+cuda13.0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:39126 [6] NCCL INFO Comm config Blocking set to 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO ncclCommInitRankConfig comm 0x556db8b8dd00 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 201c0 commId 0xdd4996e97ba7d4db - Init START
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO RAS client listening socket at ::1<28028>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v10 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v7 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO Successfully loaded external plugin libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/OFI Using Libfabric version 2.3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/OFI Configuring AWS-specific options
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/OFI Setting provider_filter to efa
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/OFI Internode latency set at 75.0 us
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39121:40117 [1] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39121:40117 [1] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39121:40117 [1] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/IB : No device found.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/Socket : Using [0]eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO Initialized NET plugin Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO Assigned NET plugin Socket to comm
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO Using network Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO ncclCommInitRankConfig comm 0x556f2a64bf80 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 101d0 commId 0xdd4996e97ba7d4db - Init START
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO RAS client listening socket at ::1<28028>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO RAS client listening socket at ::1<28028>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO RAS client listening socket at ::1<28028>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/Plugin: Plugin name set by env to libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/Plugin: Loaded net plugin Libfabric (v10)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v10 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v7 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO Successfully loaded external plugin libnccl-net.so
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.17.1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/OFI Using Libfabric version 2.3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/OFI Using CUDA driver version 13000 with runtime 13000
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/OFI Configuring AWS-specific options
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/OFI Setting provider_filter to efa
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/OFI Running on p4de.24xlarge platform, topology file /opt/amazon/ofi-nccl/share/aws-ofi-nccl/xml/p4de-24xl-topo.xml
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/OFI Internode latency set at 75.0 us
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/OFI Using transport protocol SENDRECV (platform set)
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39126:40121 [6] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):208 NCCL WARN NET/OFI Failed to initialize sendrecv protocol
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39126:40121 [6] int nccl_net_ofi_create_plugin(nccl_net_ofi_plugin_t**):353 NCCL WARN NET/OFI aws-ofi-nccl initialization failed
|
||
|
|
|
||
|
|
[2026-04-30 10:58:51] ywang29-p4d-debug-2-worker-0:39126:40121 [6] ncclResult_t nccl_net_ofi_init_v6(ncclDebugLogger_t):162 NCCL WARN NET/OFI Initializing plugin failed
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/IB : No device found.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/Socket : Using [0]eth0:10.200.143.174<0>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO Initialized NET plugin Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO Assigned NET plugin Socket to comm
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO Using network Socket
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO ncclCommInitRankConfig comm 0x559361b695d0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId a01c0 commId 0xdd4996e97ba7d4db - Init START
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO RAS client listening socket at ::1<28028>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO RAS client listening socket at ::1<28028>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO RAS client listening socket at ::1<28028>
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO Bootstrap timings total 0.000963 (create 0.000041, send 0.000076, recv 0.000220, ring 0.000207, delay 0.000002)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO Bootstrap timings total 0.596544 (create 0.000045, send 0.000095, recv 0.000129, ring 0.000120, delay 0.000001)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO Bootstrap timings total 0.821776 (create 0.000040, send 0.000156, recv 0.820908, ring 0.000117, delay 0.000001)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO Bootstrap timings total 0.181152 (create 0.000046, send 0.000079, recv 0.000191, ring 0.050217, delay 0.000001)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO Bootstrap timings total 0.459399 (create 0.000043, send 0.000077, recv 0.000260, ring 0.458590, delay 0.000001)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Bootstrap timings total 0.821808 (create 0.000040, send 0.000179, recv 0.770821, ring 0.050200, delay 0.000002)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO Bootstrap timings total 0.570657 (create 0.000038, send 0.000069, recv 0.111426, ring 0.180242, delay 0.000001)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO Bootstrap timings total 0.051126 (create 0.000047, send 0.000088, recv 0.000246, ring 0.050217, delay 0.000002)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO Setting affinity for GPU 3 to 0-23,48-71
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NVLS multicast support is not available on dev 3 (NVLS_NCHANNELS 0)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO Setting affinity for GPU 2 to 0-23,48-71
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NVLS multicast support is not available on dev 2 (NVLS_NCHANNELS 0)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO Setting affinity for GPU 4 to 24-47,72-95
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NVLS multicast support is not available on dev 4 (NVLS_NCHANNELS 0)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO Setting affinity for GPU 1 to 0-23,48-71
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NVLS multicast support is not available on dev 1 (NVLS_NCHANNELS 0)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO Setting affinity for GPU 5 to 24-47,72-95
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NVLS multicast support is not available on dev 5 (NVLS_NCHANNELS 0)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Setting affinity for GPU 0 to 0-23,48-71
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO Setting affinity for GPU 6 to 24-47,72-95
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NVLS multicast support is not available on dev 6 (NVLS_NCHANNELS 0)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO Setting affinity for GPU 7 to 24-47,72-95
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NVLS multicast support is not available on dev 7 (NVLS_NCHANNELS 0)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO comm 0x556db8b8dd00 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO comm 0x559361b695d0 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO comm 0x556f2a64bf80 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO comm 0x5570936d9f40 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO comm 0x5584194aa180 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO comm 0x55cf3325aa10 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO comm 0x55ea64d13050 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO comm 0x557c784ec2c0 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 00/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 01/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO P2P Chunksize set to 524288
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 02/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO P2P Chunksize set to 524288
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 03/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 04/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 05/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO P2P Chunksize set to 524288
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 06/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO P2P Chunksize set to 524288
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 07/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO P2P Chunksize set to 524288
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 08/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 09/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO P2P Chunksize set to 524288
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO P2P Chunksize set to 524288
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 10/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 11/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 12/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 13/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 14/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 15/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 16/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 17/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 18/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 19/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 20/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 21/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 22/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Channel 23/24 : 0 1 2 3 4 5 6 7
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO P2P Chunksize set to 524288
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40130 [6] NCCL INFO [Proxy Service UDS] Device 6 CPU core 26
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40129 [6] NCCL INFO [Proxy Service] Device 6 CPU core 90
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40132 [4] NCCL INFO [Proxy Service UDS] Device 4 CPU core 95
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40131 [4] NCCL INFO [Proxy Service] Device 4 CPU core 37
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40135 [5] NCCL INFO [Proxy Service UDS] Device 5 CPU core 88
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40133 [5] NCCL INFO [Proxy Service] Device 5 CPU core 91
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40134 [2] NCCL INFO [Proxy Service] Device 2 CPU core 6
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40136 [2] NCCL INFO [Proxy Service UDS] Device 2 CPU core 4
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40137 [0] NCCL INFO [Proxy Service] Device 0 CPU core 68
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40138 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 12
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40142 [7] NCCL INFO [Proxy Service UDS] Device 7 CPU core 36
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40141 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 10
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40140 [1] NCCL INFO [Proxy Service] Device 1 CPU core 18
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40139 [7] NCCL INFO [Proxy Service] Device 7 CPU core 25
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40143 [3] NCCL INFO [Proxy Service] Device 3 CPU core 5
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40144 [3] NCCL INFO [Proxy Service UDS] Device 3 CPU core 3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO CC Off, workFifoBytes 1048576
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO TUNER/Plugin: Using tuner plugin nccl_ofi_tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO TUNER/Plugin: Using tuner plugin nccl_ofi_tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO TUNER/Plugin: Using tuner plugin nccl_ofi_tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO NET/OFI NCCL_OFI_TUNER is not available for platform : p4de.24xlarge, Fall back to NCCL's tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO TUNER/Plugin: Using tuner plugin nccl_ofi_tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO NET/OFI NCCL_OFI_TUNER is not available for platform : p4de.24xlarge, Fall back to NCCL's tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO NET/OFI NCCL_OFI_TUNER is not available for platform : p4de.24xlarge, Fall back to NCCL's tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO ncclCommInitRankConfig comm 0x5584194aa180 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId 901d0 commId 0xdd4996e97ba7d4db - Init COMPLETE
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO ncclCommInitRankConfig comm 0x559361b695d0 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId a01c0 commId 0xdd4996e97ba7d4db - Init COMPLETE
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO ncclCommInitRankConfig comm 0x557c784ec2c0 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 901c0 commId 0xdd4996e97ba7d4db - Init COMPLETE
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40121 [6] NCCL INFO Init timings - ncclCommInitRankConfig: rank 6 nranks 8 total 0.35 (kernels 0.17, alloc 0.02, bootstrap 0.00, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.02)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO NET/OFI NCCL_OFI_TUNER is not available for platform : p4de.24xlarge, Fall back to NCCL's tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40105 [4] NCCL INFO Init timings - ncclCommInitRankConfig: rank 4 nranks 8 total 0.81 (kernels 0.17, alloc 0.02, bootstrap 0.46, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.07, rest 0.02)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO TUNER/Plugin: Using tuner plugin nccl_ofi_tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40091 [5] NCCL INFO Init timings - ncclCommInitRankConfig: rank 5 nranks 8 total 1.20 (kernels 0.18, alloc 0.04, bootstrap 0.82, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.03)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO ncclCommInitRankConfig comm 0x556f2a64bf80 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 101d0 commId 0xdd4996e97ba7d4db - Init COMPLETE
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO TUNER/Plugin: Using tuner plugin nccl_ofi_tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40117 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 8 total 0.40 (kernels 0.17, alloc 0.02, bootstrap 0.05, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.02)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO NET/OFI NCCL_OFI_TUNER is not available for platform : p4de.24xlarge, Fall back to NCCL's tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO NET/OFI NCCL_OFI_TUNER is not available for platform : p4de.24xlarge, Fall back to NCCL's tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO ncclCommInitRankConfig comm 0x5570936d9f40 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId a01d0 commId 0xdd4996e97ba7d4db - Init COMPLETE
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO ncclCommInitRankConfig comm 0x55ea64d13050 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 201d0 commId 0xdd4996e97ba7d4db - Init COMPLETE
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40099 [7] NCCL INFO Init timings - ncclCommInitRankConfig: rank 7 nranks 8 total 0.94 (kernels 0.17, alloc 0.02, bootstrap 0.60, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.03)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40101 [3] NCCL INFO Init timings - ncclCommInitRankConfig: rank 3 nranks 8 total 0.92 (kernels 0.17, alloc 0.02, bootstrap 0.57, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.05, rest 0.03)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO TUNER/Plugin: Using tuner plugin nccl_ofi_tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol.
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO NET/OFI NCCL_OFI_TUNER is not available for platform : p4de.24xlarge, Fall back to NCCL's tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO TUNER/Plugin: Using tuner plugin nccl_ofi_tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO ncclCommInitRankConfig comm 0x556db8b8dd00 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 201c0 commId 0xdd4996e97ba7d4db - Init COMPLETE
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40113 [2] NCCL INFO Init timings - ncclCommInitRankConfig: rank 2 nranks 8 total 0.54 (kernels 0.17, alloc 0.03, bootstrap 0.18, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.03)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO NET/OFI NCCL_OFI_TUNER is not available for platform : p4de.24xlarge, Fall back to NCCL's tuner
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO ncclCommInitRankConfig comm 0x55cf3325aa10 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 101c0 commId 0xdd4996e97ba7d4db - Init COMPLETE
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40089 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 8 total 1.20 (kernels 0.18, alloc 0.05, bootstrap 0.82, allgathers 0.00, topo 0.06, graphs 0.00, connections 0.06, rest 0.03)
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 12/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 13/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 12/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 14/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 13/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 15/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 16/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 14/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 17/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 15/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 18/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 16/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 19/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 12/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 20/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 17/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 18/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 21/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 13/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 14/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 22/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 19/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 12/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 15/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 20/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Channel 23/0 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 21/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 16/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 17/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 13/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 22/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 14/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 18/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 15/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Channel 23/0 : 4[4] -> 5[5] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 16/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 19/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 20/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 17/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 21/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 18/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 22/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Channel 23/0 : 6[6] -> 7[7] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 19/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 20/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 21/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 22/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Channel 23/0 : 5[5] -> 6[6] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[4] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40146 [7] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:40151 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40152 [5] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:40148 [4] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40149 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:40150 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40147 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40145 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for humaneval_plus on rank 0...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for humaneval_plus on rank 5...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for humaneval_plus on rank 3...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for humaneval_plus on rank 6...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for humaneval_plus on rank 2...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for humaneval_plus on rank 1...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for humaneval_plus on rank 4...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for humaneval_plus on rank 7...
|
||
|
|
0%| | 0/21 [00:00<?, ?it/s]
0%| | 0/20 [00:00<?, ?it/s]
0%| | 0/20 [00:00<?, ?it/s]
0%| | 0/20 [00:00<?, ?it/s]
0%| | 0/21 [00:00<?, ?it/s]
0%| | 0/21 [00:00<?, ?it/s]
0%| | 0/20 [00:00<?, ?it/s]
0%| | 0/21 [00:00<?, ?it/s]
100%|██████████| 20/20 [00:00<00:00, 1295.60it/s]
|
||
|
|
100%|██████████| 20/20 [00:00<00:00, 1282.50it/s]
|
||
|
|
100%|██████████| 20/20 [00:00<00:00, 1278.17it/s]
|
||
|
|
100%|██████████| 20/20 [00:00<00:00, 1301.55it/s]
|
||
|
|
100%|██████████| 21/21 [00:00<00:00, 1291.46it/s]
|
||
|
|
100%|██████████| 21/21 [00:00<00:00, 1267.62it/s]
|
||
|
|
100%|██████████| 21/21 [00:00<00:00, 1240.32it/s]
100%|██████████| 21/21 [00:00<00:00, 1264.45it/s]
|
||
|
|
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for mbpp on rank 4...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for mbpp on rank 5...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for mbpp on rank 3...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for mbpp on rank 7...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for mbpp on rank 1...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for mbpp on rank 2...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for mbpp on rank 6...
|
||
|
|
2026-04-30:10:58:52 INFO [api.task:312] Building contexts for mbpp on rank 0...
|
||
|
|
0%| | 0/63 [00:00<?, ?it/s]
0%| | 0/63 [00:00<?, ?it/s]
0%| | 0/63 [00:00<?, ?it/s]
0%| | 0/63 [00:00<?, ?it/s]
0%| | 0/62 [00:00<?, ?it/s]
0%| | 0/62 [00:00<?, ?it/s]
0%| | 0/62 [00:00<?, ?it/s]
0%| | 0/62 [00:00<?, ?it/s]
23%|██▎ | 14/62 [00:00<00:00, 139.42it/s]
22%|██▏ | 14/63 [00:00<00:00, 138.47it/s]
22%|██▏ | 14/63 [00:00<00:00, 136.64it/s]
23%|██▎ | 14/62 [00:00<00:00, 136.75it/s]
23%|██▎ | 14/62 [00:00<00:00, 136.39it/s]
22%|██▏ | 14/63 [00:00<00:00, 135.45it/s]
22%|██▏ | 14/63 [00:00<00:00, 135.64it/s]
23%|██▎ | 14/62 [00:00<00:00, 134.46it/s]
44%|████▍ | 28/63 [00:00<00:00, 137.70it/s]
44%|████▍ | 28/63 [00:00<00:00, 137.84it/s]
45%|████▌ | 28/62 [00:00<00:00, 137.75it/s]
45%|████▌ | 28/62 [00:00<00:00, 137.02it/s]
45%|████▌ | 28/62 [00:00<00:00, 136.82it/s]
44%|████▍ | 28/63 [00:00<00:00, 136.19it/s]
47%|████▋ | 29/62 [00:00<00:00, 141.63it/s]
46%|████▌ | 29/63 [00:00<00:00, 140.40it/s]
67%|██████▋ | 42/63 [00:00<00:00, 138.58it/s]
68%|██████▊ | 42/62 [00:00<00:00, 138.44it/s]
67%|██████▋ | 42/63 [00:00<00:00, 137.98it/s]
68%|██████▊ | 42/62 [00:00<00:00, 137.68it/s]
68%|██████▊ | 42/62 [00:00<00:00, 137.67it/s]
67%|██████▋ | 42/63 [00:00<00:00, 136.69it/s]
71%|███████ | 44/62 [00:00<00:00, 141.75it/s]
70%|██████▉ | 44/63 [00:00<00:00, 140.41it/s]
89%|████████▉ | 56/63 [00:00<00:00, 138.85it/s]
90%|█████████ | 56/62 [00:00<00:00, 138.66it/s]
89%|████████▉ | 56/63 [00:00<00:00, 137.96it/s]
90%|█████████ | 56/62 [00:00<00:00, 137.92it/s]
90%|█████████ | 56/62 [00:00<00:00, 137.75it/s]
89%|████████▉ | 56/63 [00:00<00:00, 136.68it/s]
95%|█████████▌| 59/62 [00:00<00:00, 142.23it/s]
94%|█████████▎| 59/63 [00:00<00:00, 140.92it/s]
100%|██████████| 62/62 [00:00<00:00, 141.84it/s]
|
||
|
|
100%|██████████| 63/63 [00:00<00:00, 140.57it/s]
|
||
|
|
100%|██████████| 62/62 [00:00<00:00, 138.37it/s]
|
||
|
|
100%|██████████| 62/62 [00:00<00:00, 137.66it/s]
|
||
|
|
100%|██████████| 62/62 [00:00<00:00, 137.51it/s]
|
||
|
|
100%|██████████| 63/63 [00:00<00:00, 138.48it/s]
|
||
|
|
100%|██████████| 63/63 [00:00<00:00, 137.79it/s]
|
||
|
|
100%|██████████| 63/63 [00:00<00:00, 136.47it/s]
|
||
|
|
2026-04-30:10:58:53 INFO [api.task:312] Building contexts for mbpp_plus on rank 4...
|
||
|
|
2026-04-30:10:58:53 INFO [api.task:312] Building contexts for mbpp_plus on rank 7...
|
||
|
|
2026-04-30:10:58:53 INFO [api.task:312] Building contexts for mbpp_plus on rank 5...
|
||
|
|
2026-04-30:10:58:53 INFO [api.task:312] Building contexts for mbpp_plus on rank 3...
|
||
|
|
2026-04-30:10:58:53 INFO [api.task:312] Building contexts for mbpp_plus on rank 1...
|
||
|
|
2026-04-30:10:58:53 INFO [api.task:312] Building contexts for mbpp_plus on rank 2...
|
||
|
|
2026-04-30:10:58:53 INFO [api.task:312] Building contexts for mbpp_plus on rank 6...
|
||
|
|
2026-04-30:10:58:53 INFO [api.task:312] Building contexts for mbpp_plus on rank 0...
|
||
|
|
0%| | 0/48 [00:00<?, ?it/s]
0%| | 0/47 [00:00<?, ?it/s]
0%| | 0/48 [00:00<?, ?it/s]
0%| | 0/47 [00:00<?, ?it/s]
0%| | 0/47 [00:00<?, ?it/s]
0%| | 0/47 [00:00<?, ?it/s]
0%| | 0/47 [00:00<?, ?it/s]
0%| | 0/47 [00:00<?, ?it/s]
28%|██▊ | 13/47 [00:00<00:00, 125.27it/s]
27%|██▋ | 13/48 [00:00<00:00, 124.07it/s]
28%|██▊ | 13/47 [00:00<00:00, 121.29it/s]
28%|██▊ | 13/47 [00:00<00:00, 121.41it/s]
28%|██▊ | 13/47 [00:00<00:00, 120.55it/s]
28%|██▊ | 13/47 [00:00<00:00, 120.35it/s]
28%|██▊ | 13/47 [00:00<00:00, 120.84it/s]
27%|██▋ | 13/48 [00:00<00:00, 120.22it/s]
55%|█████▌ | 26/47 [00:00<00:00, 124.44it/s]
54%|█████▍ | 26/48 [00:00<00:00, 123.36it/s]
55%|█████▌ | 26/47 [00:00<00:00, 121.89it/s]
55%|█████▌ | 26/47 [00:00<00:00, 121.67it/s]
55%|█████▌ | 26/47 [00:00<00:00, 121.25it/s]
55%|█████▌ | 26/47 [00:00<00:00, 120.65it/s]
54%|█████▍ | 26/48 [00:00<00:00, 120.51it/s]
55%|█████▌ | 26/47 [00:00<00:00, 119.51it/s]
83%|████████▎ | 39/47 [00:00<00:00, 123.77it/s]
81%|████████▏ | 39/48 [00:00<00:00, 122.78it/s]
83%|████████▎ | 39/47 [00:00<00:00, 121.97it/s]
83%|████████▎ | 39/47 [00:00<00:00, 121.77it/s]
83%|████████▎ | 39/47 [00:00<00:00, 121.41it/s]
83%|████████▎ | 39/47 [00:00<00:00, 120.67it/s]
81%|████████▏ | 39/48 [00:00<00:00, 120.44it/s]
83%|████████▎ | 39/47 [00:00<00:00, 119.90it/s]
100%|██████████| 47/47 [00:00<00:00, 123.88it/s]
|
||
|
|
100%|██████████| 47/47 [00:00<00:00, 121.89it/s]
|
||
|
|
100%|██████████| 47/47 [00:00<00:00, 121.70it/s]
|
||
|
|
100%|██████████| 47/47 [00:00<00:00, 121.34it/s]
|
||
|
|
100%|██████████| 47/47 [00:00<00:00, 120.68it/s]
|
||
|
|
100%|██████████| 48/48 [00:00<00:00, 122.76it/s]
|
||
|
|
100%|██████████| 47/47 [00:00<00:00, 119.92it/s]
|
||
|
|
100%|██████████| 48/48 [00:00<00:00, 120.33it/s]
|
||
|
|
2026-04-30:10:58:53 INFO [evaluator:575] Running generate_until requests
|
||
|
|
2026-04-30:10:58:53 INFO [evaluator:575] Running generate_until requests
|
||
|
|
2026-04-30:10:58:53 INFO [evaluator:575] Running generate_until requests
|
||
|
|
2026-04-30:10:58:53 INFO [evaluator:575] Running generate_until requests
|
||
|
|
2026-04-30:10:58:53 INFO [evaluator:575] Running generate_until requests
|
||
|
|
2026-04-30:10:58:53 INFO [evaluator:575] Running generate_until requests
|
||
|
|
2026-04-30:10:58:53 INFO [evaluator:575] Running generate_until requests
|
||
|
|
2026-04-30:10:58:53 INFO [evaluator:575] Running generate_until requests
|
||
|
|
Passed argument batch_size = auto. Detecting largest batch size
|
||
|
|
Passed argument batch_size = auto. Detecting largest batch sizePassed argument batch_size = auto. Detecting largest batch size
|
||
|
|
Passed argument batch_size = auto. Detecting largest batch size
|
||
|
|
|
||
|
|
Passed argument batch_size = auto. Detecting largest batch sizePassed argument batch_size = auto. Detecting largest batch size
|
||
|
|
|
||
|
|
Passed argument batch_size = auto. Detecting largest batch size
|
||
|
|
Running generate_until requests: 0%| | 0/153 [00:00<?, ?it/s]Passed argument batch_size = auto. Detecting largest batch size
|
||
|
|
Determined Largest batch size: 11
|
||
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
||
|
|
Determined Largest batch size: 11
|
||
|
|
Determined Largest batch size: 11
|
||
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
||
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
||
|
|
Determined Largest batch size: 11
|
||
|
|
Determined Largest batch size: 11
|
||
|
|
Determined Largest batch size: 11
|
||
|
|
Determined Largest batch size: 11
|
||
|
|
Determined Largest batch size: 11
|
||
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
||
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
||
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
||
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
||
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
||
|
|
Running generate_until requests: 1%| | 1/153 [06:43<17:02:16, 403.53s/it]
Running generate_until requests: 8%|▊ | 12/153 [07:43<1:08:50, 29.29s/it]
Running generate_until requests: 15%|█▌ | 23/153 [08:43<33:49, 15.61s/it]
Running generate_until requests: 22%|██▏ | 34/153 [08:54<17:52, 9.01s/it]
Running generate_until requests: 28%|██▊ | 43/153 [09:05<11:36, 6.33s/it]
Running generate_until requests: 35%|███▌ | 54/153 [09:11<06:51, 4.16s/it]
Running generate_until requests: 42%|████▏ | 65/153 [09:16<04:13, 2.88s/it]
Running generate_until requests: 50%|████▉ | 76/153 [09:23<02:45, 2.14s/it]
Running generate_until requests: 57%|█████▋ | 87/153 [09:31<01:49, 1.67s/it]
Running generate_until requests: 64%|██████▍ | 98/153 [09:35<01:09, 1.27s/it]
Running generate_until requests: 71%|███████ | 109/153 [09:42<00:46, 1.06s/it]2026-04-30:11:08:37 WARNING [models.huggingface:1038] Left truncation applied. Original sequence length was 4271, truncating to last 3840 tokens. Some content will be lost.
|
||
|
|
Running generate_until requests: 78%|███████▊ | 120/153 [09:48<00:29, 1.11it/s]
Running generate_until requests: 86%|████████▌ | 131/153 [09:55<00:18, 1.20it/s]
Running generate_until requests: 93%|█████████▎| 142/153 [10:00<00:07, 1.41it/s]
Running generate_until requests: 100%|██████████| 153/153 [10:01<00:00, 1.94it/s]
Running generate_until requests: 100%|██████████| 153/153 [10:01<00:00, 3.93s/it]
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 00/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 00/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 00/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 00/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 00/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 00/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 00/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 01/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 01/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 01/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 01/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 01/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 01/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 01/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 02/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 02/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 02/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 02/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 02/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 02/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 03/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 03/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 03/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 02/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 03/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 03/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 04/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 03/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 04/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 04/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 04/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 03/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 04/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 05/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 04/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 05/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 05/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 04/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 05/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 05/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 05/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 06/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 06/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 05/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 06/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 07/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 07/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 06/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 06/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 07/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 06/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 08/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 07/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 06/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 07/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 08/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 07/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 08/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 09/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 10/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 08/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 08/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 07/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 08/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 09/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 08/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 10/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 11/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 09/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 09/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 09/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 10/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 10/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 09/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 11/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 12/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 10/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 12/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 13/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 09/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 11/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 12/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 10/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 14/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 10/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 11/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 12/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 11/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 11/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 11/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 15/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 16/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 13/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 13/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 12/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 12/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 14/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 13/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 12/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 17/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 13/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 13/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 14/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 14/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 15/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 14/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 18/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 15/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 14/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 15/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 19/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 13/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 15/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 16/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 16/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 14/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 16/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 17/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 15/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 16/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 17/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 15/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 18/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 16/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 17/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 17/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 17/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 19/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 20/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 16/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 18/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 18/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 17/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 21/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 20/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 18/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 19/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 18/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 22/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 18/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 21/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 22/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 20/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 23/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 19/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 19/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 19/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 20/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 23/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 20/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 19/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 20/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 24/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 21/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 20/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 25/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 21/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 21/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 24/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 25/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 21/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 26/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 22/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 22/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 21/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 22/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 26/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 27/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 23/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 23/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 24/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 22/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 22/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 23/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 24/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 28/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 25/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 24/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 25/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 29/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 23/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 27/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 25/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 26/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 24/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 23/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 30/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:57210 [1] NCCL INFO Channel 31/1 : 1[1] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 26/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 28/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 25/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 24/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 27/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 26/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 25/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 28/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 26/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 27/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 27/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 29/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 26/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 29/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 28/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 27/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 30/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:57211 [5] NCCL INFO Channel 31/1 : 5[5] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 27/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 30/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 29/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 28/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 28/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 30/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39124:57207 [4] NCCL INFO Channel 31/1 : 4[4] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 29/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 28/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 29/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 29/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 30/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39126:57208 [6] NCCL INFO Channel 31/1 : 6[6] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 30/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 30/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:57206 [3] NCCL INFO Channel 31/1 : 3[3] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:57209 [7] NCCL INFO Channel 31/1 : 7[7] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39122:57212 [2] NCCL INFO Channel 31/1 : 2[2] -> 0[0] via P2P/CUMEM/read
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40137 [0] NCCL INFO [Service thread] Connection closed by localRank 4
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40143 [3] NCCL INFO [Service thread] Connection closed by localRank 4
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40133 [0] NCCL INFO [Service thread] Connection closed by localRank 4
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40137 [0] NCCL INFO [Service thread] Connection closed by localRank 2
|
||
|
|
ywang29-p4d-debug-2-worker-0:39123:40143 [3] NCCL INFO [Service thread] Connection closed by localRank 2
|
||
|
|
ywang29-p4d-debug-2-worker-0:39121:40140 [1] NCCL INFO [Service thread] Connection closed by localRank 2
|
||
|
|
ywang29-p4d-debug-2-worker-0:39125:40133 [0] NCCL INFO [Service thread] Connection closed by localRank 6
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40137 [0] NCCL INFO [Service thread] Connection closed by localRank 6
|
||
|
|
ywang29-p4d-debug-2-worker-0:39127:40139 [7] NCCL INFO [Service thread] Connection closed by localRank 6
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40137 [0] NCCL INFO [Service thread] Connection closed by localRank 5
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40137 [0] NCCL INFO [Service thread] Connection closed by localRank 1
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40137 [0] NCCL INFO [Service thread] Connection closed by localRank 3
|
||
|
|
ywang29-p4d-debug-2-worker-0:39120:40137 [0] NCCL INFO [Service thread] Connection closed by localRank 7
|
||
|
|
2026-04-30:11:12:27 INFO [_cli.run:449] Logging to W&B failed: 'def pass_at_k(references: list[str], predictions: list[list[str]], k: list[int] = None):\n global compute_\n assert k is not None\n if isinstance(k, int):\n k = [k]\n res = compute_.compute(\n references=references,\n predictions=predictions,\n k=k,\n )\n return res[0]\n'
|
||
|
|
2026-04-30:11:12:27 INFO [loggers.evaluation_tracker:247] Saving results aggregated
|
||
|
|
2026-04-30:11:12:27 INFO [loggers.evaluation_tracker:119] Saving per-task samples to saves/qwen3_14b/coding/fft/eval_coding/fft/saves__qwen3_14b__coding__fft/*.jsonl
|
||
|
|
hf ({'pretrained': 'saves/qwen3_14b/coding/fft', 'dtype': 'bfloat16', 'parallelize': False, 'max_length': 4096}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: auto
|
||
|
|
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|
||
|
|
|--------------|------:|-----------|-----:|---------|---|-----:|---|-----:|
|
||
|
|
|humaneval | 1|create_test| 0|pass@1 |↑ |0.7439|± |0.0342|
|
||
|
|
|humaneval_plus| 1|create_test| 0|pass@1 |↑ |0.6890|± |0.0363|
|
||
|
|
|mbpp | 1|none | 3|pass_at_1|↑ |0.7060|± |0.0204|
|
||
|
|
|mbpp_plus | 1|none | 3|pass_at_1|↑ |0.8333|± |0.0192|
|
||
|
|
|
||
|
|
wandb: updating run metadata; uploading artifact run-jm4o6xvg-evaluationeval_results-1S5ClQ; uploading artifact results
|
||
|
|
wandb: uploading artifact run-jm4o6xvg-evaluationeval_results-1S5ClQ; uploading artifact results
|
||
|
|
wandb: uploading artifact results
|
||
|
|
wandb:
|
||
|
|
wandb: Run history:
|
||
|
|
wandb: humaneval/pass@1,create_test ▁
|
||
|
|
wandb: humaneval/pass@1_stderr,create_test ▁
|
||
|
|
wandb: humaneval/sample_len ▁
|
||
|
|
wandb: humaneval_plus/pass@1,create_test ▁
|
||
|
|
wandb: humaneval_plus/pass@1_stderr,create_test ▁
|
||
|
|
wandb: humaneval_plus/sample_len ▁
|
||
|
|
wandb: mbpp/pass_at_1 ▁
|
||
|
|
wandb: mbpp/pass_at_1_stderr ▁
|
||
|
|
wandb: mbpp/sample_len ▁
|
||
|
|
wandb: mbpp_plus/pass_at_1 ▁
|
||
|
|
wandb: +2 ...
|
||
|
|
wandb:
|
||
|
|
wandb: Run summary:
|
||
|
|
wandb: humaneval/alias humaneval
|
||
|
|
wandb: humaneval/name humaneval
|
||
|
|
wandb: humaneval/pass@1,create_test 0.7439
|
||
|
|
wandb: humaneval/pass@1_stderr,create_test 0.03419
|
||
|
|
wandb: humaneval/sample_len 164
|
||
|
|
wandb: humaneval_plus/alias humaneval_plus
|
||
|
|
wandb: humaneval_plus/name humaneval_plus
|
||
|
|
wandb: humaneval_plus/pass@1,create_test 0.68902
|
||
|
|
wandb: humaneval_plus/pass@1_stderr,create_test 0.03626
|
||
|
|
wandb: humaneval_plus/sample_len 164
|
||
|
|
wandb: +10 ...
|
||
|
|
wandb:
|
||
|
|
wandb: 🚀 View run fft_coding at: https://wandb.ai/maskmoe/MFT-LM-eval/runs/jm4o6xvg
|
||
|
|
wandb: ⭐️ View project at: https://wandb.ai/maskmoe/MFT-LM-eval
|
||
|
|
wandb: Synced 5 W&B file(s), 1 media file(s), 4 artifact file(s) and 0 other file(s)
|
||
|
|
wandb: Find logs at: ./wandb/run-20260430_105344-jm4o6xvg/logs
|