ncclInternalError: Internal check failed. Proxy Call to rank 0 failed (Connect)

Question

ncclInternalError: Internal check failed. Proxy Call to rank 0 failed (Connect)

After setting up ray cluster with 2 nodes of single gpu & also direct pytroch distributed run … with the same nodes i got my distributed process registered. starting with 2 process with backed nccl

NCCL INFO :

Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2 (RayExecutor pid=423719, ip=172.16.0.2) Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2 (RayExecutor pid=508760) ---------------------------------------------------------------------------------------------------- (RayExecutor pid=508760) distributed_backend=nccl (RayExecutor pid=508760) All distributed processes registered. Starting with 2 processes (RayExecutor pid=508760) ---------------------------------------------------------------------------------------------------- (RayExecutor pid=508760) (RayExecutor pid=508760) GPU available: True (cuda), used: True (Please ignore the previous info [GPU used: False]). (RayExecutor pid=508760) hostssh:508760:508760 [0] NCCL INFO Bootstrap : Using enp3s0:172.16.96.59<0> (RayExecutor pid=508760) hostssh:508760:508760 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation (RayExecutor pid=508760) hostssh:508760:508760 [0] NCCL INFO cudaDriverVersion 11070 (RayExecutor pid=508760) NCCL version 2.14.3+cuda11.7

But as soon as this message i am getting an nccInternalError : Internal check failed

RayTaskError(RuntimeError): [36mray::RayExecutor.execute()[39m (pid=508760, ip=172.16.96.59, repr=) File "/home/windows/miniconda3/envs/ray/lib/python3.10/site-packages/ray_lightning/launchers/utils.py", line 52, in execute return fn(*args, **kwargs) File "/home/windows/miniconda3/envs/ray/lib/python3.10/site-packages/ray_lightning/launchers/ray_launcher.py", line 301, in _wrapping_function results = function(*args, **kwargs) File "/home/windows/miniconda3/envs/ray/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 811, in _fit_impl results = self._run(model, ckpt_path=self.ckpt_path) File "/home/windows/miniconda3/envs/ray/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1172, in _run self.__setup_profiler() File "/home/windows/miniconda3/envs/ray/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1797, in __setup_profiler self.profiler.setup(stage=self.state.fn._setup_fn, local_rank=local_rank, log_dir=self.log_dir) File "/home/windows/miniconda3/envs/ray/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 2249, in log_dir dirpath = self.strategy.broadcast(dirpath) File "/home/windows/miniconda3/envs/ray/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp_spawn.py", line 215, in broadcast torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD) File "/home/windows/miniconda3/envs/ray/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2084, in broadcast_object_list broadcast(object_sizes_tensor, src=src, group=group) File "/home/windows/miniconda3/envs/ray/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1400, in broadcast work = default_pg.broadcast([tensor], opts) RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1670525541990/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1269, internal error, NCCL version 2.14.3 ncclInternalError: Internal check failed. Last error: Proxy Call to rank 0 failed (Connect)

I am running in on premise cluster without any containerization . And single gpu code works successfully (with 16 batch size). 

### Versions / Dependencies


torch, ray version
('1.13.1', '2.3.0')

ray_lightning => 0.3.0

CPU: 8-Core 11th Gen Intel Core i7-11700 (-MT MCP-) speed/min/max: 901/800/4800 MHz 
Kernel: 5.4.0-128-generic x86_64 Up: 4h 46m Mem: 9283.6/64016.7 MiB (14.5%) 
Storage: 2.05 TiB (18.7% used) Procs: 332 Shell: bash 5.0.17 inxi: 3.0.38 

No LSB modules are available.
Distributor ID: Linuxmint
Description:    Linux Mint 20.3
Release:        20.3
Codename:       una

### Reproduction script

from pytorch_lightning import Trainer
from torch.utils.data import DataLoader


import ray

ray.init(runtime_env={"working_dir": utils.ROOT_PATH})

dataset_params = utils.config_parse('AUTOENCODER_DATASET')
dataset = AutoEncoderDataModule(**dataset_params)
dataset.setup()


model = AutoEncoder()
autoencoder_params = utils.config_parse('AUTOENCODER_TRAIN')
print(autoencoder_params)
print(torch.cuda.device_count())
dist_env_params = utils.config_parse('DISTRIBUTED_ENV')
strategy = None
if int(dist_env_params['horovod']) == 1:
    strategy = rl.HorovodRayStrategy(use_gpu=True, num_workers=2)
elif int(dist_env_params['model_parallel']) == 1:
    strategy = rl.RayShardedStrategy(use_gpu=True, num_workers=2)
elif int(dist_env_params['data_parallel']) == 1:
    strategy = rl.RayStrategy(use_gpu=True, num_workers=2)
trainer = Trainer(**autoencoder_params,
                strategy=strategy
                )

trainer.fit(model, dataset)

pytorch-lightning parameters:
[AUTOENCODER_TRAIN]
max_epochs = 100
weights_summary = full
precision = 16
gradient_clip_val = 0.0
auto_lr_find = True
auto_scale_batch_size = True
auto_select_gpus = True
check_val_every_n_epoch = 1
fast_dev_run = False
enable_progress_bar = True
detect_anomaly=True

python run.py



### Issue Severity

High: It blocks me from completing my task.

ncclInternalError: Internal check failed. Proxy Call to rank 0 failed (Connect)

ncclInternalError: Internal check failed. Proxy Call to rank 0 failed (Connect)

Answers (1)

Related Questions