pytorch - net.cuda() seems don't work

Question

I wrote a cnn module to do digit recognition using pytorch, then try to train the network with gpu but got following error.

Traceback (most recent call last):
  File "main.py", line 51, in <module>
    outputs = cnn(inputs)
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/daniel/Code/kaggle-competitions/digit-recognizer/Net.py", line 40, in forward
    x = self.pool(F.relu(self.conv[i](x)))
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/conv.py", line 282, in forward
    self.padding, self.dilation, self.groups)
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/functional.py", line 90, in conv2d
    return f(input, weight, bias)
RuntimeError: Input type (CUDAFloatTensor) and weight type (CPUFloatTensor) should be the same

here is my source code

It seems that cnn.cuda() didn't work properly because I got the same error after removing it. But I have no idea how to fix it.

Answer 1

Daniel's answer to his own question seems to be correct. The problem is indeed that modules are not recognized if they are appended to a list. However, Pytorch also provides built-in solutions to this problem: nn.ModuleList and nn.ModuleDict are two container types that keep track of the added content and their parameters. Both have the same functionality as their Python equivalents, but the dictionary uses named arguments and can be used to keep track of for example task-specific layers.

A working example would be:

    self.conv = torch.nn.ModuleList()
    self.conv.append(nn.Conv2d(1, channels[0], kernel_sizes[0]))
    self.conv_img_size = math.floor((self.conv_img_size - (kernel_sizes[0]-1))/2)

    for i in range(1, self.conv_layer_size):
        self.conv.append(nn.Conv2d(channels[i-1], channels[i], kernel_sizes[i]))
        self.conv_img_size = math.floor((self.conv_img_size - (kernel_sizes[i]-1))/2)
        # Modules are automatically added to the model parameters

Answer 2

I solved it by myself. It's because I assigned the child modules in non-standard way, the submodules wasn't registered to the child module list of my module. The module.parameter() won't return parameters of these unregistered submodules. And module.cuda() only move the registered parameters to GPU.

By default, if you assign the child module is this way, the submodule will be registered automatically:

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

However, I assigned the submodules by append them to a list:

class Cnn(nn.Module):
    def __init__(self, channels, kernel_sizes, dense_layers, n_classes, img_size):
        super(Cnn, self).__init__()

        ...

        self.conv = []
        self.conv.append(nn.Conv2d(1, channels[0], kernel_sizes[0]))
        self.conv_img_size = math.floor((self.conv_img_size - (kernel_sizes[0]-1))/2)
        for i in range(1, self.conv_layer_size):
            self.conv.append(nn.Conv2d(channels[i-1], channels[i], kernel_sizes[i]))
            self.conv_img_size = math.floor((self.conv_img_size - (kernel_sizes[i]-1))/2)

I need to call module.add_module manually to register these submodules.

class Cnn(nn.Module):
    def __init__(self, channels, kernel_sizes, dense_layers, n_classes, img_size):
        super(Cnn, self).__init__()

        ...

        self.conv = []
        self.conv.append(nn.Conv2d(1, channels[0], kernel_sizes[0]))
        self.conv_img_size = math.floor((self.conv_img_size - (kernel_sizes[0]-1))/2)
        self.add_module('Conv0', self.conv[0]) # Add modules manually
        for i in range(1, self.conv_layer_size):
            self.conv.append(nn.Conv2d(channels[i-1], channels[i], kernel_sizes[i]))
            self.conv_img_size = math.floor((self.conv_img_size - (kernel_sizes[i]-1))/2)
            self.add_module('Conv'+str(i), self.conv[i]) # Add modules manually

You can check the registered modules by print the module instance.

Before adding module.add_module:

>>> print(cnn)
Cnn(
  (pool): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), dilation=(1, 1), ceil_mode=False)
  (output_layer): Linear(in_features=1024, out_features=10, bias=True)
)

After:

>>> print(cnn)
Cnn(
  (Conv0): Conv2d(1, 32, kernel_size=(5, 5), stride=(1, 1))
  (Conv1): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1))
  (pool): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), dilation=(1, 1), ceil_mode=False)
  (Dense0): Linear(in_features=1024, out_features=1024, bias=True)
  (output_layer): Linear(in_features=1024, out_features=10, bias=True)
)