CODEWITHSUNDEEP
pythonpytorch
Sean
Sean

Reputation: 3385

Is there really no padding=same option for PyTorch's Conv2d?

I'm currently working on building a convolutional neural network (CNN) that will work on financial time series data. The input shape is (100, 40) - 100 time stamps by 40 features.

The CNN that I'm using uses asymmetric kernel sizes (i.e. 1 x 2 and 4 x 1) and also asymmetric strides (i.e. 1 x 2 for the 1 x 2 layers and 1 x 1 for the 4 x 1 layers).

In order to maintain the height dimension to stay 100, I needed to pad the data. In my research, I noticed that people who use TensorFlow or Keras simply use padding='same'; but this option is apparently unavailable in PyTorch.

According to some answers in What is the difference between 'SAME' and 'VALID' padding in tf.nn.max_pool of tensorflow?, and also this answer on the PyTorch discussion forum, I can manually calculate how I need to pad my data, and use torch.nn.ZeroPad2d to solve the problem - since apparently normal torch.nn.Conv2d layers don't support asymmetric padding (I believe that the total padding I need is 3 in height and 0 in width).

I tried this code:

import torch
import torch.nn as nn

conv = nn.Conv2d(1, 1, kernel_size=(4, 1))
pad = nn.ZeroPad2d((0, 0, 2, 1)) # Add 2 to top and 1 to bottom.

x = torch.randint(low=0, high=9, size=(100, 40))
x = x.unsqueeze(0).unsqueeze(0)

y = pad(x)

x.shape # (1, 1, 100, 40)
y.shape # (1, 1, 103, 40)

print(conv(x.float()).shape)
print(conv(y.float()).shape)

# Output
# x -> (1, 1, 97, 40)
# y -> (1, 1, 100, 40)

It does work, in the sense that the data shape remains the same. However, is there really no padding='same' option available? Also, how can we decide which side to pad?

Upvotes: 25

Views: 30106

Answers (3)

rocksyne
rocksyne

Reputation: 1372

padding='same' and padding='valid' is possible in Pytorch 1.10.0+. However, 'same' and 'valid' for padding is not possible for when stride > 1.

Upvotes: 1

lucidbrot
lucidbrot

Reputation: 6156

It looks like there is now, in pytorch 1.9.1, according to the docs.

padding='valid' is the same as no padding. padding='same' pads the input so the output has the shape as the input. However, this mode doesn't support any stride values other than 1.

Upvotes: 9

duburcqa
duburcqa

Reputation: 1131

I had the same issue some time ago, so I implemented it myself using a ZeroPad2d layer as you are trying to do. Here is the right formula:

from functools import reduce
from operator import __add__

kernel_sizes = (4, 1)

# Internal parameters used to reproduce Tensorflow "Same" padding.
# For some reasons, padding dimensions are reversed wrt kernel sizes,
# first comes width then height in the 2D case.
conv_padding = reduce(__add__, 
    [(k // 2 + (k - 2 * (k // 2)) - 1, k // 2) for k in kernel_sizes[::-1]])

pad = nn.ZeroPad2d(conv_padding)
conv = nn.Conv2d(1, 1, kernel_size=kernel_sizes)

print(x.shape) # (1, 1, 103, 40)
print(conv(y.float()).shape) # (1, 1, 103, 40)

Also, as mentioned by @akshayk07 and @Separius, I can confirm that it is the dynamic nature of pytorch that makes it hard. Here is a post about this point from a Pytorch developper.

Upvotes: 9

Related Questions