CODEWITHSUNDEEP
pythonpytorch
krpytix
krpytix

Reputation: 33

PyTorch not working when using Pytorch with cuda 11.1: Dataloader

So, I'm running this deep neural network. And normally, when i train it with the cpu using the pytorch library WITHOUT cuda, it runs fine.

However, I noticed when I installed pytorch+cuda 11.1, whenever I try to enumerate over the Dataloader, the following error gets thrown out:

OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading "C:\Users\Name\Desktop\Folder\lib\site-packages\torch\lib\cudnn_adv_train64_8.dll" or one of its dependencies.

But, when i use pip to uninstall pytorch+cuda and install pytorch without cuda, my script runs fine.

both versions of pytorch are 1.8.0

Does anyone know how to go about fixing this? Would love to be able to use this gpu.

Note: This is how i set up the dataloader:

    train_dataloader = DataLoader(
        train_dataset,
        batch_size=args.batch_size,
        num_workers=args.num_workers,
        shuffle=True,
        pin_memory=(device == "cuda"),
    )

Upvotes: 0

Views: 2300

Answers (2)

Thomas Erdösi
Thomas Erdösi

Reputation: 546

This error also occurs when the page file is currently being extended by the OS. Setting the page file to a FIXED SIZE solved the issue for me.

Proceed as follows:

  • Go to the advanced system settings
  • Select tab "Advanced"
  • Click the first "Settings" button
  • Select tab "Advanced"
  • Click on "Change" button
  • Uncheck the "Automatically manage paging file size for all drives"
  • Select option "Custom size"
  • Enter the desired size as "Initial size" AND as "Maximum size" (this ensures that the page file is created with the maximum size and does not need to be extended)
  • Confirm all dialog windows and reboot

Upvotes: 0

James
James

Reputation: 36598

See this issue on the PyTorch github page: https://github.com/Spandan-Madan/Pytorch_fine_tuning_Tutorial/issues/10

Basically you need to turn off the automatic paging file management in Windows.

  • Windows Key
  • Search for: advanced system settings
  • Advanced tab
  • Performance - "Settings" button
  • Advanced tab - "Change" button
  • Uncheck the "Automatically manage paging file size for all drives" checkbox
  • Select the "System managed size" radio button.
  • Reboot

Upvotes: 1

Related Questions