CODEWITHSUNDEEP
pythongoogle-colaboratory
nilsinelabore
nilsinelabore

Reputation: 5105

File too large to read in Colab

Usually I would upload a csv file to google drive, then in Google Colab, mount drive at /content/drive and use pd.read_csv to read the file.

The uploaded files usually appear under /content/drive/My Drive/ like this after mounting:

enter image description here

This time I uploaded a csv file of 2GB in size but could not find it in the /content/drive/My Drive/ section in order to open it. I checked a few posts but didn't seem to solve my problem. Is there a way I can read the large csv file?

Update:

Upon trying to get the sharable link with edit right, it appeared there was no option for edit, as shown below:enter image description here

Upvotes: 2

Views: 3915

Answers (2)

Chris Tanner
Chris Tanner

Reputation: 90

As @korakot mentioned, you can use the following command to download the file from your drive to the current directory in Colab Pro using the following command:

!gdown --id 1GNjhogITyxMiaLno4trJoGM6I-MofFG8

This assumes that the file is made shareable in Drive.

If you receive the Permission Denied error and have verified that the file is indeed shareable, you have unfortunately reached a quota. This happens when large files in Drive are accessed too frequently in a 24 hour period. I experienced this recently with a very large file and was able to access it after waiting a day or so.

To get around this, there is a thread around this topic in the Colab GitHub, and a potential solution to download the file in chunks in this comment specifically.

Upvotes: 1

korakot
korakot

Reputation: 40828

You can try an alternative method to mounting drive.

!gdown --id xxxxxxxxxxx

This will download the file from GDrive to the current directory.

You need to share the file publicly, and get the ID (xxxxxx) from the shared URL.

Upvotes: 0

Related Questions