PHP - Google App Engine Uploading large files to Google Drive as chunks (Resumable Upload) - 500 Error

Question

I'm uploading a file from the Google Cloud storage bucket to Google Drive folder using a php code that runs in the background. My problem is when the size of the file increase, the Google Drive(or GAE) gives a 500 error.

Error: Server Error
The server encountered an error and could not complete your request.
Please try again in 30 seconds.

I'm using the following dependency.

"nao-pon/flysystem-google-drive": "^1.1",

Following is my code part that gives this error. This works perfectly for small files:

$fileN = $this->clean($att->ATT_TITLE);

$fileData = file_get_contents("https://storage.googleapis.com/xxxxxx/".$att->ATT_FILE);

$dir = sys_get_temp_dir();
$tmp = tempnam($dir, $fileN);
file_put_contents($tmp, $fileData);
Storage::cloud()->putFileAs( $folderPath, new File($tmp),$fileN);

I have set the configurations in .env file

FILESYSTEM_CLOUD=google
GOOGLE_DRIVE_CLIENT_ID=xxxxapps.googleusercontent.com
GOOGLE_DRIVE_CLIENT_SECRET=xxxxx
GOOGLE_DRIVE_REFRESH_TOKEN=xxxxxx
GOOGLE_DRIVE_FOLDER_ID=xxxx
#GOOGLE_DRIVE_TEAM_DRIVE_ID=xxx

I have setup the disk as below in the filesystem.php

'google' => [
        'driver' => 'google',
        'clientId' => env('GOOGLE_DRIVE_CLIENT_ID'),
        'clientSecret' => env('GOOGLE_DRIVE_CLIENT_SECRET'),
        'refreshToken' => env('GOOGLE_DRIVE_REFRESH_TOKEN'),
        'folderId' => env('GOOGLE_DRIVE_FOLDER_ID'),
    // 'teamDriveId' => env('GOOGLE_DRIVE_TEAM_DRIVE_ID'),
    ],

I have set the php.ini file as below

upload_max_filesize = 300M
post_max_size = 300M
memory_limit = 3000M

App.yaml

runtime: php72
handlers: 
- url: /assets
  static_dir: public/assets
env_variables:
   APP_KEY: xxxxx
   APP_STORAGE: /tmp
   CACHE_DRIVER: file
   VIEW_COMPILED_PATH: /tmp
   SESSION_DRIVER: database
   DB_DATABASE: xxx
   DB_USERNAME: xx
   DB_PASSWORD: xxxx
   DB_SOCKET: "/cloudsql/xxxxx"
runtime_config:
    document_root: public

Can anyone provide me a hint or give me another alternative way to do this? My requirement is basically to upload a large file (like 250MB) to Google Drive by a background job in Google App Engine.

Answer 1

For this situation, Uploading in chunks solve the issue.

Refer to the following example from Google API PHP Client https://github.com/googleapis/google-api-php-client/blob/master/examples/large-file-upload.php

Sample Working Code:

$fileData = file_get_contents("https://storage.googleapis.com/xxxx/".$att->ATT_FILE);

$dir = sys_get_temp_dir();
$tmp = tempnam($dir, $fileN);
file_put_contents($tmp, $fileData);       
     
$fileToUpload = new File($tmp);
$filename = $fileN;
$mimeType = mime_content_type($tmp);
                   
$driveService = Storage::cloud();
$client = new \Google_Client();
$client->setClientId('xxxx');
$client->setClientSecret('xxx');
$client->refreshToken('xxx');
$client->setApplicationName('xxx');

$service = new \Google_Service_Drive($client);


// Test 1
$file = new \Google_Service_Drive_DriveFile();
$file->title = $fileN;
$file->name = $fileN;
$file->parents = array($basePath);
$chunkSizeBytes = 1 * 1024 * 1024;

// Call the API with the media upload, defer so it doesn't immediately return.
$client->setDefer(true);
$request = $service->files->create($file);

// Create a media file upload to represent our upload process.
$media = new \Google_Http_MediaFileUpload(
  $client,
  $request,
  $mimeType,
  null,
  true,
  $chunkSizeBytes
);
$media->setFileSize(filesize($tmp));

// Upload the various chunks. $status will be false until the process is
// complete.
$status = false;
$handle = fopen($tmp, "rb");
while (!$status && !feof($handle)) {
  $chunk = fread($handle, $chunkSizeBytes);
  $status = $media->nextChunk($chunk);
 }

// The final value of $status will be the data from the API for the object
// that has been uploaded.
$result = false;
if($status != false) {
  $result = $status;
}

fclose($handle);

Answer 2

Since you are calling Cloud Storage API to upload the files which are big that could be accomplished the upload within the 60 seconds which is the default timeout for Google Cloud Storage API. You can self modify the API timeout. Here it is the stackoverflow case for you to refer to set the API timeout by your own.