Alexander Stoige
Alexander Stoige

Reputation: 13

Read large files - 2GB+ for Google Drive API Upload

I'm currently working on a small backup tool written in C# that is supposed to upload files contained within a specified folder to Google Drive via its API. The program largely functions as it's supposed to, the only problem that it is unable to handle files larger than 2GB.

The problem is caused by the upload function itself which is attached down below, it uses a byte array to read the file to subsequently create a Memory Stream. As far as I'm aware (I'm still a beginner when it comes to c#), a byte array can only contain 2GB of information before returning an overflow exception. To combat this I've tried to utilize FileStream.Read (second bit of code attached below) instead of System.IO.File.ReadAllBytes, though this again lead to an overflow exception of the byte Array. I know that at this point I'd have to split the file up, however, due to the rather limited documentation of the GDrive API for C# - at least from what I've seen - and my limited knowledge of C# I've got little to no clue on how to tackle this problem.

I'm sorry for the long read, all help on this matter is highly appreciated.

Upload Function V1 (System.IO.File.ReadAllBytes):

    private static Google.Apis.Drive.v3.Data.File UploadFile(Boolean useFolder, String mime, DriveService _service, string _uploadFile, string _parent, string _descrp = "")
    {
        if (System.IO.File.Exists(_uploadFile))
        {
            Google.Apis.Drive.v3.Data.File body = new Google.Apis.Drive.v3.Data.File
            {
                Name = System.IO.Path.GetFileName(_uploadFile),
                Description = _descrp,
                MimeType = mime
            };
            if (useFolder)
            {
                body.Parents = new List<string> { _parent };
            }
            byte[] byteArray = System.IO.File.ReadAllBytes(_uploadFile);
            MemoryStream stream = new System.IO.MemoryStream(byteArray);
            try
            {
                FilesResource.CreateMediaUpload request = _service.Files.Create(body, stream, mime);
                request.SupportsTeamDrives = true;
                request.Upload();
                return request.ResponseBody;
            }
            catch (Exception e)
            {
                Console.WriteLine("Error Occured: " + e);
                return null;
            }
        }
        else
        {
            Console.WriteLine("The file does not exist. 404");
            return null;
        }
    }

Upload Method V2 (FileStream):

    private static Google.Apis.Drive.v3.Data.File UploadFile(Boolean useFolder, String mime, DriveService _service, string _uploadFile, string _parent, string _descrp = "")
    {
        if (System.IO.File.Exists(_uploadFile))
        {
            Google.Apis.Drive.v3.Data.File body = new Google.Apis.Drive.v3.Data.File
            {
                Name = System.IO.Path.GetFileName(_uploadFile),
                Description = _descrp,
                MimeType = mime
            };
            if (useFolder)
            {
                body.Parents = new List<string> { _parent };
            }
            //byte[] byteArray = System.IO.File.ReadAllBytes(_uploadFile);
            using (FileStream fileStream = new FileStream(_uploadFile, FileMode.Open, FileAccess.Read))
            {
                Console.WriteLine("ByteArrayStart");
                byte[] byteArray = new byte[fileStream.Length];
                int bytesToRead = (int)fileStream.Length;
                int bytesRead = 0;
                while (bytesRead > 0)
                {
                    int n = fileStream.Read(byteArray, bytesRead, bytesToRead);
                    if (n == 0)
                    {
                        break;
                    }
                    bytesRead += n;
                    Console.WriteLine("Bytes Read: " + bytesRead);
                    bytesToRead -= n;
                    Console.WriteLine("Bytes to Read: " + bytesToRead);
                }
                bytesToRead = byteArray.Length;
                MemoryStream stream = new System.IO.MemoryStream(byteArray);
                try
                {
                    FilesResource.CreateMediaUpload request = _service.Files.Create(body, stream, mime);
                    request.SupportsTeamDrives = true;
                    request.Upload();
                    return request.ResponseBody;
                }
                catch (Exception e)
                {
                    Console.WriteLine("Error Occured: " + e);
                    return null;
                }
            }
        }
        else
        {
            Console.WriteLine("The file does not exist. 404");
            return null;
        }
    }

Upvotes: 1

Views: 724

Answers (1)

Linda Lawton - DaImTo
Linda Lawton - DaImTo

Reputation: 117226

MemoryStream's constructors only work with byte arrays that are limited to Int32.MaxValue bytes. Why not just use your FileStream object directly?

var fileMetadata = new Google.Apis.Drive.v3.Data.File()
{
    Name = "flag.jpg"
};

FilesResource.CreateMediaUpload request;
using (var stream = new System.IO.FileStream(@"C:\temp\flag.jpg", System.IO.FileMode.Open))
{
    request = service.Files.Create(fileMetadata, stream, "image/jpeg");
    request.Fields = "id";
    request.Upload();
}
var file = request.ResponseBody;

Really a file that big you should be using resumable upload but im going to have to dig around for some sample code for that.

Upvotes: 4

Related Questions