OutOfMemoryException when upload file to Google Drive using Google Drive API

Question

This is a web application.I am tying to upload large files to Google Drive using GoogleDrive API. My code works fine with smaller files. But when it comes to larger files error occurs.

I had search a lot, but couldn't find a solution for work and some of the tread that do not has a answer.

Error occurs in the code part where the the file is converted into byte[].

byte[] fileData = System.IO.File.ReadAllBytes(dir);

For web.config I had trying to change the request length, but it not work for a large files.

<httpRuntime targetFramework="4.5" executionTimeout="520000" maxRequestLength="920000"  />

Anyone has a solution for this? Thanks in advance!

Below is the full code.

---

using System;
using System.Diagnostics;
using DotNetOpenAuth.OAuth2;
using Google.Apis.Authentication.OAuth2;
using Google.Apis.Authentication.OAuth2.DotNetOpenAuth;
using Google.Apis.Drive.v2;
using Google.Apis.Drive.v2.Data;
using Google.Apis.Util;
using ASPSnippets.GoogleAPI;
using System.Web.Script.Serialization;
using System.Web;
using System.Collections.Generic;
using System.Net.Http;
using System.Net;
using System.IO;
using System.Reflection;

namespace GoogleDriveAutoUpdate
{
    public partial class GoogleDriveSample : System.Web.UI.Page
    {
        static int flagUpdateFile;
        protected string[] dirs = Directory.GetFiles(@"C:\Users\RNKP74\Desktop\GoogleDrive");


        protected void Page_Load(object sender, EventArgs e)
        {
            GoogleConnect.ClientId = "";
            GoogleConnect.ClientSecret = "";
            GoogleConnect.RedirectUri = Request.Url.AbsoluteUri.Split('?')[0];
            GoogleConnect.API = EnumAPI.Drive;

            if (string.IsNullOrEmpty(Request.QueryString["code"]))
                GoogleConnect.Authorize("https://www.googleapis.com/auth/drive.file");

            if (flagUpdateFile == 0)
                UploadFile();
        }

        protected HttpPostedFile ConstructHttpPostedFile(byte[] data, string filename, string contentType = null)
        {
            // Get the System.Web assembly reference
            Assembly systemWebAssembly = typeof(HttpPostedFileBase).Assembly;
            // Get the types of the two internal types we need
            Type typeHttpRawUploadedContent = systemWebAssembly.GetType("System.Web.HttpRawUploadedContent");
            Type typeHttpInputStream = systemWebAssembly.GetType("System.Web.HttpInputStream");

            // Prepare the signatures of the constructors we want.
            Type[] uploadedParams = { typeof(int), typeof(int) };
            Type[] streamParams = { typeHttpRawUploadedContent, typeof(int), typeof(int) };
            Type[] parameters = { typeof(string), typeof(string), typeHttpInputStream };

            // Create an HttpRawUploadedContent instance
            object uploadedContent = typeHttpRawUploadedContent
                .GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, uploadedParams, null)
                .Invoke(new object[] { data.Length, data.Length });

            // Call the AddBytes method
            typeHttpRawUploadedContent
                .GetMethod("AddBytes", BindingFlags.NonPublic | BindingFlags.Instance)
                .Invoke(uploadedContent, new object[] { data, 0, data.Length });

            // This is necessary if you will be using the returned content (ie to Save)
            typeHttpRawUploadedContent
                .GetMethod("DoneAddingBytes", BindingFlags.NonPublic | BindingFlags.Instance)
                .Invoke(uploadedContent, null);

            // Create an HttpInputStream instance
            object stream = (Stream)typeHttpInputStream
                .GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, streamParams, null)
                .Invoke(new object[] { uploadedContent, 0, data.Length });

            // Create an HttpPostedFile instance
            HttpPostedFile postedFile = (HttpPostedFile)typeof(HttpPostedFile)
                .GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, parameters, null)
                .Invoke(new object[] { filename, contentType, stream });

            return postedFile;
        }

        protected void UploadFile()
        {
            int successCount = 0;
            HttpFileCollection MyFileCollection = Request.Files;
            Response.Write("Total number file = " + dirs.Length + "<br/>");

            foreach (string dir in dirs)
            {

                byte[] fileData = System.IO.File.ReadAllBytes(dir);
                string fileName = Path.GetFileName(dir);
                flagUpdateFile = 1;
                //Content Type is null
                Session["File"] = ConstructHttpPostedFile(fileData, fileName);

                //Token return from Google API
                if (!string.IsNullOrEmpty(Request.QueryString["code"]) && flagUpdateFile == 1)
                {
                    string code = Request.QueryString["code"];
                    string json = GoogleConnect.PostFile(code, (HttpPostedFile)Session["File"], "");
                    flagUpdateFile = 0;
                    System.IO.File.Delete(dir);
                    successCount++;
                }

                if (Request.QueryString["error"] == "access_denied")
                {
                    ClientScript.RegisterClientScriptBlock(this.GetType(), "alert", "alert('Access denied.')", true);
                    Console.Write("Access denied,check your Authorized redirect URIs!");
                }

                HttpContext.Current.Session.Remove("File");
                HttpContext.Current.Session.Abandon();

            }
            Response.Write("Total file uploaded = " + successCount);
            successCount = 0;
            Response.Write("<span id='Label1' style='color:red'><br/>Sucess if the number is identical</span>");

        }


    }
}

Answer 1

Acording to MSDN you can setup it at web.config as:

<configuration>
  <runtime>
    <gcAllowVeryLargeObjects enabled="true" />
  </runtime>
</configuration>

but its says:

true: Arrays greater than 2 GB in total size are enabled on 64-bit platforms.

so its not work on 32bit, meaning that you also need to run your pool on 64-bit.

ref: http://msdn.microsoft.com/en-us/library/hh285054.aspx

Answer 2

I admit I don't know the GoogleDrive API, but your code seems incredible complicated. Why do you do reflection at all and why for each file? Why do you load your whole file into a byte array just to put it in a MemoryStream, when you could be using a FileStream from the start, that doesn't copy your whole file into memory?

I'd say you need to simplify your code. All that Reflection? Write the actual two lines of code you want to write. Then use a filestream as InputStream.

You seem to want to fit all your files into a single request. The API seems to want you to fit a single file in one request. Maybe you should do that.