CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-functionsgoogle-cloud-storage
user5155835
user5155835

Reputation: 4742

Cloud Function - Getting file contents more than 10 MB

The Google Cloud Function's quota page: https://cloud.google.com/functions/quotas says that the Maximum amount of Data sent by HTTP Function in an HTTP Response is 10 MB. I have files of 100 MB in Google Storage Bucket which I want to stream from the Cloud Function to my application. Due to security aspects of our architecture, my application cannot read directly from the Storage Bucket.

A Sample Go Code in the Cloud Function is:

func MyFn(w http.ResponseWriter, r *http.Request) {

    var appendedBytes []byte
    for {
        decryptedFileBuffer, err := fh.GetNextDecryptedPage()
        if err != nil {
            break
        }
        appendedBytes = append(appendedBytes, decryptedFileBuffer...)
    }
    w.Write(appendedBytes)
}

If I try to get files more than 10 MB using Cloud Function, I get following errors:

Error: incorrect function response. Function invocation was interrupted.

Error: could not handle the request

How can I get files from than 10 MB from the Cloud Function?

Upvotes: 4

Views: 2744

Answers (1)

Kolban
Kolban

Reputation: 15246

When we look at the quota page, we see a no against whether or not the quota can be increased. A new player in town is Google Cloud Run ... see here. This provides very similar capabilities to Cloud Functions but uses Kubernetes as the function hosting environment. It's quota allowance is 32MB ... but this too is less than what you need and is also flagged as no for increase.

An immediate thought that occurs to me is to trade convenience (a simple cloud function) for flexibility and run your own compute resources on GCP. For example, running up Compute Engines, App Engines or a Kubernetes cluster where the contained application serves responses through a Web Server is a perfectly fine proposition and not constrained in throughtput. You will have to do the math on billing to determine the cost differentials as the metrics will be different but from a technical perspective, the game changes dramatically. The cloud functions/cloud run capabilities are there primarily to support very short natured microservices and I am guessing that the transmission of 100MB directly is not considered a microservice. It also may not be as cost effective as one may think. It is my understanding that Cloud Functions bills per unit of time utilized. Transmitting 100MBs is likely to be slow latency proposition and may not be as cost effective compared to always running compute as we may think.

Upvotes: 4

Related Questions