long-running job on GCP cloud run

Question

I am reading 10 million records from BigQuery and doing some transformation and creating the .csv file, the same .csv stream data I am uploading to SFTP server using Node.JS.

This job taking approximately 5 to 6 hrs to complete the request locally.

Solution has been delpoyed on GCP Cloud run but after 2 to 3 second cloud run is closing the container with 503 error.

Please find below configuration of GCP Cloud Run.

Autoscaling: Up to 1 container instances CPU allocated: default Memory allocated: 2Gi Concurrency: 10 Request timeout: 900 seconds

Is GCP Cloud Run is good option for long running background process?

Answer 1

If you use Cloud Run for long-running CPU-intensive tasks, there are several downsides to consider:

1. Request Timeouts: Cloud Run has a maximum request duration limit, which is currently set to 60 minutes. If your CPU-intensive task exceeds this limit, the request will be forcibly terminated. This can be problematic if your task requires more time to complete.

2. Resource Allocation: Cloud Run is designed to handle short-lived and stateless requests, where resources are allocated dynamically based on incoming traffic. However, long-running CPU-intensive tasks might require sustained high CPU usage, which could strain the resource allocation model of Cloud Run.

3. Cost Considerations: Cloud Run charges based on the CPU and memory resources consumed by your requests. Long-running CPU-intensive tasks can consume significant resources, leading to higher costs compared to alternative compute options specifically designed for sustained CPU-intensive workloads, such as Compute Engine or Kubernetes Engine.

Given these downsides, it's generally recommended to use Cloud Run for short-lived, stateless, and burstable workloads rather than long-running CPU-intensive tasks. If you have CPU-intensive tasks that require sustained high CPU usage and longer execution times, you can consider cloud run Jobs or other compute options like Compute Engine or Kubernetes Engine might be more suitable.

Cloud Run Job

Cloud Run Job is specifically designed for running short-lived, one-off tasks or batch jobs.

• It is suited for running tasks that have a defined start and end, such as data processing, periodic cron jobs, or ETL (Extract, Transform, Load) operations.
• Unlike regular Cloud Run services, Cloud Run Job instances are not automatically scaled based on incoming traffic.
• Cloud Run Job instances run until completion or until a configurable timeout is reached, allowing you to run tasks that may take longer to execute.

Answer 2

Update: 2021-Oct

Cloudrun supports background activities.

Configure CPU to be always-allocated if you use background activities
Background activity is anything that happens after your HTTP response has been delivered. To determine whether there is background activity in your service that is not readily apparent, check your logs for anything that is logged after the entry for the HTTP request.

Configure CPU to be always-allocated
If you want to support background activities in your Cloud Run service, set your Cloud Run service CPU to be always allocated so you can run background activities outside of requests and still have CPU access.

Answer 3

You can use a VM instance with your container deployed and perform you job on it. At the end kill or stop your VM.

But, personally, I prefer serverless solution and approach, like Cloud Run. However, Long running job on Cloud Run will come, a day! Until this, you have to deal with the limit of 60 minutes or to use another service.

As workaround, I propose you to use Cloud Build. Yes, Cloud Build for running any container in it. I wrote an article on this. I ran a Terraform container on Cloud Build, but, in reality, you can run any container.

Set the timeout correctly, take care of default service account and assigned role, and, thing not yet available on Cloud Run, choose the number of CPUs (1, 8 or 32) for the processing and speed up your process.

Want a bonus? You have 120 minutes free per day and per billing account (be careful, it's not per project!)

Answer 4

I will try to use Dataflow for creating .csv file from Big Query and will upload that file to GCS.

Answer 5

You can try using an Apache Beam pipeline deployed via Cloud Dataflow. Using Python, you can perform the task with the following steps:

Stage 1. Read the data from BigQuery table.

beam.io.Read(beam.io.BigQuerySource(query=your_query,use_standard_sql=True))

Stage 2. Upload Stage 1 result into a CSV file on a GCS bucket.

beam.io.WriteToText(file_path_prefix="", \
                    file_name_suffix='.csv', \
                    header='list of csv file headers')

Stage 3. Call a ParDo function which will then take CSV file created in Stage 2 and upload it to the SFTP server. You can refer this link.

Answer 6

Is GCP Cloud Run is good option for long running background process?

Not a good option because your container is 'brought to life' by incoming HTTP request and as soon as the container responds (e.g. sends something back), Google assumes the processing of the request is finished and cuts the CPU off.

Which may explain this:

Solution has been delpoyed on GCP Cloud run but after 2 to 3 second cloud run is closing the container with 503 error.

Answer 7

You may consider a serverless, event-driven approach:

• configure google storage trigger cloud function running transformation
• extract/export BigQuery to CF trigger bucker - this is the fastest way to get BigQuery data out

Sometimes exported data in that way may be too large not be suitable in that form for Cloud Function processing, due to restriction like max execution time (9 min currently) or memory limitation 2GB, In that case, you can split the original data file to smaller pieces and/or push then to Pub/Sub with storage mirror

All that said we've used CF to process a billion records from building bloom filters to publishing data to aerospike under a few minutes end to end.