Unable to drain/cancel Dataflow job. It keeps pending state

Question

Some jobs are remaining with pending pending state and I can't cancel them.

How do I cancel the job.

Web console shows like this.

• "The graph is still being analyzed."
• All logs are "No entries found matching current filter."
• Job status: "Starting..." There isn't appered a cancel button yet.

There are no instances in the Compute Engline tab.

What I did is below. I created a streaming job. it was simple template job, Pubsub subscription to BigQuery. I set machineType as e2-micro because it was just a testing.

I also tried to drain and cancel by gcloud but it doesn't work.

$ gcloud dataflow jobs drain --region asia-northeast1 JOBID

Failed to drain job [...]: (...): Workflow modification failed. Causes: (...): 
Operation drain not allowed for JOBID. 
Job is not yet ready for draining. Please retry in a few minutes. 
Please ensure you have permission to access the job and the `--region` flag, asia-northeast1, matches the job's
region.

This is jobs list

$ gcloud dataflow jobs list --region asia-northeast1
JOB_ID  NAME                               TYPE       CREATION_TIME        STATE      REGION
JOBID1  pubsub-to-bigquery-udf4            Streaming  2021-02-09 04:24:23  Pending    asia-northeast1
JOBID2  pubsub-to-bigquery-udf2            Streaming  2021-02-09 03:20:35  Pending    asia-northeast1
...other jobs...

Please let me know how to stop/cancel/delete these streaming jobs.

Job IDs:

• 2021-02-08_20_24_22-11667100055733179687
• 2021-02-08_20_24_22-11667100055733179687

WebUI: https://i.sstatic.net/B75OX.png

https://i.sstatic.net/LzUGQ.png

Answer 1

As per personal experience some time few instance get stuck either they keep on running, or they cannot be canceled or you can not see thr graphical data flow pipelines. Best way to handle this kind of issue is to leave them in thr status, unless it is not impacting your solution by exceeding maximum concurrent runs at a moment. It will be canceled automatically or by Google team, since Dataflow is a google managed.

Answer 2

In GCP console Dataflow UI, if you have running Dataflow jobs, you will see the "STOP" button just like the below image.

Press the STOP button.

When you successfully stop your job, you will see the status like below. (I was too slow to stop the job with the first try, so I had to test it again. :) )