CODEWITHSUNDEEP
javagoogle-app-enginegoogle-cloud-platformtasktask-queue
Craig D
Craig D

Reputation: 411

Google Cloud Platform Default Queue Pausing Itself

In the last 3 days, our GCP "default" queue went into pause by itself 5 times (we didn't manually pause it). This never happened in the past, ever and we've been using GAE / GCP task queues for 4 years. Manually resuming the queue resolves the problem, but it's super inconvenient as we're using it for processing small to medium size tasks all day long which are critical to the business. We only create about 200-300 tasks per day at most, however.

Has something changed or broken?

Coincidentally, a few days ago a strange error appeared when deploying our app engine standard app where it failed at the very end of the deploy referring to a queue issue. I found a link in the error message where it went to a page to enable an API. I had to enable that API (which referred to either a task or queue api), which resolved the deploy issue. I was quite sure our app isn't using that API (we don't manage the queue at all in our app) - we only add tasks to the queue.

I'm not sure if the 2 events are related, but seems very coincidently and could be the proximate cause.

I turned on logging for the default queue, no messages specific to the pause event appear at the time it paused using "default" level of severity. There are messages showing tasks arriving in queue.

I did a Google search for this issue, couldn't find it.

Upvotes: 1

Views: 770

Answers (1)

Craig D
Craig D

Reputation: 411

I was able to determine that the default queue went into pause every time I deployed a new version to app engine. I was able to correct the issue by defining the default queue in the queue.xml file, which previously only had the backup queue. This is my new queue.xml file...

<queue-entries>    
  <queue>
    <name>default</name>
    <rate>5/s</rate>
    <max-concurrent-requests>100</max-concurrent-requests>
    <retry-parameters>
      <!-- only 20 retry attempts (default is unlimited) -->
      <task-retry-limit>20</task-retry-limit>
    </retry-parameters>    
  </queue>

  <queue>
    <name>backup-queue</name>
    <rate>15/s</rate>
    <!-- recommended to be rate/5 -->
    <bucket-size>3</bucket-size>
    <max-concurrent-requests>10</max-concurrent-requests>
    <target>ah-builtin-python-bundle</target>
    <retry-parameters>
      <!-- only 2 retry attempts -->
      <task-retry-limit>2</task-retry-limit>
    </retry-parameters>
  </queue>
</queue-entries>

Upvotes: 3

Related Questions