CODEWITHSUNDEEP
cadence-workflowuber-cadence
user2052668
user2052668

Reputation: 35

Issues with matching service in Cadence

Two days ago, we started presenting some issues with our cadence setup. The first thing we noticed is the Open workflows were not disappearing from the list once they completed. For example this workflow appears as Open in the list: enter image description here

But when you click on it, you will see that it’s actually completed:

enter image description here

At the same time this started to happen, we noticed how several workflows would take quite a long time to complete, several of them would stuck in “Schedule” states and never go further from there. After checking the logs, the only error we saw was this:

{"level":"error","ts":"2021-03-06T19:12:04.865Z","msg":"Persistent store operation failure","service":"cadence-matching","component":"matching-engine","wf-task-list-name":"cadence-sys-history-scanner-tasklist-0","wf-task-list-type":1,"store-operation":"create-task","error":"InternalServiceError{Message: CreateTasks operation failed. Error : Request on table cadence.tasks with ttl of 630720000 seconds exceeds maximum supported expiration date of 2038-01-19T03:14:06+00:00. In order to avoid this use a lower TTL, change the expiration date overflow policy or upgrade to a version where this limitation is fixed. See CASSANDRA-14092 for more details.}","wf-task-list-name":"cadence-sys-history-scanner-tasklist-0","wf-task-list-type":1,"number":6300094,"next-number":6300094,"logging-call-at":"taskWriter.go:176","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/cadence/common/log/loggerimpl/logger.go:134\ngithub.com/uber/cadence/service/matching.(*taskWriter).taskWriterLoop\n\t/cadence/service/matching/taskWriter.go:176"}

Does somebody have an idea of why this is happening?

Upvotes: 0

Views: 362

Answers (1)

Long Quanzheng
Long Quanzheng

Reputation: 2391

The first one is because of visibility sampling being enabled by default(to protect default core DB). You can disable it by configure system.enableVisibilitySampling to false.

But when you doing that, it’s better to separate the visibility and default store into different database cluster so that visibility doesn’t bring down the default(core data model) DB.

see more in https://github.com/uber/cadence/issues/3884

The second is a bug fixed in 0.16.0 It should be resolved if you upgrade server.

See https://github.com/uber/cadence/pull/3627 and https://docs.datastax.com/en/dse-trblshoot/doc/troubleshooting/recoveringTtlYear2038Problem.html

Upvotes: 1

Related Questions