CODEWITHSUNDEEP
cadence-workflowuber-cadence
Voobr
Voobr

Reputation: 243

reset failed (code:deadline-exceeded message:timeout)

We are able to use the Cadence cli to read data from the system just fine (e.g. listing workflows, listing the history of a given workflow, etc).

However, when trying to reset a workflow, we are receiving a timeout error. We've tried increasing the timeout manually (via the --context_timeout flag), but that doesn't seem to help.

We've checked the workflow's history, and it doesn't seem like anything is happening to the workflow in question (i.e. no new run has been started or anything like that).

./cli.sh --context_timeout 240 --domain ourdomain workflow reset -w 15a3839e-d1b6-41f5-ac50-fd262f0f4bfb -r 17551fe8-ce21-4272-8101-8f9bccc614c9 --reset_type LastDecisionCompleted --reason "Retrying due to a failed activity"

Error: reset failed
Error Details: code:deadline-exceeded message:timeout
Stack trace:
goroutine 1 [running]:
runtime/debug.Stack(0xd, 0x0, 0x0)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/uber/cadence/tools/cli.printError(0x1f70b43, 0xc, 0x23f24c0, 0xc000762180)
        /cadence/tools/cli/util.go:536 +0x2ad
github.com/uber/cadence/tools/cli.ErrorAndExit(0x1f70b43, 0xc, 0x23f24c0, 0xc000762180)
        /cadence/tools/cli/util.go:547 +0x49
github.com/uber/cadence/tools/cli.ResetWorkflow(0xc000410f20)
        /cadence/tools/cli/workflowCommands.go:1545 +0x6b8
github.com/uber/cadence/tools/cli.newWorkflowCommands.func20(0xc000410f20)
        /cadence/tools/cli/workflow.go:251 +0x2b
github.com/urfave/cli.HandleAction(0x1b702c0, 0x2027280, 0xc000410f20, 0xc000410f20, 0x0)
        /go/pkg/mod/github.com/urfave/[email protected]/app.go:528 +0x127
github.com/urfave/cli.Command.Run(0x1f663bc, 0x5, 0x0, 0x0, 0xc0000817a0, 0x1, 0x1, 0x1fc5c3b, 0x33, 0x0, ...)
        /go/pkg/mod/github.com/urfave/[email protected]/command.go:174 +0x528
github.com/urfave/cli.(*App).RunAsSubcommand(0xc0002d2a80, 0xc000410c60, 0x0, 0x0)
        /go/pkg/mod/github.com/urfave/[email protected]/app.go:407 +0x882
github.com/urfave/cli.Command.startApp(0x1f6b11a, 0x8, 0x0, 0x0, 0xc000081b20, 0x1, 0x1, 0x1f88e40, 0x18, 0x0, ...)
        /go/pkg/mod/github.com/urfave/[email protected]/command.go:373 +0x845
github.com/urfave/cli.Command.Run(0x1f6b11a, 0x8, 0x0, 0x0, 0xc000081b20, 0x1, 0x1, 0x1f88e40, 0x18, 0x0, ...)
        /go/pkg/mod/github.com/urfave/[email protected]/command.go:102 +0x9d6
github.com/urfave/cli.(*App).Run(0xc000350000, 0xc0000d6000, 0xf, 0xf, 0x0, 0x0)
        /go/pkg/mod/github.com/urfave/[email protected]/app.go:279 +0x731
main.main()
        /cadence/cmd/tools/cli/main.go:36 +0x4f

We have also tried using --event_id instead of --reset_type, and we've tried operating on different workflow ids, but we're still encountering the same timeout error.

Are we using the cli incorrectly?

Upvotes: 0

Views: 705

Answers (1)

Long Quanzheng
Long Quanzheng

Reputation: 2391

You are probably hitting a bug of resetting workflows if there is a current open run.

https://github.com/uber/cadence/pull/3727

The bug was introduced into 0.15.x. It's currently fixed in master, and we are working on release in 0.16.0

As a workaround, you can terminate the current open run of workflow and then reset it, then reset closed workflows should work.

The bug still bites if the runID that you want to reset bases on is closed, as long as there is a open runID for the same workflowID. Because the issue in the bug is having problem terminating the current run. But reset require terminating the current one so that it can start a new one.

You can find the current open runID by:

./cadence --domain ourdomain workflow describe -w 15a3839e-d1b6-41f5-ac50-fd262f0f4bfb

Without runID, this will return the current runID if that's running/open.

Upvotes: 1

Related Questions