CODEWITHSUNDEEP
akkaakka-persistence
Pavel Voronin
Pavel Voronin

Reputation: 13993

akka.persistence.RecoveryTimedOut: Recovery timed out, didn't get snapshot within 30000 milliseconds

I have a failing test because of the timeout. Here's what I see in log output:

2018-05-15 10:47:56.152 WARN  com.datastax.driver.core.NettyUtil [UserDataServiceSpec-cassandra-plugin-default-dispatcher-27] [] [] - Found Netty's native epoll transport, but not running on linux-based operating system. Using NIO instead.
2018-05-15 10:48:38.616 ERROR n.f.c.indexing.UniquelyIndexingActor [UserDataServiceSpec-akka.actor.default-dispatcher-39] [UserDataServiceSpec-akka.actor.default-dispatcher-29] [akka.tcp://[email protected]:51627/user/$c/user-email-indexer] - Persistence failure when replaying events for persistenceId [user-email-indexer]. Last known sequence number [0]
akka.persistence.RecoveryTimedOut: Recovery timed out, didn't get snapshot within 30000 milliseconds
2018-05-15 10:48:38.617 ERROR a.c.sharding.PersistentShardCoordinator [UserDataServiceSpec-akka.actor.default-dispatcher-39] [UserDataServiceSpec-akka.actor.default-dispatcher-30] [akka.tcp://[email protected]:51627/system/sharding/userdataCoordinator/singleton/coordinator] - Persistence failure when replaying events for persistenceId [/sharding/userdataCoordinator]. Last known sequence number [0]
akka.persistence.RecoveryTimedOut: Recovery timed out, didn't get snapshot within 30000 milliseconds
2018-05-15 10:48:38.618 INFO  akka.actor.LocalActorRef [UserDataServiceSpec-akka.actor.default-dispatcher-39] [UserDataServiceSpec-akka.actor.default-dispatcher-35] [akka://UserDataServiceSpec/user/$c/user-email-indexer] - Message [akka.persistence.SnapshotProtocol$LoadSnapshotFailed] from Actor[akka://UserDataServiceSpec/system/cassandra-snapshot-store#-750137778] to Actor[akka://UserDataServiceSpec/user/$c/user-email-indexer#1201357728] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2018-05-15 10:48:38.619 INFO  akka.actor.LocalActorRef [UserDataServiceSpec-akka.actor.default-dispatcher-37] [UserDataServiceSpec-akka.actor.default-dispatcher-39] [akka://UserDataServiceSpec/system/sharding/userdataCoordinator/singleton/coordinator] - Message [akka.cluster.sharding.ShardCoordinator$RebalanceTick$] from Actor[akka://UserDataServiceSpec/system/sharding/userdataCoordinator/singleton/coordinator#-75387958] to Actor[akka://UserDataServiceSpec/system/sharding/userdataCoordinator/singleton/coordinator#-75387958] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

(Don't ask my why we are not using in memory storage to test persistent actors. It's not relevant to the problem right now.)

I am not experienced in Akka and JVM, but the messages I see are just the equivalence of "You screwed up, man". There is no hint in them how to fix the problem or why this RecoveryTimedOut occurs.

If someone could give me a valuable advice how to diagnose the problem, it would be nice.

UniquelyIndexingActor is created as cluster singleton.

Upvotes: 0

Views: 1470

Answers (1)

Shellum
Shellum

Reputation: 3179

Try adding this to your config:

akka {
  persistence {
    journal-plugin-fallback {
      recovery-event-timeout = 60s
    }
  }
}

It solved the problem for me. I found a reference to it in https://github.com/akka/akka/blob/master/akka-persistence/src/main/resources/reference.conf

Upvotes: 1

Related Questions