CODEWITHSUNDEEP
akkaakka-persistence
Karan Khanna
Karan Khanna

Reputation: 2157

Akka: Persistence failure when replaying events

We are working on an event sourced application with akka-persistance using Oracle database as event store. The application have been running in production for sometime now. Lately we are seeing the following error in the application for some of the persistent actors.

Persistence failure when replaying events for persistenceId [some-persistence-id]. Last known sequence number [0]

Can someone who faced a similar issue in their application share their experience of why this happens?

Also, going through Akka documentation at: https://doc.akka.io/docs/akka/current/persistence.html, onRecoveryFailure is responsible for handling such failures. Is there a way we can override this method to ignore the persisted events in case we see failures while replaying events? In our scenario replaying the events is not very critical and we can serve the users even by ignoring the m.

Upvotes: 0

Views: 1053

Answers (2)

Roiocam
Roiocam

Reputation: 116

I guess the exeption you met was:

akka.persistence.typed.internal.JournalFailureException: Exception during recovery. Last known sequence number [0]. PersistenceId [PersistenceID], due to: Replay timed out, didn't get event within [30000 milliseconds], highest sequence number seen [0]
    at akka.persistence.typed.internal.ReplayingEvents.onRecoveryFailure(ReplayingEvents.scala:267)
    at

In akka-persistence internal, replaying is an asynchronous behavior, which means it does not block the actor thread.

However, when the database (journal plugin) crashes/degradation, you cannot indefinitely keep an actor in a replaying state. To handle this situation, akka-persistence initiates a timer to cancel the replaying step. By default, this timer is set to a duration of 30 seconds.

If you encounter the exception log again, it indicates that the database is degrading or experiencing issues.

Upvotes: 0

Levi Ramsey
Levi Ramsey

Reputation: 20591

That log is typically a manifestation of something else. Since the failure is from sequence number zero, that points an actual query to the DB failing (e.g. timeout). There should be other logs around the time of that log which will provide further information.

Akka Persistence has a fairly strong assumption that the persisted state is important (otherwise why would you be persisting?). Off the top of my head, I would consider separating the parts of the actor which are affected by persistence from the parts which aren't: the non-persistent actor can spawn a persistent child and interact with it (it can do tricks with stashing, for instance, to present an illusion that it and its child are the same actor).

Upvotes: 1

Related Questions