Neo4j 2.1.5 incremental backup fails but full backup succeeds

Question

Very similar to this question but I am running the backup from Java and using against a local embedded Neo4J 2.1.5 Enterprise instance. Using:

OnlineBackup backup = OnlineBackup.from(backupTarget);
backup.backup(backupDirectory);
if (!backup.isConsistent()) {
    ...
}

The initial (full) backup works just fine, however running subsequent incremental backups fail (internally) with:

org.neo4j.consistency.ConsistencyCheckingError: Inconsistencies in transaction:
    Start[3,xid=GlobalId[NEOKERNL|9146491774354043793|781|-1], BranchId[ 52 49 52 49 52 49 ],master=-1,me=-1,time=2014-11-26 12:57:55.236+0000/1417006675236,lastCommittedTxWhenTransactionStarted=28650]
    2PC[3, txId=28651, 2014-11-26 12:57:55.254+0000/1417006675254]
    ConsistencySummaryStatistics{
        Number of errors: 2
        Number of warnings: 0
        Number of inconsistent RELATIONSHIP records: 2
    }

at org.neo4j.consistency.checking.incremental.intercept.CheckingTransactionInterceptor.complete(CheckingTransactionInterceptor.java:181) ~[neo4j-consistency-check-2.1.5.jar:2.1.5]
    at org.neo4j.kernel.impl.transaction.xaframework.LogEntryVisitorAdapter.apply(LogEntryVisitorAdapter.java:62) ~[neo4j-kernel-2.1.5.jar:2.1.5]
    at org.neo4j.kernel.impl.transaction.xaframework.LogEntryVisitorAdapter.apply(LogEntryVisitorAdapter.java:28) ~[neo4j-kernel-2.1.5.jar:2.1.5]
    at org.neo4j.kernel.impl.nioneo.xa.command.LogFilter.endLog(LogFilter.java:87) ~[neo4j-kernel-2.1.5.jar:2.1.5]
    at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.applyTransaction(XaLogicalLog.java:1121) ~[neo4j-kernel-2.1.5.jar:2.1.5]
    at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.applyCommittedTransaction(XaResourceManager.java:866) ~[neo4j-kernel-2.1.5.jar:2.1.5]
    at org.neo4j.kernel.impl.transaction.xaframework.XaDataSource.applyCommittedTransaction(XaDataSource.java:246) ~[neo4j-kernel-2.1.5.jar:2.1.5]
    at org.neo4j.com.ServerUtil.applyReceivedTransactions(ServerUtil.java:461) ~[neo4j-com-2.1.5.jar:2.1.5]
    at org.neo4j.backup.BackupService.unpackResponse(BackupService.java:423) ~[neo4j-backup-2.1.5.jar:2.1.5]
    at org.neo4j.backup.BackupService.incrementalWithContext(BackupService.java:330) ~[neo4j-backup-2.1.5.jar:2.1.5]
    at org.neo4j.backup.BackupService.doIncrementalBackup(BackupService.java:269) ~[neo4j-backup-2.1.5.jar:2.1.5]
    at org.neo4j.backup.BackupService.doIncrementalBackup(BackupService.java:222) ~[neo4j-backup-2.1.5.jar:2.1.5]
    at org.neo4j.backup.BackupService.doIncrementalBackupOrFallbackToFull(BackupService.java:243) ~[neo4j-backup-2.1.5.jar:2.1.5]
    at org.neo4j.backup.OnlineBackup.backup(OnlineBackup.java:93) ~[neo4j-backup-2.1.5.jar:2.1.5]
    at com.clarifimedia.backup.runner.Neo4jBackupRunner.backup(Neo4jBackupRunner.java:86) ~[land-backup-1.0-SNAPSHOT.jar:na]
    at com.clarifimedia.backup.runner.Neo4jBackupRunner$1.run(Neo4jBackupRunner.java:67) ~[land-backup-1.0-SNAPSHOT.jar:na]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) ~[spring-context-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) [spring-context-3.2.8.RELEASE.jar:3.2.8.RELEASE]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_65]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_65]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_65]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [na:1.7.0_65]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]

The database will be experiencing a number of writes between each backup and quiet possibly at the time of backup. I am aware that I could use the backup(backupTarget, false) variant to presumably avoid the issue but I would worry about the quality of my backup. I could also always force a full backup but that seems like a sledgehammer approach that might not scale so well as our database grows.

Possible tangent, when using a previous version of Neo I was (only occasionally, weirdly) seeing this warning on both full and incremental backups (the Node was always a spatial bbox, and there were 7 reported errors, not 2):

2014-11-25 18:00:07.457+0000 INFO  [org.neo4j]: ERROR: The property chain contains multiple properties that have the same property key id, which means that the entity has at least one duplicate property.
    Node[5,used=true,group=14,prop=42,labels=Inline(0x0:[]),light]

So the question is, how can I perform a backup that I can trust?

Edit - 2.1.6

It ran the full and several incremental backups without fail (running at 15 minute intervals), but as soon as any writes were made to the database the backup started failing with:

org.neo4j.consistency.ConsistencyCheckingError: Inconsistencies in transaction:
    Start[3,xid=GlobalId[NEOKERNL|-7692684822052087636|1385|-1], BranchId[ 52 49 52 49 52 49 ],master=-1,me=-1,time=2014-11-27 01:00:04.487+0000/1417050004487,lastCommittedTxWhenTransactionStarted=28847]
    1PC[3, txId=28848, 2014-11-27 01:00:04.491+0000/1417050004491]
    ConsistencySummaryStatistics{
    Number of errors: 1
    Number of warnings: 0
    Number of inconsistent PROPERTY records: 1

}

I can delete the backup folder and run a full backup without issue.

Answer 1

Could you check if the issue persists using the recently released 2.1.6? According to the release notes some backup related issues were fixed.