CODEWITHSUNDEEP
javaspringmongodbspring-dataspring-data-mongodb
marknorkin
marknorkin

Reputation: 4074

Spring Data Mongo: How to save batch ignoring all duplicate key errors?

I have the following domain object:

@Document
class Foo {
    @Id
    private final String bar;
    private final String baz;
    // getters, setters, constructor omitted
}

Which is inserted as follows:

Collection<Foo> foos = ...;
mongoTemplate.insert(foos, Foo.class);

How to save all results in one call ignoring all duplicate key exceptions ?

Upvotes: 5

Views: 5392

Answers (2)

Johannes Rudolph
Johannes Rudolph

Reputation: 35731

In my case it was not suitable to allow modification/overwriting of the existing documents as in @marknorkin's answer. Instead, I only wanted to insert new documents. I came up with this using MongoOperations, which is injectable in Spring. The code below is in Kotlin. 

 try {
        // we do not want to overwrite existing documents, especially not behind the event horizon
        // we hence use unordered inserts and supresss the duplicate key exceptions
        // as described in: https://docs.mongodb.com/v3.2/reference/method/db.collection.insertMany/#unordered-inserts
        mongoOps.bulkOps(BulkOperations.BulkMode.UNORDERED, EventContainer::class.java)
            .insert(filtered)
            .execute()
      } catch (ex: BulkOperationException) {
        if (!isDuplicateKeyException(ex)) {
          throw ex
        }
      }

With this little helper

private fun isDuplicateKeyException(ex: BulkOperationException): Boolean {
    val duplicateKeyErrorCode = 11000
    return ex.errors.all { it.code == duplicateKeyErrorCode }
}

Upvotes: 5

marknorkin
marknorkin

Reputation: 4074

I searched through spring data mongo documentation and other resources, but didn't find expected answer.

Seems like Mongo inserts batch docs until unique key constraint is met, and it's up to DB to decide.

So for example if you need to insert 100 docs and document on position 50 already exists in DB then the first 49 will be inserted and the second 50 will not.

What I came up is the next solution:

Set<String> ids = foos.stream().map(Foo::getBar).collect(toSet()); // collect all ids from docs that will be inserted
WriteResult writeResult = mongoTemplate.remove(new Query(Criteria.where("_id").in(ids)), Foo.class); // perform remove with collected ids
mongoTemplate.insert(foos, Foo.class); // now can safely insert batch

So DB will be called twice. Also as bar is indexed field the remove operation will be fast.

Upvotes: 4

Related Questions