CODEWITHSUNDEEP
google-cloud-datastore
Siva Reddy Vippala
Siva Reddy Vippala

Reputation: 211

Best way to re-persist millions of data-store entities?

What is the best way to re-persist a data-store kind, which has 40 millions of entities in it. [The reason why I have to re-persist is to have the index created on one of the existing attributes]?

Kindly provide some sample code if possible?

BTW, I have a crude way of doing as in using Remote API and implemented Executor Service to run multiple parallel threads, each thread re-persists a batch of 100 entities.

I think there should be some other better approach with GAE?

Appreciate your suggestions?

Upvotes: 0

Views: 63

Answers (1)

pgiecek
pgiecek

Reputation: 8220

Look at MapReduce for App Engine. I think it could be the right solution for you.

MapReduce is a programming model for processing large amounts of data in a parallel and distributed fashion. It is useful for large, long-running jobs that cannot be handled within the scope of a single request.

Upvotes: 1

Related Questions