Reputation: 233
Aerospike taking almost 70 min to insert 10 million key values. Steps that I follow to insert are:
I have all key value in memory
register aerospike udf on aerospike server
start iterating 10 million time for java code and call aerospike udf module for each key and value.
Inside udf, I am checking whether key is already present. If yes then update otherwise insert.
I have 2 aerospike server with replica factor 2.
Is there any way to increase write performance or perform batch write. I know that aerospike doesn't support batch write. Any suggestion to optimized write operation.
Code that i used
private void put(String namespace, String setName, String binName, String keyVlaue, List<Integer> listIdsToBeAdd, List<Integer> listIdToBeRemoved) {
Key key = new Key(namespace, setName, keyVlaue);
getAerospikeClient().execute(writePolicy, key, "aerospike_udf", "update_record", Value.get(listIdsToBeAdd), Value.get(listIdToBeRemoved), Value.get(binName));
}
UDF module
function update_record( record, list1, list2, bin_name)
local store_list = record[bin_name]
-- get a list from database record. Create one if it doesn't exits
if store_list == nil then
store_list = map()
end
if list2 then
for i=1, #list2 do
map.remove(store_list, list2[i])
end
end
if list1 then
for i=1, #list1 do
store_list[list1[i]] = 1
end
end
if map.size(store_list) == 0 then
record[bin_name] = nil
else
record[bin_name] = store_list
end
if not aerospike:exists(record) then
aerospike:create(record)
else
aerospike:update(record)
end
end
Upvotes: 0
Views: 921
Reputation: 233
In place of using sync Aerospike client to execute, i start using Async Aerospike client. Now i am able to insert 10 million record within ~2.6 min.
Upvotes: 1