visingh
visingh

Reputation: 233

Aerospike taking too long to insert 10 million key values

Aerospike taking almost 70 min to insert 10 million key values. Steps that I follow to insert are:

  1. I have all key value in memory

  2. register aerospike udf on aerospike server

  3. start iterating 10 million time for java code and call aerospike udf module for each key and value.

  4. Inside udf, I am checking whether key is already present. If yes then update otherwise insert.

  5. I have 2 aerospike server with replica factor 2.

Is there any way to increase write performance or perform batch write. I know that aerospike doesn't support batch write. Any suggestion to optimized write operation.

Code that i used

private void put(String namespace, String setName, String binName, String keyVlaue, List<Integer> listIdsToBeAdd, List<Integer> listIdToBeRemoved) {
    Key key = new Key(namespace, setName, keyVlaue);
    getAerospikeClient().execute(writePolicy, key, "aerospike_udf", "update_record", Value.get(listIdsToBeAdd), Value.get(listIdToBeRemoved), Value.get(binName));
}

UDF module

function update_record( record, list1, list2, bin_name) 
  local store_list = record[bin_name]
  -- get a list from database record. Create one if it doesn't exits
  if store_list == nil then
    store_list = map()
  end

  if list2 then
    for i=1, #list2 do
      map.remove(store_list, list2[i])
    end
  end

  if list1 then
    for i=1, #list1 do
      store_list[list1[i]] = 1
    end
  end

  if map.size(store_list) == 0 then
    record[bin_name] = nil
  else
    record[bin_name] = store_list
  end

  if not aerospike:exists(record) then
    aerospike:create(record)
  else
    aerospike:update(record)
  end 
end

Upvotes: 0

Views: 921

Answers (1)

visingh
visingh

Reputation: 233

In place of using sync Aerospike client to execute, i start using Async Aerospike client. Now i am able to insert 10 million record within ~2.6 min.

Upvotes: 1

Related Questions