CODEWITHSUNDEEP
pythonredis
Snackerino
Snackerino

Reputation: 149

python - how can I make getting redis keys from for loop faster

I am using python and the redis package to get/set keys to redis. For saving data into redis I am using pipelines, can I use them to make this process of getting data from redis faster?

def compare_list_by_action_and_user(self, source_list: list, user: str, action: str):
        """Compares given list against combination of user and action data in redis"""
        compared_list = []
        for row in source_list:
            compared_action = self.__redis.get(f'{user.upper()}_{action.upper()}_{row.message_id}')

            if compared_action is not None:
                compared_list.append(ComparedLoggedAction(row, pickle.loads(compared_action)))

        return compared_list

Basically this method has a given source_list through which I iterate. For every object in the list I try to get the redis key by the given pattern. Then if the value is not empty I create a new object from the given object from list and the unpickled object from redis.

Could I use pipelines to make this faster? On 400 000 keys it takes about 30 minutes this way.

Upvotes: 2

Views: 820

Answers (1)

Anton
Anton

Reputation: 4052

Redis MGET command seems to be the right option here.

Redis python would expose it as:

self.__redis.mget('key1','key2',...)

Upvotes: 2

Related Questions