CODEWITHSUNDEEP
postflaskpymongoeve
ben26941
ben26941

Reputation: 1730

How to do bulk lookups of mongo documents with post in Eve?

I am using Eve (flask) as an API interface to mongodb. Now I am trying to set up bulk lookups of items over this interface. This means I want to find documents in bulk.

Usually you would use get for this. So, for example, if you have a collection of persons with fields name and phone number, you could get request the endpoint /person/ less name greater and get back the corresponding document including the phone number.

If you want to lookup several documents at once, say, following our example, you need the phone number of several people, the lookup of each person separately would be expensive, i.e. it would take a long time to get the phone numbers.

You can use something like ?where={"name" $in ["Pedro", "Juan"]} to get the phone numbers for two persons with one get request, which is more efficient (=faster) than doing two get requests, one for each. However, this method has a limited applicability because the maximum length of get requests is about 64K (1), which means that only on an order of thousand documents could be retrieved this way, if we assume name lengths of around 10-20 characters.

The natural method for larger lookups seems therefore using post requests. However, the eve post interface, as far as I can see, is for insertions, not for lookups (find).

I've implemented manually a post lookup for some endpoints with flask, but is there a way to use Eve for this? To the point: is there a way to do bulk lookups of documents with post requests?

Upvotes: 0

Views: 205

Answers (1)

Nicola Iarocci
Nicola Iarocci

Reputation: 6576

You could use a dynamic lookup filter. Basically you hook a callback function to every GET request. In your callback then update the lookup as needed to (source):

def pre_GET(resource, request, lookup):
    lookup["name"] = {'$in': list_of_names}


 app = Eve()

 app.on_pre_GET += pre_GET
 app.run()

Or you could even set a predefined data filter (source):

people = {
    'datasource': {
        'filter': {'username': {'$exists': True}}
        }
}

The second option makes the filter static, but you can still mix it with a dynamic filter.

In general though, this whole thing smells bad design. I don't know your use case, but consider alternative such as, maybe, introducing a "customer_type" field to query on. That would probably make it easier on both performance and code maintenance.

In any case, if you are doing queries, POST is not the place to look for.

Upvotes: 1

Related Questions