CODEWITHSUNDEEP
pythonarraysparsingrequesttwisted
element
element

Reputation: 305

How to properly parse request strings with Twisted (python)

I am trying to implement a simple twisted HTTP server which would respond to requests for loading tiles from a database and return them. However I find that the way it interprets request strings quite odd.

This is what I POST to the server:

curl -d "request=loadTiles&grid[0][x]=17&grid[0][y]=185&grid[1][x]=18&grid[1][y]=184" http://localhost:8080/fetch/

What I expect the request.args to be:

{'request': 'loadTiles', 'grid': [{'x': 17, 'y': 185}, {'x': 18, 'y': 184}]}

How Twisted interprets request.args:

{'grid[1][y]': ['184'], 'grid[0][y]': ['185'], 'grid[1][x]': ['18'], 'request': ['loadTiles'], 'grid[0][x]': ['17']}

Is it possible to have it automatically parse the request string and create a list for the grid parameter or do I have to do it manually?

I could json encode the grid parameter and then decode it server side, but it seems like an unneccssary hack.

Upvotes: 2

Views: 2378

Answers (2)

PaulMcG
PaulMcG

Reputation: 63729

Maybe instead of a parser, how about something to post-process the request.args you are getting?

from pyparsing import Suppress, alphas, alphanums, nums, Word
from itertools import groupby

# you could do this with regular expressions too, if you prefer
LBRACK,RBRACK = map(Suppress, '[]')
ident = Word('_' + alphas, '_' + alphanums)
integer = Word(nums).setParseAction(lambda t : int(t[0]))
subscriptedRef = ident + 2*(LBRACK + (ident | integer) + RBRACK)


def simplify_value(v):
    if isinstance(v,list) and len(v)==1:
        return simplify_value(v[0])
    if v == integer:
        return int(v)
    return v

def regroup_args(dd):
    ret = {}
    subscripts = []
    for k,v in dd.items():
        # this is a pyparsing short-cut to see if a string matches a pattern
        # I also used it above in simplify_value to test for integerness of a string
        if k == subscriptedRef:
            subscripts.append(tuple(subscriptedRef.parseString(k))+
                                    (simplify_value(v),))
        else:
            ret[k] = simplify_value(v)

    # sort all the matched subscripted args, and then use groupby to
    # group by name and list index
    # this assumes all indexes 0-n are present in the parsed arguments
    subscripts.sort()
    for name,nameitems in groupby(subscripts, key=lambda x:x[0]):
        ret[name] = []
        for idx,idxitems in groupby(nameitems, key=lambda x:x[1]):
            idd = {}
            for item in idxitems:
                name, i, attr, val = item
                idd[attr] = val
            ret[name].append(idd)

    return ret

request_args = {'grid[1][y]': ['184'], 'grid[0][y]': ['185'], 'grid[1][x]': ['18'], 'request': ['loadTiles'], 'grid[0][x]': ['17']} 
print regroup_args(request_args)

prints

{'grid': [{'y': 185, 'x': 17}, {'y': 184, 'x': 18}], 'request': 'loadTiles'}

Note that this also simplifies the single-element lists to just the 0'th element value, and converts the numeric strings to actual integers.

Upvotes: 1

Glyph
Glyph

Reputation: 31860

I don't know why you would expect your urlencoded data to be decoded according to some ad-hoc non-standard rules, or why you would consider the standard treatment "odd"; [ isn't special in query strings. What software decodes them this way?

In any event, this isn't really Twisted, but Python (and more generally speaking, the web-standard way of parsing this data). You can see the sort of data you'll get back via the cgi.parse_qs function interactively. For example:

>>> import cgi
>>> cgi.parse_qs("")
{}
>>> cgi.parse_qs("x=1")
{'x': ['1']}
>>> cgi.parse_qs("x[something]=1")
{'x[something]': ['1']}
>>> cgi.parse_qs("x=1&y=2")
{'y': ['2'], 'x': ['1']}
>>> cgi.parse_qs("x=1&y=2&x=3")
{'y': ['2'], 'x': ['1', '3']}

I hope that clears things up for you.

Upvotes: 4

Related Questions