CODEWITHSUNDEEP
pythonjson
dkgirl
dkgirl

Reputation: 4749

Python: Convert JSON (returned by URL) into List

I am requesting youtube search terms for use with jquery autocomplete, but am having a hard time converting the URL response into a proper format.

In my (Django/Python) view I do:

data2 = urllib2.urlopen('http://suggestqueries.google.com/complete/search?hl=en&ds=yt&client=youtube&hjson=t&jsonp=window.yt.www.suggest.handleResponse&q=jum&cp=3')

(I hardcoded the search term = 'jump' for simplicity)

If I do data2.read() I get what I believe is JSON (copy-pasting the url into a browser also returns this.)

window.yt.www.suggest.handleResponse(["jum",[["jumpstyle","","0"],["jump","","1"],["jump around","","2"],["jump on it","","3"],["jumper","","4"],["jump around house of pain","","5"],["jumper third eye blind","","6"],["jumbafund","","7"],["jump then fall taylor swift","","8"],["jumpstyle music","","9"]],"","","","","",{}])

I need to return this in a format that jquery autocomplete can read. I know it will work if I can get it into a list, for example, mylist = ['jumpstyle', 'jump', 'jump around', ...]

and then convert it back into json before returning it:

json.dumps(mylist)

(This works if I directly define mylist directly as written above.)

But I cannot get from the data that is returned by the URL to either a simple list (which I then convert back into JSON) or to some form of JSON that I can return directly to be used by auto complete.

I've tried, among other things,

j2 = json.loads(data2)

and 

j2 = json.loads(data2.read())

Hope someone can help!

Upvotes: 9

Views: 16871

Answers (4)

Xavier Barbosa
Xavier Barbosa

Reputation: 3947

remove the &jsonp=window.yt.www.suggest.handleResponse part

import json
import urllib2

data = urllib2.urlopen('http://suggestqueries.google.com/complete/search?hl=en&ds=yt&client=youtube&hjson=t&q=jum&cp=3')

j = json.load(data)
k = [i for i, j, k in j[1]]
l = json.dumps(k)

Upvotes: 14

Jon Nylander
Jon Nylander

Reputation: 8963

You are doing a JSON-P request which automatically wraps the JSON in a javascript callback function, the one you have specified in the request in fact :)

Strip away the JSON-P parameter from your request and you will get straight JSON directly from the request without having to do any extra python stuff at all.

This should be your request:

http://suggestqueries.google.com/complete/search?hl=en&ds=yt&client=youtube&hjson=t&q=jum&cp=3

and it will return:

["jum",[["jumpstyle","","0"],["jump","","1"],["jump around","","2"],["jump on it","","3"],["jumper","","4"],["jump around house of pain","","5"],["jumper third eye blind","","6"],["jumbafund","","7"],["jump then fall taylor swift","","8"],["jumpstyle music","","9"]],"","","","","",{}]

Upvotes: 3

virhilo
virhilo

Reputation: 6793

it's not json it's javascript, if you want to use it as json you must strip the javascript part:

j2 = json.loads(data2[37:-1])

but you can just change the url(remove the 'jsonp=window.yt.www.suggest.handleResponse' part) to have pure json output:

>>> data2 = urllib2.urlopen('http://suggestqueries.google.com/complete/search?hl=en&ds=yt&client=youtube&hjson=t&q=jum&cp=3')
>>> json.loads(data2.read())
[u'jum', [[u'jumpstyle', '', u'0'], [u'jump', '', u'1'], [u'jump around', '', u'2'], [u'jump on it', '', u'3'], [u'jumper', '', u'4'], [u'jump around house of pain', '', u'5'], [u'jumper third eye blind', '', u'6'], [u'jumbafund', '', u'7'], [u'jump then fall taylor swift', '', u'8'], [u'jumpstyle music', '', u'9']], '', '', '', '', '', {}]

Upvotes: 0

Mihai Toader
Mihai Toader

Reputation: 12253

The output from the page is not a proper json encoded data. You need to remove the js function call wrapping it.

do this:

import urllib2
import re
import json

data2 = urllib2.urlopen('http://suggestqueries.google.com/complete/search?' +    
   'hl=en&ds=yt&client=youtube&hjson=t&jsonp=window.yt.' + 
   'www.suggest.handleResponse&q=jum&cp=3')

data = re.compile('^[^\(]+\(|\)$').sub('', data2.read())
parsedData = json.loads(data)

parsedData is python array now.

Upvotes: 0

Related Questions