CODEWITHSUNDEEP
pythondjangopython-3.xurllib2django-1.7
Muhammad Ahmed
Muhammad Ahmed

Reputation: 428

Urllib2 and JSON Object Error

I am following a tutorial to add 'Search Function' in my project. However the tutorial is based on Python2x & Django 1.7 while I am using Python 3.4 & Django 1.7.

The search code uses bing_search along with urllib2. However urllib2 is not supported in Python 3 and the same has been distributed in several subsequent modules.

Some how I have managed to get my code compatible with Python 3. However now I am having the following error when I submit the search query. 

The JSON object must be str, not 'bytes'

Here is my code for your review:

import json
import urllib
import urllib.request
from urllib.parse import quote
import urllib.error

# Add your BING_API_KEY

BING_API_KEY = 'key removed for the sake of privacy'

def run_query(search_terms):
    # Specify the base
    root_url = 'https://api.datamarket.azure.com/Bing/Search/'
    source = 'Web'

    # Specify how many results we wish to be returned per page.
    # Offset specifies where in the results list to start from.
    # With results_per_page = 10 and offset = 11, this would start from page 2.
    results_per_page = 10
    offset = 0

    # Wrap quotes around our query terms as required by the Bing API.
    # The query we will then use is stored within variable query.
    query = "'{0}'".format(search_terms)
    query = urllib.parse.quote(query)

    # Construct the latter part of our request's URL.
    # Sets the format of the response to JSON and sets other properties.
    search_url = "{0}{1}?$format=json&$top={2}&$skip={3}&Query={4}".format(
        root_url,
        source,
        results_per_page,
        offset,
        query)

    # Setup authentication with the Bing servers.
    # The username MUST be a blank string, and put in your API key!
    username = ''


    # Create a 'password manager' which handles authentication for us.
    password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
    password_mgr.add_password(None, search_url, username, BING_API_KEY)

    # Create our results list which we'll populate.
    results = []

    try:
        # Prepare for connecting to Bing's servers.
        handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
        opener = urllib.request.build_opener(handler)
        urllib.request.install_opener(opener)

        # Connect to the server and read the response generated.
        response = urllib.request.urlopen(search_url).read()

        # Convert the string response to a Python dictionary object.
        json_response = json.loads(response)

        # Loop through each page returned, populating out results list.
        for result in json_response['d']['results']:
            results.append({
            'title': result['Title'],
            'link': result['Url'],
            'summary': result['Description']})

    # Catch a URLError exception - something went wrong when connecting!
    except urllib.error.URLError as e:
        print("Error when querying the Bing API: ", e)

    # Return the list of results to the calling function.
    return results

Please guide me what should I do now.

Thank you in advance....!

PS: I've seen some of similar posts but I could not workout the problem.

Upvotes: 0

Views: 787

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1121914

You'll need to decode the response before passing it to the json module:

response = urllib.request.urlopen(search_url)
codec = response.info().get_param('charset', 'utf8')  # default JSON encoding
json.loads(response.read().decode(codec))

The server is meant to provide you with a charset=... parameter in theContent-Type header, which the message.get_param() method above would retrieve for you. If the parameter is absent, the code falls back to UTF-8, the default encoding for JSON as per the RFC.

Upvotes: 1

Related Questions