How to urlencode a querystring in Python?

Question

I am trying to urlencode this string before I submit.

queryString = 'eventName=' + evt.fields["eventName"] + '&' + 'eventDescription=' + evt.fields["eventDescription"]; 

Answer 1

Python 3

Use urllib.parse.urlencode:

>>> import urllib.parse
>>> f = { 'eventName' : 'myEvent', 'eventDescription' : 'cool event'}
>>> urllib.parse.urlencode(f)
eventName=myEvent&eventDescription=cool+event

Note that this does not do url encoding in the commonly used sense (look at the output). For that use urllib.parse.quote_plus.

Python 2

You need to pass your parameters into urllib.urlencode() as either a mapping (dict), or a sequence of 2-tuples, like:

>>> import urllib
>>> f = { 'eventName' : 'myEvent', 'eventDescription' : 'cool event'}
>>> urllib.urlencode(f)
'eventName=myEvent&eventDescription=cool+event'

Answer 2

Python 3

In Python 3, the urllib package has been broken into smaller components. You'll use urllib.parse.quote_plus (note the parse child module)

import urllib.parse
safe_string = urllib.parse.quote_plus(...)

Python 2

What you're looking for is urllib.quote_plus:

safe_string = urllib.quote_plus('string_of_characters_like_these:$#@=?%^Q^$')

#Value: 'string_of_characters_like_these%3A%24%23%40%3D%3F%25%5EQ%5E%24'

Answer 3

urllib parse wasn't quite sufficient in my case.

Two additions were needed:

1. use safe kwarg to unmark / as a safe character (urllib.parse.quote(string, safe=''))

2. use .replace() for remaining special characters - in particular, note that javascript's encodeURIComponent and C# Uri.EscapeDataString behave differently. For working with a C# backend that expects EscapeDataStrings, then the following all need to be additionally replaced: _-.~

Ultimately I have something like this

import urllib

def aggressive_urlencode(inp: str) -> str:
    return (
        urllib.parse.quote(inp, safe='')
        .replace('-', '%2D')
        .replace('.', '%2E')
        .replace('_', '%5F')
        .replace('~', '%7E')
    )

Answer 4

import urllib.parse
query = 'Hellö Wörld@Python'
urllib.parse.quote(query) # returns Hell%C3%B6%20W%C3%B6rld%40Python

Answer 5

If you don't want to use urllib.

https://github.com/wayne931121/Python_URL_Decode

URL_RFC_3986 = {
"!": "%21", "#": "%23", "$": "%24", "&": "%26", "'": "%27", "(": "%28", ")": "%29", "*": "%2A", "+": "%2B", 
",": "%2C", "/": "%2F", ":": "%3A", ";": "%3B", "=": "%3D", "?": "%3F", "@": "%40", "[": "%5B", "]": "%5D",
}

def url_encoder(b):
    # https://zh.wikipedia.org/wiki/%E7%99%BE%E5%88%86%E5%8F%B7%E7%BC%96%E7%A0%81
    if type(b)==bytes:
        b = b.decode(encoding="utf-8") #byte can't insert many utf8 charaters
    result = bytearray() #bytearray: rw, bytes: read-only
    for i in b:
        if i in URL_RFC_3986:
            for j in URL_RFC_3986[i]:
                result.append(ord(j))
            continue
        i = bytes(i, encoding="utf-8")
        if len(i)==1:
            result.append(ord(i))
        else:
            for c in i:
                c = hex(c)[2:].upper()
                result.append(ord("%"))
                result.append(ord(c[0:1]))
                result.append(ord(c[1:2]))
    result = result.decode(encoding="ascii")
    return result

#print(url_encoder("我好棒==%%0.0:)")) ==> '%E6%88%91%E5%A5%BD%E6%A3%92%3D%3D%%0.0%3A%29'

Answer 6

Context

• Python (version 2.7.2 )

Problem

• You want to generate a urlencoded query string.
• You have a dictionary or object containing the name-value pairs.
• You want to be able to control the output ordering of the name-value pairs.

Solution

• urllib.urlencode
• urllib.quote_plus

Pitfalls

• dictionary output arbitrary ordering of name-value pairs
  • (see also: Why is python ordering my dictionary like so?)
  • (see also: Why is the order in dictionaries and sets arbitrary?)
• handling cases when you DO NOT care about the ordering of the name-value pairs
• handling cases when you DO care about the ordering of the name-value pairs
• handling cases where a single name needs to appear more than once in the set of all name-value pairs

Example

The following is a complete solution, including how to deal with some pitfalls.

### ********************
## init python (version 2.7.2 )
import urllib

### ********************
## first setup a dictionary of name-value pairs
dict_name_value_pairs = {
  "bravo"   : "True != False",
  "alpha"   : "http://www.example.com",
  "charlie" : "hello world",
  "delta"   : "1234567 !@#$%^&*",
  "echo"    : "user@example.com",
  }

### ********************
## setup an exact ordering for the name-value pairs
ary_ordered_names = []
ary_ordered_names.append('alpha')
ary_ordered_names.append('bravo')
ary_ordered_names.append('charlie')
ary_ordered_names.append('delta')
ary_ordered_names.append('echo')

### ********************
## show the output results
if('NO we DO NOT care about the ordering of name-value pairs'):
  queryString  = urllib.urlencode(dict_name_value_pairs)
  print queryString 
  """
  echo=user%40example.com&bravo=True+%21%3D+False&delta=1234567+%21%40%23%24%25%5E%26%2A&charlie=hello+world&alpha=http%3A%2F%2Fwww.example.com
  """

if('YES we DO care about the ordering of name-value pairs'):
  queryString  = "&".join( [ item+'='+urllib.quote_plus(dict_name_value_pairs[item]) for item in ary_ordered_names ] )
  print queryString
  """
  alpha=http%3A%2F%2Fwww.example.com&bravo=True+%21%3D+False&charlie=hello+world&delta=1234567+%21%40%23%24%25%5E%26%2A&echo=user%40example.com
  """ 

Answer 7

Python 3:

urllib.parse.quote_plus(string, safe='', encoding=None, errors=None)

Answer 8

Try requests instead of urllib and you don't need to bother with urlencode!

import requests
requests.get('http://youraddress.com', params=evt.fields)

EDIT:

If you need ordered name-value pairs or multiple values for a name then set params like so:

params=[('name1','value11'), ('name1','value12'), ('name2','value21'), ...]

instead of using a dictionary.

Answer 9

For Python 3 urllib3 works properly, you can use as follow as per its official docs :

import urllib3

http = urllib3.PoolManager()
response = http.request(
     'GET',
     'https://api.prylabs.net/eth/v1alpha1/beacon/attestations',
     fields={  # here fields are the query params
          'epoch': 1234,
          'pageSize': pageSize 
      } 
 )
response = attestations.data.decode('UTF-8')

Answer 10

For use in scripts/programs which need to support both python 2 and 3, the six module provides quote and urlencode functions:

>>> from six.moves.urllib.parse import urlencode, quote
>>> data = {'some': 'query', 'for': 'encoding'}
>>> urlencode(data)
'some=query&for=encoding'
>>> url = '/some/url/with spaces and %;!<>&'
>>> quote(url)
'/some/url/with%20spaces%20and%20%25%3B%21%3C%3E%26'

Answer 11

Another thing that might not have been mentioned already is that urllib.urlencode() will encode empty values in the dictionary as the string None instead of having that parameter as absent. I don't know if this is typically desired or not, but does not fit my use case, hence I have to use quote_plus.

Answer 12

If the urllib.parse.urlencode( ) is giving you errors , then Try the urllib3 module .

The syntax is as follows :

import urllib3
urllib3.request.urlencode({"user" : "john" }) 

Answer 13

Try this:

urllib.pathname2url(stringToURLEncode)

urlencode won't work because it only works on dictionaries. quote_plus didn't produce the correct output.

Answer 14

In Python 3, this worked with me

import urllib

urllib.parse.quote(query)

Answer 15

for future references (ex: for python3)

>>> import urllib.request as req
>>> query = 'eventName=theEvent&eventDescription=testDesc'
>>> req.pathname2url(query)
>>> 'eventName%3DtheEvent%26eventDescription%3DtestDesc'

Answer 16

Note that the urllib.urlencode does not always do the trick. The problem is that some services care about the order of arguments, which gets lost when you create the dictionary. For such cases, urllib.quote_plus is better, as Ricky suggested.