CODEWITHSUNDEEP
pythonpython-requests
jpmelos
jpmelos

Reputation: 3562

Library `requests` getting different results unpredictably

Why does this code:

import requests


response = requests.post('http://evds.tcmb.gov.tr/cgi-bin/famecgi', data={
    'cgi': '$ozetweb',
    'ARAVERIGRUP': 'bie_yymkpyuk.db',
    'DIL': 'UK',
    'ONDALIK': '5',
    'wfmultiple_selection': 'ZAMANSERILERI',
    'f_begdt': '07-01-2005',
    'f_enddt': '07-10-2016',
    'ZAMANSERILERI': ['TP.PYUK1', 'TP.PYUK2', 'TP.PYUK21', 'TP.PYUK22', 'TP.PYUK3', 'TP.PYUK4', 'TP.PYUK5', 'TP.PYUK6'],
    'YON': '3',
    'SUBMITDEG': 'Report',
    'GRTYPE': '1',
    'EPOSTA': 'xxx',
    'RESIMPOSTA': '***',
})

print(response.text)

produces different results in Python 2 (2.7.12) and Python 3 (3.5.2)? I'm using requests==2.11.1. Since the requests library supports both Python versions with the same API, I guess the result should be the same.

The expected result is the one obtained from running the code with Python 2. It works every single time. When ran with Python 3, the server sometimes returns an error, and sometimes it works. (This is the intriguing part.)

Since it works with Python 2, I figure the error must happen in the client side. Is there any caveat to how Python 3 handles encoding, or sending the data through the socket, that I should be aware of?

EDIT: In the comments below, a person was able to reproduce this and confirms this issue exists.

Upvotes: 3

Views: 167

Answers (1)

Padraic Cunningham
Padraic Cunningham

Reputation: 180391

It does seem to come down to different between dicts in python2 vs python3 in relation to Hash randomization is enabled by default since python3.3 and the server needing at least the cgi field to come first, the following can reproduce:

good = requests.post('http://evds.tcmb.gov.tr/cgi-bin/famecgi', data=([
    ('cgi', '$ozetweb'),
    ('ARAVERIGRUP', 'bie_yymkpyuk.db'),
    ('DIL', 'UK'),
    ('ONDALIK', '5'),
    ('wfmultiple_selection', 'ZAMANSERILERI'),
    ('f_begdt', '07-01-2005'),
    ('f_enddt', '07-10-2016'),
    ('ZAMANSERILERI',
     ['TP.PYUK1', 'TP.PYUK2', 'TP.PYUK21', 'TP.PYUK22', 'TP.PYUK3', 'TP.PYUK4', 'TP.PYUK5', 'TP.PYUK6']),
    ('YON', '3'),
    ('SUBMITDEG', 'Report'),
    ('GRTYPE', '1'),
    ('EPOSTA', 'xxx'),
    ('RESIMPOSTA', '***')]))


bad = requests.post('http://evds.tcmb.gov.tr/cgi-bin/famecgi', data=([
    ('ARAVERIGRUP', 'bie_yymkpyuk.db'),
    ('cgi', '$ozetweb'),
    ('DIL', 'UK'),
    ('wfmultiple_selection', 'ZAMANSERILERI'),
    ('ONDALIK', '5'),
    ('f_begdt', '07-01-2005'),
    ('f_enddt', '07-10-2016'),
    ('ZAMANSERILERI',
     ['TP.PYUK1', 'TP.PYUK2', 'TP.PYUK21', 'TP.PYUK22', 'TP.PYUK3', 'TP.PYUK4', 'TP.PYUK5', 'TP.PYUK6']),
    ('YON', '3'),
    ('SUBMITDEG', 'Report'),
    ('GRTYPE', '1'),
    ('EPOSTA', 'xxx'),
    ('RESIMPOSTA', '***')]))

Running the code above using python2:

In [6]: print(good.request.body)
   ...: print(bad.request.body)
   ...: 
   ...: print(len(good.text), len(bad.text))
   ...: 
cgi=%24ozetweb&ARAVERIGRUP=bie_yymkpyuk.db&DIL=UK&ONDALIK=5&wfmultiple_selection=ZAMANSERILERI&f_begdt=07-01-2005&f_enddt=07-10-2016&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&YON=3&SUBMITDEG=Report&GRTYPE=1&EPOSTA=xxx&RESIMPOSTA=%2A%2A%2A
ARAVERIGRUP=bie_yymkpyuk.db&cgi=%24ozetweb&DIL=UK&wfmultiple_selection=ZAMANSERILERI&ONDALIK=5&f_begdt=07-01-2005&f_enddt=07-10-2016&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&YON=3&SUBMITDEG=Report&GRTYPE=1&EPOSTA=xxx&RESIMPOSTA=%2A%2A%2A
(71299, 134)

And python3:

In [4]: print(good.request.body)
   ...: print(bad.request.body)
   ...: 
   ...: print(len(good.text), len(bad.text))
   ...: 
cgi=%24ozetweb&ARAVERIGRUP=bie_yymkpyuk.db&DIL=UK&ONDALIK=5&wfmultiple_selection=ZAMANSERILERI&f_begdt=07-01-2005&f_enddt=07-10-2016&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&YON=3&SUBMITDEG=Report&GRTYPE=1&EPOSTA=xxx&RESIMPOSTA=%2A%2A%2A
ARAVERIGRUP=bie_yymkpyuk.db&cgi=%24ozetweb&DIL=UK&wfmultiple_selection=ZAMANSERILERI&ONDALIK=5&f_begdt=07-01-2005&f_enddt=07-10-2016&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&YON=3&SUBMITDEG=Report&GRTYPE=1&EPOSTA=xxx&RESIMPOSTA=%2A%2A%2A
71299 134

Passing your dict as posted in python2:

In [4]: response.request.body
Out[4]: 'cgi=%24ozetweb&DIL=UK&f_enddt=07-10-2016&YON=3&RESIMPOSTA=%2A%2A%2A&wfmultiple_selection=ZAMANSERILERI&ARAVERIGRUP=bie_yymkpyuk.db&GRTYPE=1&SUBMITDEG=Report&f_begdt=07-01-2005&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&ONDALIK=5&EPOSTA=xxx'

In [5]: len(response.text)
Out[5]: 71299

And the same dict in python3:

In [3]: response.request.body
Out[3]: 'EPOSTA=xxx&ARAVERIGRUP=bie_yymkpyuk.db&DIL=UK&SUBMITDEG=Report&cgi=%24ozetweb&GRTYPE=1&f_enddt=07-10-2016&wfmultiple_selection=ZAMANSERILERI&ONDALIK=5&f_begdt=07-01-2005&RESIMPOSTA=%2A%2A%2A&YON=3&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6'

In [4]: len(response.text)
Out[4]: 134

And running ~$ export PYTHONHASHSEED=1234 before starting another ipython2 shell:

In [4]: response.request.body
Out[4]: 'DIL=UK&GRTYPE=1&ARAVERIGRUP=bie_yymkpyuk.db&f_begdt=07-01-2005&RESIMPOSTA=%2A%2A%2A&ONDALIK=5&EPOSTA=xxx&YON=3&SUBMITDEG=Report&wfmultiple_selection=ZAMANSERILERI&cgi=%24ozetweb&ZAMANSERILERI=TP.PYUK1&ZAMANSERILERI=TP.PYUK2&ZAMANSERILERI=TP.PYUK21&ZAMANSERILERI=TP.PYUK22&ZAMANSERILERI=TP.PYUK3&ZAMANSERILERI=TP.PYUK4&ZAMANSERILERI=TP.PYUK5&ZAMANSERILERI=TP.PYUK6&f_enddt=07-10-2016'

In [5]: os.environ["PYTHONHASHSEED"]
Out[5]: '1234'
In [6]: len(response.text)
Out[6]: 134

You can run the code numerous times to the same end but definitely ('cgi', '$ozetweb') coming first is essential for the code to work, it happened to work using python3 intermittently as the order of the keys sometimes put cgi first. There is a bit more on the hashing topic

Upvotes: 1

Related Questions