CODEWITHSUNDEEP
pythoncurlpython-requests
user11322373
user11322373

Reputation:

convert curl to requests gives 400

curl 'https://2fwotdvm2o-2.algolianet.com/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%20(lite)%203.25.1%3Breact%20(16.9.0)%3Breact-instantsearch%20(6.2.0)%3BJS%20Helper%20(3.1.0)&x-algolia-application-id=2FWOTDVM2O&x-algolia-api-key=ac96de6fef0e02bb95d433d8d5c7038a' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0' -H 'Accept: application/json' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.goat.com' -H 'Referer: https://www.goat.com/sneakers/brand/air%20jordan' -H 'Connection: keep-alive' --data '{"requests":[{"indexName":"product_variants_v2","params":"highlightPreTag=%3Cais-highlight-0000000000%3E&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&distinct=true&maxValuesPerFacet=30&page=1&query=&facets=%5B%22instant_ship_lowest_price_cents%22%2C%22single_gender%22%2C%22presentation_size%22%2C%22shoe_condition%22%2C%22product_category%22%2C%22brand_name%22%2C%22color%22%2C%22silhouette%22%2C%22designer%22%2C%22upper_material%22%2C%22midsole%22%2C%22category%22%5D&tagFilters=&facetFilters=%5B%5B%22product_category%3Ashoes%22%5D%2C%5B%22brand_name%3Aair%20jordan%22%5D%5D"},{"indexName":"product_variants_v2","params":"highlightPreTag=%3Cais-highlight-0000000000%3E&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&distinct=true&maxValuesPerFacet=30&page=0&query=&hitsPerPage=1&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=product_category&facetFilters=%5B%5B%22brand_name%3Aair%20jordan%22%5D%5D"},{"indexName":"product_variants_v2","params":"highlightPreTag=%3Cais-highlight-0000000000%3E&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&distinct=true&maxValuesPerFacet=30&page=0&query=&hitsPerPage=1&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=brand_name&facetFilters=%5B%5B%22product_category%3Ashoes%22%5D%5D"}]}'

I used curl trillworks but I am getting 400 response, when I used curl it works fine

>>> response.text
'{"message":"lexical error: invalid char in json text. Around \'%7B%22requ\' near line:1 column:1","status":400}'

How can I fix this?

Upvotes: 1

Views: 222

Answers (2)

Kelvin
Kelvin

Reputation: 296

So the problem above is due request content type is 

Content-Type: application/x-www-form-urlencoded

but data that server expect is JSON,

without 

json.dumps(payload)

we will receive 

{"message":"lexical error: invalid char in json text. Around 'requests=p' near line:1 column:1","status":400}

you can do it using requests like this.

import json
import requests

url = "https://2fwotdvm2o-2.algolianet.com/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%20(lite)%203.25.1%3Breact%20(16.9.0)%3Breact-instantsearch%20(6.2.0)%3BJS%20Helper%20(3.1.0)&x-algolia-application-id=2FWOTDVM2O&x-algolia-api-key=ac96de6fef0e02bb95d433d8d5c7038a"

headers = {
    "Content-Type": "application/x-www-form-urlencoded",
    "Accept": "application/json",
    "Accept-Language": "en-US,en;q=0.5",
    "Origin": "https://www.goat.com",
    "Connection": "keep-alive",
    "Referer": "Referer: https://www.goat.com/sneakers/brand/air%20jordan",
}

payload = {
    "requests": [
        {
            "indexName": "product_variants_v2",
            "params": "highlightPreTag=%3Cais-highlight-0000000000%3E&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&distinct=true&maxValuesPerFacet=30&page=1&query=&facets=%5B%22instant_ship_lowest_price_cents%22%2C%22single_gender%22%2C%22presentation_size%22%2C%22shoe_condition%22%2C%22product_category%22%2C%22brand_name%22%2C%22color%22%2C%22silhouette%22%2C%22designer%22%2C%22upper_material%22%2C%22midsole%22%2C%22category%22%5D&tagFilters=&facetFilters=%5B%5B%22product_category%3Ashoes%22%5D%2C%5B%22brand_name%3Aair%20jordan%22%5D%5D",
        },
        {
            "indexName": "product_variants_v2",
            "params": "highlightPreTag=%3Cais-highlight-0000000000%3E&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&distinct=true&maxValuesPerFacet=30&page=0&query=&hitsPerPage=1&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=product_category&facetFilters=%5B%5B%22brand_name%3Aair%20jordan%22%5D%5D",
        },
        {
            "indexName": "product_variants_v2",
            "params": "highlightPreTag=%3Cais-highlight-0000000000%3E&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&distinct=true&maxValuesPerFacet=30&page=0&query=&hitsPerPage=1&attributesToRetrieve=%5B%5D&attributesToHighlight=%5B%5D&attributesToSnippet=%5B%5D&tagFilters=&analytics=false&clickAnalytics=false&facets=brand_name&facetFilters=%5B%5B%22product_category%3Ashoes%22%5D%5D",
        },
    ]
}

response = requests.post(url=url, headers=headers, data=json.dumps(payload))
print(response.text)

Upvotes: 1

AzyCrw4282
AzyCrw4282

Reputation: 7744

curl twillworls is not 100% accurate. It's even stated in the converted text box

NB. Original query string below. It seems impossible to parse and reproduce query strings 100% accurately

Consider using pycurl. It's more accurate and easy to use.

An example of usage in pycurl

import pycurl
from io import BytesIO 

b_obj = BytesIO() 
crl = pycurl.Curl() 

# Set URL value
crl.setopt(crl.URL, 'https://wiki.python.org/moin/BeginnersGuide')

# Write bytes that are utf-8 encoded
crl.setopt(crl.WRITEDATA, b_obj)

# Perform a file transfer 
crl.perform() 

# End curl session
crl.close()

# Get the content stored in the BytesIO object (in byte characters) 
get_body = b_obj.getvalue()

# Decode the bytes stored in get_body to HTML and print the result 
print('Output of GET request:\n%s' % get_body.decode('utf8'))

Read more here

Upvotes: 0

Related Questions