CODEWITHSUNDEEP
curlnetwork-programmingpython-requests
Elegant_Cow
Elegant_Cow

Reputation: 73

Curl requests to an internal website

I am trying to use cURL (and python-requests) to retrieve information from a internal website. I have tried 

curl -v -u "username:password" https://internalwebsite.com/

but I keep getting the following error (similar error for python-requests too)

curl: (56) Received HTTP code 502 from proxy after CONNECT.

Here is an image of the entire error print out.imgur link

I can access the link using a browser and if I look at the request under the network tab of the developer options of the browser and copy the request as cURL. I get 

curl "https://internalwebsite.com/" -H "Host: myhosthere" -H "User-Agent: some user agent here" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "Connection: keep-alive" -H "Upgrade-Insecure-Requests: 1" -H "Authorization: Basic somebase64stringhere"

but this also results in the same error as before.Now I am aware an exact solution to this will be probably hard to find, I just want to collect more information before I ask the IT team about a tool they probably don't support.

Basically I want to know, how is a curl request different from a browser at the server end? The link (Rest API) works absolutely fine through a browser but not through curl, despite using the same user agent of my browser.

There is an "outgoing" proxy that I need to access the internet. Is there also such a thing as an "incoming proxy" that I may need to connect to in order to access the internal website via a curl request? The internal website is only accessible from within the corporate network and all my cURL requests were made from within the corporate network. Seems my browser is authenticated on the network but not cURL?

UPDATE:

For an exact solution, see the comments below the accepted answer.

Upvotes: 1

Views: 5692

Answers (1)

Daniel Stenberg
Daniel Stenberg

Reputation: 58002

curl finds an environment variable that tells it to use a proxy so it connects to that and issues a CONNECT there, which then gets 502 back...

If you want to disable the proxy for this command line do one of these:

  • use -x ""
  • use --noproxy to specify the domain(s) the proxy should not be used for
  • remove the environment variable (*_proxy something)
  • add this domain to the NO_PROXY environment variable

Upvotes: 4

Related Questions