CODEWITHSUNDEEP
npmhttpsfiddler
mark
mark

Reputation: 62876

Debugging HTTP traffic originating from npm install using Fiddler

First, allow me to show that npm install works fine without Fiddler:

C:\Temp\1> npm config list
; "user" config from C:\Users\p11f70f\.npmrc

//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/:_password = (protected)
//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/:email = "npm requires email to be set but doesn't use the value"
//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/:username = "whatever"
//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/registry/:_password = (protected)
//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/registry/:email = "npm requires email to be set but doesn't use the value"
//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/registry/:username = "whatever"
registry = "https://registry.npmjs.org/"

; node bin location = C:\Program Files\nodejs\node.exe
; node version = v20.11.1
; npm local prefix = C:\Temp\1
; npm version = 10.2.4
; cwd = C:\Temp\1
; HOME = C:\Users\p11f70f
; Run `npm config ls -l` to show all defaults.
C:\Temp\1> npm cache clean --force
npm WARN using --force Recommended protections disabled.
C:\Temp\1> dir
C:\Temp\1> npm i --no-package-lock --no-save [email protected] && dir

added 1 package in 3s

    Directory: C:\Temp\1

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----          12/11/2024  2:41 PM                node_modules

C:\Temp\1>

So it works without any proxy. Now I am going to start Fiddler, configure HTTPS decryption and retry:

C:\Temp\1> npm config set proxy http://127.0.0.1:8888/
C:\Temp\1> npm config set https-proxy http://127.0.0.1:8888/
C:\Temp\1> npm config set strict-ssl false
C:\Temp\1> npm config list
; "user" config from C:\Users\p11f70f\.npmrc

//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/:_password = (protected)
//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/:email = "npm requires email to be set but doesn't use the value"
//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/:username = "whatever"
//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/registry/:_password = (protected)
//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/registry/:email = "npm requires email to be set but doesn't use the value"
//pkgs.dev.azure.com/xyz/_packaging/platform_catalog/npm/registry/:username = "whatever"
https-proxy = "http://127.0.0.1:8888/"
proxy = "http://127.0.0.1:8888/"
registry = "https://registry.npmjs.org/"
strict-ssl = false

; node bin location = C:\Program Files\nodejs\node.exe
; node version = v20.11.1
; npm local prefix = C:\Temp\1
; npm version = 10.2.4
; cwd = C:\Temp\1
; HOME = C:\Users\p11f70f
; Run `npm config ls -l` to show all defaults.
C:\Temp\1> npm cache clean --force
npm WARN using --force Recommended protections disabled.
C:\Temp\1> del -r -Force *
C:\Temp\1> npm i --no-package-lock --no-save [email protected] && dir
npm ERR! code E400
npm ERR! 400 Bad Request - GET https://registry.npmjs.org/simple-test-package

npm ERR! A complete log of this run can be found in: C:\Users\p11f70f\AppData\Local\npm-cache\_logs\2024-12-11T19_43_58_782Z-debug-0.log
C:\Temp\1>

Here is what Fiddler captured:

Request

GET http://registry.npmjs.org:443/simple-test-package HTTP/1.1
user-agent: npm/10.2.4 node/v20.11.1 win32 x64 workspaces/false
pacote-version: 17.0.4
pacote-req-type: packument
pacote-pkg-id: registry:simple-test-package
accept: application/vnd.npm.install-v1+json; q=1.0, application/json; q=0.8, */*
npm-auth-type: web
npm-command: install
Accept-Encoding: gzip,deflate
Host: registry.npmjs.org:443
Connection: Keep-Alive
connection: keep-alive

Response

HTTP/1.1 400 Bad Request
Server: cloudflare
Date: Wed, 11 Dec 2024 19:44:01 GMT
Content-Type: text/html
Content-Length: 253
Connection: close
CF-RAY: -

<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>cloudflare</center>
</body>
</html>

Notice the url is BS - http://registry.npmjs.org:443/simple-test-package. This is contrary to what npm claims in the error message - https://registry.npmjs.org/simple-test-package.

Now this difference in URL is all that matters:

C:\Temp\1> curl.exe -LsSfO https://registry.npmjs.org/simple-test-package
C:\Temp\1> dir .\simple-test-package

    Directory: C:\Temp\1

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          12/11/2024  2:51 PM           7374 simple-test-package

C:\Temp\1> del -r -Force .\simple-test-package
C:\Temp\1> curl.exe -LsSfO http://registry.npmjs.org:443/simple-test-package
curl: (22) The requested URL returned error: 400
C:\Temp\1>

So my question is - what is going on? Why am I unable to intercept HTTPS traffic generated by npm install with Fiddler? I suspect the problem is that npm install botches the URL, but why? How do I fix it?

Upvotes: 1

Views: 33

Answers (1)

mark
mark

Reputation: 62876

I managed to resolve it.

The problem with npm is that it looks at the proxy protocol schema and assumes it for the request url. So, even though the registry url is https://registry.npmjs.org/, because the https proxy url is http://127.0.0.1:8888 the resulting url used by npm would be http://registry.npmjs.org:443/, which is wrong.

This seems like a bug in the http library used by npm 10.2.4 at least.

I fixed it by adding the following code to the OnBeginRequest in the Fiddler Script (I use C# as the script language):

if (!oSession.HTTPMethodIs("CONNECT") && !oSession.isHTTPS && oSession.port == 443) {
    oSession["ui-backcolor"] = "lime";
    oSession["ui-bold"] = "changing URL for this session";
    oSession.fullUrl = "https://" + oSession.url;
}

Upvotes: 1

Related Questions