Reputation: 11
I am trying to log all URLs that the user visited in a session on a remote device. I’m thinking of using Privoxy to capture such urls. The traffic from browser would be routed to Privoxy and then to internet. The issue is Privoxy logs are very verbose. It logs all the URLs from the main one also. To capture only, user specific domains, I’m thinking of using filters based on HTTP(S) request headers. But unfortunately Privoxy doesn’t look request headers. Is there is any way to do this in Privoxy?
Following are the logs for stackoverflow.com for reference.
2022-10-20 17:03:38.372 0016ff6f Header: scan: GET http://stackoverflow.com/ HTTP/1.1
2022-10-20 17:03:38.372 0016ff6f Header: scan: Host: stackoverflow.com
2022-10-20 17:03:38.373 0016ff6f Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:38.373 0016ff6f Header: scan: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
2022-10-20 17:03:38.373 0016ff6f Header: scan: Accept-Language: en-US,en;q=0.5
2022-10-20 17:03:38.373 0016ff6f Header: scan: Accept-Encoding: gzip, deflate
2022-10-20 17:03:38.373 0016ff6f Header: scan: Connection: keep-alive
2022-10-20 17:03:38.374 0016ff6f Header: scan: Upgrade-Insecure-Requests: 1
2022-10-20 17:03:38.374 0016ff6f Header: Keeping the client header 'Connection: keep-alive' around. The server connection will be kept alive if possible.
2022-10-20 17:03:38.374 0016ff6f Request: stackoverflow.com/
2022-10-20 17:03:38.375 0016ff6f Header: New HTTP Request-Line: GET / HTTP/1.1
2022-10-20 17:03:38.422 0016ff6f Header: scan: HTTP/1.1 301 Moved Permanently
2022-10-20 17:03:38.423 0016ff6f Header: scan: Connection: close
2022-10-20 17:03:38.423 0016ff6f Header: scan: Content-Length: 0
2022-10-20 17:03:38.423 0016ff6f Header: scan: Server: Varnish
2022-10-20 17:03:38.423 0016ff6f Header: scan: Retry-After: 0
2022-10-20 17:03:38.424 0016ff6f Header: scan: Location: https://stackoverflow.com/
2022-10-20 17:03:38.424 0016ff6f Header: scan: Accept-Ranges: bytes
2022-10-20 17:03:38.424 0016ff6f Header: scan: Date: Thu, 20 Oct 2022 11:33:38 GMT
2022-10-20 17:03:38.424 0016ff6f Header: scan: Via: 1.1 varnish
2022-10-20 17:03:38.424 0016ff6f Header: scan: X-Served-By: cache-bom4739-BOM
2022-10-20 17:03:38.424 0016ff6f Header: scan: X-Cache: HIT
2022-10-20 17:03:38.425 0016ff6f Header: scan: X-Cache-Hits: 0
2022-10-20 17:03:38.425 0016ff6f Header: scan: X-Timer: S1666265619.543907,VS0,VE0
2022-10-20 17:03:38.425 0016ff6f Header: scan: Strict-Transport-Security: max-age=31557600
2022-10-20 17:03:38.425 0016ff6f Header: scan: X-DNS-Prefetch-Control: off
2022-10-20 17:03:38.425 0016ff6f Header: Adding: Proxy-Connection: keep-alive
2022-10-20 17:03:38.429 0016ff6f Header: scan: CONNECT stackoverflow.com:443 HTTP/1.1
2022-10-20 17:03:38.430 0016ff6f Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:38.430 0016ff6f Header: scan: Proxy-Connection: keep-alive
2022-10-20 17:03:38.430 0016ff6f Header: scan: Connection: keep-alive
2022-10-20 17:03:38.430 0016ff6f Header: scan: Host: stackoverflow.com:443
2022-10-20 17:03:38.430 0016ff6f Header: crumble crunched: Proxy-Connection: keep-alive!
2022-10-20 17:03:38.431 0016ff6f Header: Replaced: 'Connection: keep-alive' with 'Connection: close'
2022-10-20 17:03:38.431 0016ff6f Request: stackoverflow.com:443/
2022-10-20 17:03:38.431 0016ff6f Header: New HTTP Request-Line: CONNECT / HTTP/1.1
2022-10-20 17:03:38.898 0016fffb Header: scan: CONNECT ajax.googleapis.com:443 HTTP/1.1
2022-10-20 17:03:38.899 0016fffb Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:38.899 00170087 Header: scan: CONNECT cdn.sstatic.net:443 HTTP/1.1
2022-10-20 17:03:38.899 0016fffb Header: scan: Proxy-Connection: keep-alive
2022-10-20 17:03:38.899 00170087 Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:38.899 00170113 Header: scan: CONNECT cdn.sstatic.net:443 HTTP/1.1
2022-10-20 17:03:38.899 0017019f Header: scan: CONNECT cdn.sstatic.net:443 HTTP/1.1
2022-10-20 17:03:38.899 0016fffb Header: scan: Connection: keep-alive
2022-10-20 17:03:38.899 0017022b Header: scan: CONNECT cdn.sstatic.net:443 HTTP/1.1
2022-10-20 17:03:38.900 001702b7 Header: scan: CONNECT cdn.sstatic.net:443 HTTP/1.1
2022-10-20 17:03:38.900 00170087 Header: scan: Proxy-Connection: keep-alive
2022-10-20 17:03:38.901 0017022b Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:38.900 00170113 Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:38.900 00170343 Header: scan: CONNECT cdn.sstatic.net:443 HTTP/1.1
2022-10-20 17:03:38.900 0017019f Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:38.900 0016fffb Header: scan: Host: ajax.googleapis.com:443
2022-10-20 17:03:38.901 001702b7 Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:38.901 00170087 Header: scan: Connection: keep-alive
2022-10-20 17:03:38.902 0016fffb Header: crumble crunched: Proxy-Connection: keep-alive!
2022-10-20 17:03:38.901 0017022b Header: scan: Proxy-Connection: keep-alive
2022-10-20 17:03:38.901 00170113 Header: scan: Proxy-Connection: keep-alive
2022-10-20 17:03:38.901 00170343 Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:38.902 0017019f Header: scan: Proxy-Connection: keep-alive
2022-10-20 17:03:38.902 00170087 Header: scan: Host: cdn.sstatic.net:443
2022-10-20 17:03:38.903 0017019f Header: scan: Connection: keep-alive
2022-10-20 17:03:38.902 001702b7 Header: scan: Proxy-Connection: keep-alive
2022-10-20 17:03:38.903 00170087 Header: crumble crunched: Proxy-Connection: keep-alive!
2022-10-20 17:03:38.903 0016fffb Header: Replaced: 'Connection: keep-alive' with 'Connection: close'
2022-10-20 17:03:38.903 0017022b Header: scan: Connection: keep-alive
2022-10-20 17:03:38.903 00170113 Header: scan: Connection: keep-alive
2022-10-20 17:03:38.903 00170343 Header: scan: Proxy-Connection: keep-alive
2022-10-20 17:03:38.904 001702b7 Header: scan: Connection: keep-alive
2022-10-20 17:03:38.904 0017019f Header: scan: Host: cdn.sstatic.net:443
2022-10-20 17:03:38.905 001702b7 Header: scan: Host: cdn.sstatic.net:443
2022-10-20 17:03:38.904 00170087 Header: Replaced: 'Connection: keep-alive' with 'Connection: close'
2022-10-20 17:03:38.905 0016fffb Request: ajax.googleapis.com:443/
2022-10-20 17:03:38.905 00170113 Header: scan: Host: cdn.sstatic.net:443
2022-10-20 17:03:38.905 0017022b Header: scan: Host: cdn.sstatic.net:443
2022-10-20 17:03:38.905 00170343 Header: scan: Connection: keep-alive
2022-10-20 17:03:38.906 0017019f Header: crumble crunched: Proxy-Connection: keep-alive!
2022-10-20 17:03:38.907 00170343 Header: scan: Host: cdn.sstatic.net:443
2022-10-20 17:03:38.907 0017019f Header: Replaced: 'Connection: keep-alive' with 'Connection: close'
2022-10-20 17:03:38.906 001702b7 Header: crumble crunched: Proxy-Connection: keep-alive!
2022-10-20 17:03:38.906 00170087 Request: cdn.sstatic.net:443/
2022-10-20 17:03:38.907 0016fffb Header: New HTTP Request-Line: CONNECT / HTTP/1.1
2022-10-20 17:03:38.907 00170113 Header: crumble crunched: Proxy-Connection: keep-alive!
2022-10-20 17:03:38.907 0017022b Header: crumble crunched: Proxy-Connection: keep-alive!
2022-10-20 17:03:38.908 00170343 Header: crumble crunched: Proxy-Connection: keep-alive!
2022-10-20 17:03:38.909 0017022b Header: Replaced: 'Connection: keep-alive' with 'Connection: close'
2022-10-20 17:03:38.909 00170343 Header: Replaced: 'Connection: keep-alive' with 'Connection: close'
2022-10-20 17:03:38.908 0017019f Request: cdn.sstatic.net:443/
2022-10-20 17:03:38.909 00170113 Header: Replaced: 'Connection: keep-alive' with 'Connection: close'
2022-10-20 17:03:38.908 001702b7 Header: Replaced: 'Connection: keep-alive' with 'Connection: close'
2022-10-20 17:03:38.908 00170087 Header: New HTTP Request-Line: CONNECT / HTTP/1.1
2022-10-20 17:03:38.910 0017022b Request: cdn.sstatic.net:443/
2022-10-20 17:03:38.912 001702b7 Request: cdn.sstatic.net:443/
2022-10-20 17:03:38.911 00170343 Request: cdn.sstatic.net:443/
2022-10-20 17:03:38.911 0017019f Header: New HTTP Request-Line: CONNECT / HTTP/1.1
2022-10-20 17:03:38.911 00170113 Request: cdn.sstatic.net:443/
2022-10-20 17:03:38.913 001702b7 Header: New HTTP Request-Line: CONNECT / HTTP/1.1
2022-10-20 17:03:38.912 0017022b Header: New HTTP Request-Line: CONNECT / HTTP/1.1
2022-10-20 17:03:38.914 00170113 Header: New HTTP Request-Line: CONNECT / HTTP/1.1
2022-10-20 17:03:38.914 00170343 Header: New HTTP Request-Line: CONNECT / HTTP/1.1
2022-10-20 17:03:38.927 001703cf Header: scan: CONNECT safebrowsing.googleapis.com:443 HTTP/1.1
2022-10-20 17:03:38.927 001703cf Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:38.928 001703cf Header: scan: Proxy-Connection: keep-alive
2022-10-20 17:03:38.928 001703cf Header: scan: Connection: keep-alive
2022-10-20 17:03:38.928 001703cf Header: scan: Host: safebrowsing.googleapis.com:443
2022-10-20 17:03:38.929 001703cf Header: crumble crunched: Proxy-Connection: keep-alive!
2022-10-20 17:03:38.929 001703cf Header: Replaced: 'Connection: keep-alive' with 'Connection: close'
2022-10-20 17:03:38.929 001703cf Request: safebrowsing.googleapis.com:443/
2022-10-20 17:03:38.929 001703cf Header: New HTTP Request-Line: CONNECT / HTTP/1.1
2022-10-20 17:03:39.029 00170087 Header: scan: POST http://ocsp.pki.goog/gts1c3 HTTP/1.1
2022-10-20 17:03:39.030 00170087 Header: scan: Host: ocsp.pki.goog
2022-10-20 17:03:39.030 00170087 Header: scan: User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
2022-10-20 17:03:39.030 00170087 Header: scan: Accept: */*
2022-10-20 17:03:39.030 00170087 Header: scan: Accept-Language: en-US,en;q=0.5
2022-10-20 17:03:39.030 00170087 Header: scan: Accept-Encoding: gzip, deflate
2022-10-20 17:03:39.030 00170087 Header: scan: Content-Type: application/ocsp-request
2022-10-20 17:03:39.031 00170087 Header: scan: Content-Length: 84
2022-10-20 17:03:39.031 00170087 Header: scan: Connection: keep-alive
2022-10-20 17:03:39.031 00170087 Header: scan: Pragma: no-cache
2022-10-20 17:03:39.031 00170087 Header: scan: Cache-Control: no-cache
2022-10-20 17:03:39.031 00170087 Header: Keeping the client header 'Connection: keep-alive' around. The server connection will be kept alive if possible.
2022-10-20 17:03:39.032 00170087 Request: ocsp.pki.goog/gts1c3
2022-10-20 17:03:39.032 00170087 Header: New HTTP Request-Line: POST /gts1c3 HTTP/1.1
2022-10-20 17:03:39.147 00170087 Header: scan: HTTP/1.1 200 OK
Upvotes: 1
Views: 474