Web request works in browser, but it gives 'HTTP 403' in Java

Question

I am running a Java program to download data from a Stock API. The URL is

https://tvc4.investing.com/49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918

If I run this URL from the browser, I get a JSON as a response. However, if I run it from a Java program I get a 403 forbidden. I looked up the Developer tools in Chrome and set up all the HTTP request headers in the Java program. I added User-Agent/Cookie, etc. However, I still get the Forbidden 403 error.

Below are the HTTP Headers that I have set:

Below is the stack trace that I get in my Java program.

java.io.IOException: Server returned HTTP response code: 403 for URL: https://tvc4.investing.com/49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1997)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
    at java.base/java.net.URLConnection.getContent(URLConnection.java:753)
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getContent(HttpsURLConnectionImpl.java:404)
    at ai.jeet.test.DataDownload.main(DataDownload.java:36)

When I run this API call via Postman I get a forbidden 403, but I also get an HTML response which says enable cookies and JavaScript.

I fail to understand how a server can differentiate between a web request call. Obviously, I am doing something wrong, but what is it?

Answer 1

The main difference from what I could see is the User Agent header value. The server might be checking for a few specific header values. I have one value working for me as shown below:

Request with 403

curl -v 'https://tvc4.investing.com/49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918' -H 'User-Agent: Mozilla' -o /dev/null

Response

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2606:4700::6812:9a:443...
* Connected to tvc4.investing.com (2606:4700::6812:9a) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
} [323 bytes data]
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2326 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=investing.com
*  start date: Aug  1 00:00:00 2022 GMT
*  expire date: Aug  1 23:59:59 2023 GMT
*  subjectAltName: host "tvc4.investing.com" matched cert's "*.investing.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x137012600)
> GET /49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918 HTTP/2
> Host: tvc4.investing.com
> accept: */*
> user-agent: Mozilla
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 403
< date: Wed, 23 Nov 2022 14:22:29 GMT
< content-type: text/html; charset=UTF-8
< cache-control: max-age=15
< expires: Wed, 23 Nov 2022 14:22:44 GMT
< x-frame-options: SAMEORIGIN
< set-cookie: __cf_bm=uwoeNXvcnmEDY7ACRnFQEDqGZN4Yfx2_cyXbho.D6.M-1669213349-0-AUqobigV2idaMayGrBR+OdyTBo8pbjfS77vjhSh6bA4wiaBuz79/5kbwvXwD2loYoHJt1BsTguMEYh7WRm2ikPo=; path=/; expires=Wed, 23-Nov-22 14:52:29 GMT; domain=.investing.com; HttpOnly; Secure; SameSite=None
< server: cloudflare
< cf-ray: 76ea8eab8cbbb06a-ATL
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
{ [972 bytes data]
100  4537    0  4537    0     0  23760      0 --:--:-- --:--:-- --:--:-- 25346
* Connection #0 to host tvc4.investing.com left intact

Request with 200 Response

curl -v 'https://tvc4.investing.com/49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918' -H 'User-Agent: Mozilla/5.0' -o /dev/null

Response

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2606:4700::6812:19a:443...
* Connected to tvc4.investing.com (2606:4700::6812:19a) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
} [323 bytes data]
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2326 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=investing.com
*  start date: Aug  1 00:00:00 2022 GMT
*  expire date: Aug  1 23:59:59 2023 GMT
*  subjectAltName: host "tvc4.investing.com" matched cert's "*.investing.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x13800b600)
> GET /49137b20ec52f5d1133789e270e21db8/1668879605/56/56/23/history?symbol=18325&resolution=5&from=1668448858&to=1668880918 HTTP/2
> Host: tvc4.investing.com
> accept: */*
> user-agent: Mozilla/5.0
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Wed, 23 Nov 2022 14:22:13 GMT
< content-type: text/html; charset=utf-8
< access-control-allow-origin: https://tvc-invdn-com.investing.com
< x-requested-with: XMLHttpRequest
< access-control-allow-methods: POST, GET, OPTIONS, PUT, DELETE
< access-control-allow-headers: Content-Type, Depth, User-Agent, X-File-Size, X-Requested-With, If-Modified-Since, X-File-Name, Cache-Control, accept, sessionid, x-csrftoken, content-type
< x-benchmark-1a: 0ms, mem alloc - 768.00Kb start
< x-benchmark-1b: 3ms, mem alloc - 1024.00Kb getPairIdBySymbol
< x-benchmark-1c: 3ms, mem alloc - 1024.00Kb getIntervalByResolution
< x-benchmark-1d: 15ms, mem alloc - 1024.00Kb getPairDataAndAttributes in TradingviewConnector::findCandles
< x-benchmark-1e: 23ms, mem alloc - 3.50Mb getCandles in TradingviewConnector::findCandles
< x-benchmark-1f: 24ms, mem alloc - 3.75Mb end of TradingviewConnector::findCandles
< x-benchmark-1g: 25ms, mem alloc - 3.50Mb findCandles
< vary: Accept-Encoding,User-Agent
< content-security-policy: upgrade-insecure-requests; block-all-mixed-content
< cf-cache-status: DYNAMIC
< set-cookie: __cf_bm=z7PZf80TAcglrPBqnAC0p4ApGjbAfRvCP6uZDBinzzs-1669213333-0-AYMziboPajXlvpP44SA/dGfK246VFdm8eaSnco06Ug7FiLfKcTheS1UjW4S8yD0EEJFSW7kqengoBx6dH8W7YLg=; path=/; expires=Wed, 23-Nov-22 14:52:13 GMT; domain=.investing.com; HttpOnly; Secure; SameSite=None
< server: cloudflare
< cf-ray: 76ea8e461cc9ad68-ATL
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
{ [314 bytes data]
100 19882    0 19882    0     0  61780      0 --:--:-- --:--:-- --:--:-- 62719
* Connection #0 to host tvc4.investing.com left intact