Kubernetes: using container as proxy

Question

I have the following pod setup:

apiVersion: v1
kind: Pod
metadata:
  name: proxy-test
  namespace: test
spec:
  containers:
    - name: container-a
      image: <Image>
      imagePullPolicy: Always
      ports:
        - name: http-port
          containerPort: 8083
    - name: container-proxy
      image: <Image>
      ports:
        - name: server
          containerPort: 7487
          protocol: TCP
    - name: container-b
      image: <Image>

I exec into container-b and execute following curl request:

curl --proxy localhost:7487 -X POST http://localhost:8083/

Due to some reason, http://localhost:8083/ is directly getting called and proxy is ignored. Can someone explain why this can happen ?

Answer 1

Environment

I replicated the scenario on kubeadm and GCP GKE kubernetes clusters to see if there is any difference - no, they behave the same, so I assume AWS EKS should behave the same too.

I created a pod with 3 containers within:

apiVersion: v1
kind: Pod
metadata:
  name: proxy-pod
spec:
  containers:
  - image: ubuntu # client where connection will go from
    name: ubuntu
    command: ['bash', '-c', 'while true ; do sleep 60; done']
  - name: proxy-container # proxy - that's obvious
    image: ubuntu
    command: ['bash', '-c', 'while true ; do sleep 60; done']
  - name: server # regular nginx server which listens to port 80
    image: nginx

For this test stand I installed squid proxy on proxy-container (what is squid and how to install it). By default it listens to port 3128.

As well as curl was installed on ubuntu - client container. (net-tools package as a bonus, it has netstat).

Tests

Note!

• I used 127.0.0.1 instead of localhost because squid has some resolving questions, didn't find an easy/fast solution.
• curl is used with -v flag for verbosity.

We have proxy on 3128 and nginx on 80 within the pod:

# netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      -

curl directly:

# curl 127.0.0.1 -vI

*   Trying 127.0.0.1:80... # connection goes directly to port 80 which is expected
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> HEAD / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.68.0
> Accept: */*

curl via proxy:

# curl --proxy 127.0.0.1:3128 127.0.0.1:80 -vI

*   Trying 127.0.0.1:3128... # connecting to proxy!
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0) # connected to proxy
> HEAD http://127.0.0.1:80/ HTTP/1.1 # going further to nginx on `80`
> Host: 127.0.0.1
> User-Agent: curl/7.68.0
> Accept: */*

squid logs:

# cat /var/log/squid/access.log

1635161756.048      1 127.0.0.1 TCP_MISS/200 958 GET http://127.0.0.1/ - HIER_DIRECT/127.0.0.1 text/html
1635163617.361      0 127.0.0.1 TCP_MEM_HIT/200 352 HEAD http://127.0.0.1/ - HIER_NONE/- text/html

NO_PROXY

NO_PROXY environment variable might be set up, however by default it's empty.

I added it manually:

# export NO_PROXY=127.0.0.1

# printenv | grep -i proxy
NO_PROXY=127.0.0.1

Now curl request via proxy will look like:

# curl --proxy 127.0.0.1:3128 127.0.0.1 -vI

* Uses proxy env variable NO_PROXY == '127.0.0.1' # curl detects NO_PROXY envvar
*   Trying 127.0.0.1:80... # and ignores the proxy, connection goes directly
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> HEAD / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.68.0
> Accept: */*

It's possible to override NO_PROXY envvar while executing curl command with --noproxy flag.

--noproxy no-proxy-list

Comma-separated list of hosts which do not use a proxy, if one is specified. The only wildcard is a single * character, which matches all hosts, and effectively disables the proxy. Each name in this list is matched as either a domain which contains the hostname, or the hostname itself. For example, local.com would match local.com, local.com:80, and www.local.com, but not www.notlocal.com. (Added in 7.19.4).

Example:

# curl --proxy 127.0.0.1:3128 --noproxy "" 127.0.0.1 -vI

*   Trying 127.0.0.1:3128... # connecting to proxy as it was supposed to
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0) # connection to proxy is established
> HEAD http://127.0.0.1/ HTTP/1.1 # connection to nginx on port 80
> Host: 127.0.0.1
> User-Agent: curl/7.68.0
> Accept: */*

This proves that proxy works! with localhost.

Another option is something incorrectly configured in proxy which is used in the question. You can get this pod and install squid and curl into both containers and try yourself.