Nginx real_ip_header through a GCE ingress?

Question

My goal is to filter access by IP address of an angular app deployed on Kubernetes Engine served by nginx through a GCE ingress.

But on my nginx the remote_addr is not right.

$LB_IP is the ip defined here : kubernetes.io/ingress.global-static-ip-name: app-angular

I'm using set_real_ip_from on nginx to set the ip from X-Forwarded-For

set_real_ip_from $LB_IP;
real_ip_header X-Forwarded-For;

The original ip comes in the X-Forwarded-For header as expected from the google doc : https://cloud.google.com/compute/docs/load-balancing/http/#components

I can see the X-Forwarded-For contains the $CLIENT_IP but the remote_addr is not correct, and by the way my filter on IP is not working. Any idea ?

My nginx logs :

10.40.40.40 - - [07/Feb/2018:11:29:48 +0000] "GET /styles.bundle.css HTTP/1.1" 200 35908 "http://MY_URL/home" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36" "$CLIENT_IP, $LB_IP"

But i want :

$CLIENT_IP - - [07/Feb/2018:11:29:48 +0000] "GET /styles.bundle.css HTTP/1.1" 200 35908 "http://MY_URL/home" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36" "$CLIENT_IP, $LB_IP"

My Ingress configuration :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: app-angular
spec:
  rules:
    - host: MY_URL
      http:
        paths:
        - backend:
            serviceName: app-backend
            servicePort: 80

My backend configuration :

apiVersion: v1
kind: Service
metadata:
  name: app-backend
spec:
  type: NodePort
  selector:
    app: app-angular
  ports:
  - port: 80
    targetPort: 80

My deployment configuration :

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: app-angular
spec:
  replicas: 2
  revisionHistoryLimit: 2
  template:
    metadata:
      labels:
        app: app-angular
        tier: frontend
    spec:
      containers:
      - name: app-angular
        image: gcr.io/MY_PROJECT/app-angular:MY_TAG
        imagePullPolicy: Always
        ports:
          - containerPort: 80

My nginx configuration :

server {
  listen 80;

  root /usr/share/nginx/html;

  location / {

    set_real_ip_from $LB_IP;
    real_ip_header X-Forwarded-For;

    allow 130.211.0.0/22; #Google IP
    allow $CLIENT_IP; # FILTERED IP
    allow 10.0.0.0/8; #internal network
    deny all;

    index index.html index.htm;
    try_files $uri $uri/ /index.html =404;
  }

}

Any idea ?

Answer 1

They are two solutions (Thks to gcbirzan for helping me on GCP slack):

1) Update My nginx configuration with the good IPS from IP addresses ranges :

IP addresses ranges can be found here : https://console.cloud.google.com/networking/networks/list. You can just add set_real_ip_from for a region or all needed regions. Don't forget real_ip_recursive on;

server {
  listen 80;

  root /usr/share/nginx/html;

  location / {

    set_real_ip_from 10.128.0.0/20;
    ...
    real_ip_header X-Forwarded-For;
    real_ip_recursive on;

    allow 130.211.0.0/22; #Google IP
    allow $CLIENT_IP; # FILTERED IP
    allow 10.0.0.0/8; #internal network
    deny all;

    index index.html index.htm;
    try_files $uri $uri/ /index.html =404;
  }

}

2) Update My backend configuration with :

externalTrafficPolicy: Local

apiVersion: v1
kind: Service
metadata:
  name: app-backend
spec:
  type: NodePort
  selector:
    app: app-angular
  ports:
  - port: 80
    targetPort: 80
  externalTrafficPolicy: Local

Update nginx configuration :

now the IP shown in $remote_addr will be set with the Load balancer IP for your client requests and with google Infra IPS : 130.211.0.0/22,35.191.0.0/16

Don't forget real_ip_recursive on;

server {
  listen 80;

  root /usr/share/nginx/html;

  location / {

    set_real_ip_from $LB_IP;
    set_real_ip_from 130.211.0.0/22;
    set_real_ip_from 35.191.0.0/16;
    real_ip_header X-Forwarded-For;
    real_ip_recursive on;

    allow 130.211.0.0/22; #Google IP
    allow 35.191.0.0/16; #Google IP
    allow $CLIENT_IP; # FILTERED IP
    allow 10.0.0.0/8; #internal network
    deny all;

    index index.html index.htm;
    try_files $uri $uri/ /index.html =404;
  }

}

Answer 2

We're using a HTTP GCLB ingress in front of nginx ingress. To get the RealIP to work I added

kind: ConfigMap
apiVersion: v1
metadata:
  name: ingress-nginx
  labels:
    k8s-addon: ingress-nginx.addons.k8s.io
data:
  ##
  ## Google Infra Blocks needed
  ##  130.211.0.0/22,35.191.0.0/16
  ##
  ## Google Cloud Load Balancer
  ##  GCLB-IP/32
  proxy-real-ip-cidr: 130.211.0.0/22,35.191.0.0/16,[YOUR-GCLB-IP]/32

This allows the nginx to trust the X-Forwarded-For from Google's infrastructure.

Answer 3

The link you provided is for GCP load balancers specifically. As for the nginx Load Balancer, you should consider adding the ExternalTrafficPolicy annotation to your NodePort service to preserve the client IP.

I also found a similar issue dating back to mid-2017 which shows there was an issue with certain versions of the nginx controller, make sure that the image you are using does not still have this issue.

Finally, you may need to add use-proxy-protocol: "true" with the configuration ConfigMap since you are using an ingress which acts as an HTTP(S) Load Balancer which uses proxies.