Ory Hydra 403 With Reverse Proxy

Question

I am trying to get Ory Hydra working in Docker-Compose with Nginx. Due to my iterative approach, I already had a working system before adding Nginx. In other words, it was working, now it isn't.

The changes which I think might affect this process are: Nginx, Hydra's host name, oauth2 config in my demo application. Also, my setup is based on the Kratos-Hydra integration demo. Of course Kratos and the UI are now also accessed from Nginx, so that obviously has changed as well, but I don't think that's causing problems.

So here's what happens when I try to access a secured endpoint in my demo app:

• Redirect to kratos-ui for login
• Enter details and send request
• Login succeeds
• Hydra returns 403: You are not allowed to perform this action.

Nginx:

# kratos-selfservice-ui-node
server {
    server_name self.localhost;

    proxy_set_header Host self.localhost;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    location / {
        proxy_pass http://self:3000;
    }
}

# kratos
server {
    server_name login.localhost;

    #proxy_set_header Host ...;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    location / {
        proxy_pass http://kratos:4433;
    }
}

#hydra
server {
    server_name oidc.localhost;

    #proxy_set_header Host 127.0.0.1:4444;
    #proxy_set_header Host oidc.localhost;
    #proxy_set_header X-Real-IP $remote_addr;
    #proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    location / {
        proxy_pass http://hydra:4444;
    }
}

Request:

GET /oauth2/auth?client_id=auth-code-client&login_verifier=8b5f6d3f964c4470ab2e42fac90ae1c2&nonce=XTr2FJETXFsr6kxw3SlZsbh7rbQ_RMw8SdK3MeMCAs0&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Flogin%2Foauth2%2Fcode%2Fhydra&response_type=code&scope=openid+profile&state=4OSX7C_A84-u-6MlUZOlzjAAXiBYIzbKGfGwcAp1n1M%3D HTTP/1.1
Host: hydra:4444
User-Agent: <stuff>
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://self.localhost/
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache

Hydra entry in docker-compose:

  # OIDC Server
  # Configured to use Kratos for identities
  hydra:
    image: oryd/hydra:v1.6.0-alpine
    container_name: hydra
    depends_on:
      - hydra-migrate
    #ports:
      #- 4444:4444 # Public port
      #- 4445:4445 # Admin port
      #- 5555:5555 # Port for hydra token user
    command:
      serve all --dangerous-force-http
    volumes:
      -
        type: bind
        source: ./config/hydra
        target: /home/ory
    environment:
      - DSN=postgres://pguser:secret@postgres:5432/hydra?sslmode=disable
      - OIDC_SUBJECT_IDENTIFIERS_SUPPORTED_TYPES=public,pairwise
      - LOG_LEAK_SENSITIVE_VALUES=true
      ##- URLS_SELF_ISSUER=http://127.0.0.1:4444
      ##- URLS_SELF_PUBLIC=http://127.0.0.1:4444
      #- URLS_SELF_ISSUER=http://hydra:4444
      #- URLS_SELF_PUBLIC=http://hydra:4444
      - URLS_SELF_ISSUER=http://oidc.localhost
      - URLS_SELF_PUBLIC=http://oidc.localhost
      - URLS_CONSENT=http://self.localhost/auth/hydra/consent
      - URLS_LOGIN=http://self.localhost/auth/hydra/login
      - URLS_LOGOUT=http://self.localhost/logout
      - SECRETS_SYSTEM=youReallyNeedToChangeThis
      - OIDC_SUBJECT_IDENTIFIERS_PAIRWISE_SALT=youReallyNeedToChangeThis
      - OAUTH2_EXPOSE_INTERNAL_ERRORS=true;
      - OAUTH2_INCLUDE_LEGACY_ERROR_FIELDS=true
    restart: on-failure
    networks:
      - <ory>

Spring Boot App config:

spring:
  security:
    oauth2:
      client:
        registration:
          hydra:
            client-name: Demo OIDC Client with Spring Boot :D
            client-id: auth-code-client
            client-secret: secret
        provider:
          hydra:
            issuer-uri: http://oidc.localhost/

And here's the client that I created:

docker exec hydra \
    hydra clients create \
    --endpoint http://127.0.0.1:4445 \
    --id auth-code-client \
    --secret secret \
    --grant-types authorization_code,refresh_token \
    --response-types code,id_token \
    --scope openid,profile \
    --callbacks http://localhost:8080/login/oauth2/code/hydra

/etc/hosts stuff I added:

# Dev stuff
127.0.0.1   self.localhost
127.0.0.1   login.localhost
127.0.0.1   oidc.localhost
127.0.0.1   oidc-demo.localhost
127.0.0.1   hello.localhost

Hydra logs:

  < THIS IS FROM THE INITIAL REQUEST TO THE KRATOS UI >
time=2022-01-24T12:49:00Z level=info msg=started handling request http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8 accept-encoding:gzip, deflate accept-language:en-US,en;q=0.5 cache-control:no-cache user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0] host:127.0.0.1:4444 method:GET path:/oauth2/auth query:response_type=code&client_id=auth-code-client&state=-__end_skoEpW7KSAfzng1yZyOdJoF2-Cfzls-dccD4%3D&redirect_uri=http://localhost:8080/login/oauth2/code/hydra remote:192.168.16.11:43608 scheme:http]

time=2022-01-24T12:49:00Z level=info msg=access allowed audience=audit http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8 accept-encoding:gzip, deflate accept-language:en-US,en;q=0.5 cache-control:no-cache user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0] host:127.0.0.1:4444 method:GET path:/oauth2/auth query:response_type=code&client_id=auth-code-client&state=-__end_skoEpW7KSAfzng1yZyOdJoF2-Cfzls-dccD4%3D&redirect_uri=http://localhost:8080/login/oauth2/code/hydra remote:192.168.16.11:43608 scheme:http] service_name=ORY Hydra service_version=v1.6.0

time=2022-01-24T12:49:00Z level=info msg=completed handling request http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8 accept-encoding:gzip, deflate accept-language:en-US,en;q=0.5 cache-control:no-cache user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0] host:127.0.0.1:4444 method:GET path:/oauth2/auth query:response_type=code&client_id=auth-code-client&state=-__end_skoEpW7KSAfzng1yZyOdJoF2-Cfzls-dccD4%3D&redirect_uri=http://localhost:8080/login/oauth2/code/hydra remote:192.168.16.11:43608 scheme:http] http_response=map[status:302 text_status:Found took:15.9869ms]

time=2022-01-24T12:49:00Z level=info msg=started handling request http_request=map[headers:map[accept:application/json] host:hydra:4445 method:GET path:/oauth2/auth/requests/login query:login_challenge=3a6891edb669434f821a0d5413519bfe remote:192.168.16.2:54218 scheme:http]

time=2022-01-24T12:49:00Z level=info msg=completed handling request http_request=map[headers:map[accept:application/json] host:hydra:4445 method:GET path:/oauth2/auth/requests/login query:login_challenge=3a6891edb669434f821a0d5413519bfe remote:192.168.16.2:54218 scheme:http] http_response=map[status:200 text_status:OK took:3.034ms]

 < THIS IS AFTER LOGIN >

time=2022-01-24T12:49:59Z level=info msg=started handling request http_request=map[headers:map[accept:application/json] host:hydra:4445 method:GET path:/oauth2/auth/requests/login query:login_challenge=3a6891edb669434f821a0d5413519bfe remote:192.168.16.2:54292 scheme:http]

time=2022-01-24T12:49:59Z level=info msg=completed handling request http_request=map[headers:map[accept:application/json] host:hydra:4445 method:GET path:/oauth2/auth/requests/login query:login_challenge=3a6891edb669434f821a0d5413519bfe remote:192.168.16.2:54292 scheme:http] http_response=map[status:200 text_status:OK took:3.7631ms]

time=2022-01-24T12:49:59Z level=info msg=started handling request http_request=map[headers:map[accept:application/json] host:hydra:4445 method:PUT path:/oauth2/auth/requests/login/accept query:login_challenge=3a6891edb669434f821a0d5413519bfe remote:192.168.16.2:54296 scheme:http]

time=2022-01-24T12:49:59Z level=info msg=completed handling request http_request=map[headers:map[accept:application/json] host:hydra:4445 method:PUT path:/oauth2/auth/requests/login/accept query:login_challenge=3a6891edb669434f821a0d5413519bfe remote:192.168.16.2:54296 scheme:http] http_response=map[status:200 text_status:OK took:8.8812ms]

time=2022-01-24T12:49:59Z level=info msg=started handling request http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8 accept-encoding:gzip, deflate accept-language:en-US,en;q=0.5 cache-control:no-cache referer:http://self.localhost/ user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0] host:127.0.0.1:4444 method:GET path:/oauth2/auth query:client_id=auth-code-client&login_verifier=fedb596a040648b8b626e0f7e4f3f04a&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Flogin%2Foauth2%2Fcode%2Fhydra&response_type=code&state=-__end_skoEpW7KSAfzng1yZyOdJoF2-Cfzls-dccD4%3D remote:192.168.16.11:43694 scheme:http]

time=2022-01-24T12:49:59Z level=info msg=access denied audience=audit error=map[message:request_forbidden reason:You are not allowed to perform this action. status:Forbidden status_code:403] http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8 accept-encoding:gzip, deflate accept-language:en-US,en;q=0.5 cache-control:no-cache referer:http://self.localhost/ user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0] host:127.0.0.1:4444 method:GET path:/oauth2/auth query:client_id=auth-code-client&login_verifier=fedb596a040648b8b626e0f7e4f3f04a&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Flogin%2Foauth2%2Fcode%2Fhydra&response_type=code&state=-__end_skoEpW7KSAfzng1yZyOdJoF2-Cfzls-dccD4%3D remote:192.168.16.11:43694 scheme:http] service_name=ORY Hydra service_version=v1.6.0

time=2022-01-24T12:49:59Z level=info msg=completed handling request http_request=map[headers:map[accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8 accept-encoding:gzip, deflate accept-language:en-US,en;q=0.5 cache-control:no-cache referer:http://self.localhost/ user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0] host:127.0.0.1:4444 method:GET path:/oauth2/auth query:client_id=auth-code-client&login_verifier=fedb596a040648b8b626e0f7e4f3f04a&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Flogin%2Foauth2%2Fcode%2Fhydra&response_type=code&state=-__end_skoEpW7KSAfzng1yZyOdJoF2-Cfzls-dccD4%3D remote:192.168.16.11:43694 scheme:http] http_response=map[status:302 text_status:Found took:8.6448ms]

Update

In the course of trying out everything before posting this question to stackoverflow.com, I went back to an older git commit which was working.

Bad news, it doesn't work anymore. I have the official kratos-hydra integration checked out and built ($ git status -> On branch hydra-integration \n Your branch is up to date with 'origin/hydra-integration'.), and I did the required steps, and now I get this:

$ docker exec hydra_hydra_1 \                  
    hydra token user \
    --client-id auth-code-client \
    --client-secret secret \
    --endpoint http://127.0.0.1:4444/ \
    --port 5555 \
    --scope openid,offline 
Config file not found because "Config File ".hydra" Not Found in "[/home/ory]""
Setting up home route on http://127.0.0.1:5555/
Setting up callback listener on http://127.0.0.1:5555/callback
Press ctrl + c on Linux / Windows or cmd + c on OSX to end the process.
If your browser does not open automatically, navigate to:

    http://127.0.0.1:5555/

 < then I navigate to 127.0.0.1:5555, click on authorize application, I have to enter log in details, and then I get redirected to an error page >

Got error: The request is not allowed
http: Server closed

The browser doesn't add much info to that:

An error occurred
request_forbidden

The request is not allowed

You are not allowed to perform this action.

I've tried deleting all containers, images, volumes, and networks, browser cookies, using a different browser, restarting docker, restarting my computer. Same problem.

Something that seems odd is that the app always asks me to log in, even if I am logged in already when I go to the UI url manually. I remember that if I was already logged in, it wouldn't ask me to log in again?

Update

I was on the hydra-integration branch for some reason, instead of the hydra-integration-2021, which is why going to back to the basics didn't work. That's my mistake.

The actual project is not working, but after reevaluating the work required and benefits/drawbacks/requirements I decided to switch from Kratos to werther.

Answer 1

To bring some sanity to this I would first update to good internal and external URLs. The crux of the problem feels like you need to configure Ory Hydra (running inside the cluster) with an internet URL used in browsers etc, and this will be different to Ory Hydra's physical URL.

SIMILAR CURITY EXAMPLE

This feels like a similar setup to yours - it's worth taking a little time to understand resources:

• Docker Compose
• Tutorial Article
• Authorization Server Configuration

Look at the base-url property at the top of the third link above, which is what internet clients such as browsers use to connect to the Authorization Server. There will be a property like this that you can set in Hydra.