Unable to access actuator endpoints when deployed on Google App Engine 11

Question

I have an inherited spring boot application that was happily living on GAE 8 standard running on jetty. We are in the process of upgrading it to GAE 11 standard.

Based the Differences between Java 8 and Java 11/17 we determined that we would try to Migrating to Java 11/17 with bundled services and followed the instructions to Access bundled services using the App Engine APIs JAR..

The appengine-web.xml and pom.xml were updated as specified above, although we do not have a web.xml, we needed <app-engine-apis>true</app-engine-apis> to prevent some errors on start up because we previously used <sessions-enabled> to secure actuator endpoints. We do not use an app.yaml yet.

There are some release scripts in the code that suggest I should be able to access the actuator endpoints for smoke testing our DEV project to compare against our production endpoints prior to release, for instance /_ah/health, so that is where I am starting to validate my upgrade. So far...

• I can access /_ah/health in our current version in production (GAE 8).
• I can access /_ah/health in our current version in development (GAE 8).
• I can access /_ah/health locally on http:8080 after clean package appengine:run (GAE 11, branch), Google App Engine Maven plugin (deploy)
• I cannot access /_ah/health and get 404 Error: Not Found when deployed to out dev (GAE 11, branch)

I've turned up the logs. I can see that is falls through several security filters but I still get a 404:

1. WebAsyncManagerIntegrationFilter
2. SecurityContextPersistenceFilter
3. HeaderWriterFilter
4. CorsFilter
5. LogoutFilter
6. BasicAuthenticationFilter
7. RequestCacheAwareFilter
8. SecurityContextHolderAwareRequestFilter
9. AnonymousAuthenticationFilter
10. SessionManagementFilter
11. ExceptionTranslationFilter

So I am thinking this is related to the Security Configuration.

The intention is to allow the /health and /health/** for all but secure all other actuator endpoints with basic authentication (configured user/pass) in application.yml

Any help would be appreciated. Here is what I think are some valid config files. notes and logs...

• All of the necessary work to upgrade the underlying spring boot application from java 8 to 11 (as suggested by many articles/checklists on the web) was completed many months ago and now we are compiling to java 11 and upgrading our GAE deployment.

appengine-web.xml updated for java 11

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <service>my-api</service>
  <runtime>java11</runtime>
  <instance-class>F4</instance-class>
  <app-engine-apis>true</app-engine-apis>
  <!-- To allow securing actuator endpoints with a login -->
  <sessions-enabled>true</sessions-enabled>
  <automatic-scaling>
  <min-idle-instances>1</min-idle-instances>
  </automatic-scaling>
  <system-properties>
    <property name="java.util.logging.config.file" value="WEB-INF/classes/logging.properties"/>
  </system-properties>
</appengine-web-app>

application.yml

# ...
management:
  endpoints:
    web:
      # GAE Standard Runtime looks for health checks under /_ah - not sure if valid any more
      base-path: /_ah
      exposure:
        include: env,health
  health:
    probes:
    # This enables base-path/health/liveness and base-path/health/readiness
      enabled: true
    # This health check will fail on GAE Standard Runtime
    diskspace:
      enabled: false
spring:
  security:
    user:
      name: foo
      password: bar
      roles: ADMIN
# ...

SecurityConfig.java

@Configuration
public class SecurityConfig extends WebSecurityConfigurerAdapter {

    @Override
    protected void configure(final HttpSecurity http) throws Exception {
        http
            .cors().and().csrf().disable()
            .authorizeRequests()
                .requestMatchers(EndpointRequest.to("health")).permitAll()
                .requestMatchers(EndpointRequest.toAnyEndpoint()).hasRole("ADMIN")
                .antMatchers("/**").anonymous()
            .and().httpBasic();
    }
}

Application.java

@EnableWebSecurity
@SpringBootApplication
public class Application {

    public static void main(final String[] args) {
        SpringApplication.run(Application.class, args);
    }
  // ...
}

ServletInitializer.java

Public class ServletInitializer extends SpringBootServletInitializer {

    @Override
    protected SpringApplicationBuilder configure(final SpringApplicationBuilder application) {
        return application.sources(Application.class);
    }

}

Answer 1

As confirmed in the comments section, using an older working version of Cloud SDK (in this case v371.0.0) resulted in being able to successfully access the endpoints again.

Additionally, the issue has already been reported in the issue tracker: App Engine Standard Java 8: 404 Not Found