CODEWITHSUNDEEP
javaspringspring-boot
Chloe
Chloe

Reputation: 26294

How do I serve robots.txt in Spring framework?

I saw this but it's obsolete. I tried the following:

Created src/main/resources/static/robots.txt. 

@Configuration
public class WebConfig extends WebMvcConfigurerAdapter{

    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {
//        registry.addResourceHandler("/robots.txt").addResourceLocations("classpath:/static/robots.txt");
//        registry.addResourceHandler("/robots.txt").addResourceLocations("/static/robots.txt");
//        registry.addResourceHandler("/robots.txt").addResourceLocations("/static/");
        registry.addResourceHandler("/robots.txt").addResourceLocations("classpath:/static/");

Each time,

$ curl -i http://localhost:8080/robots.txt
HTTP/1.1 302
Set-Cookie: JSESSIONID=D1E5304FE8E693115A1FFB0F76786ABD; Path=/; HttpOnly
Location: http://localhost:8080/pageNotFound
Content-Length: 0

However static CSS works.

$ curl -iq http://localhost:8080/css/style.css | head
HTTP/1.1 200

I killed the server and ran mvn spring-boot:run each time I changed the code.

It says this when I start the server:

2018-05-03 17:22:18.639  INFO 25148 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/robots.txt] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
...
2018-05-03 17:22:39.139  INFO 25148 --- [nio-8080-exec-1] c.s.shorturl.apis.ShortUrlApiController  : Method name: doUrlRequest() request http method is GET

It is executing this instead.

@Controller
@Scope("session")
@ApiIgnore
public class ProjectShortUrlController implements ErrorController{
    @RequestMapping(value = "/*")
    public void doUrlRequest(HttpServletRequest request, HttpServletResponse response) {
      CustomLogger.info(TAG, "rediection: ", "Method name: doUrlRequest() request http method is "+request.getMethod());

Documentation: https://docs.spring.io/spring/docs/5.1.0.BUILD-SNAPSHOT/spring-framework-reference/web.html#mvc-config-static-resources

Spring 4.3.10

I found if I do this

@Configuration
public class WebConfig extends WebMvcConfigurationSupport {
    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {
registry.addResourceHandler("/robots.txt").addResourceLocations("/static/robots.txt");

Then robots.txt works

$ curl -i http://localhost:8080/robots.txt
HTTP/1.1 200
Content-Type: text/plain
Content-Length: 28

User-agent: *
Disallow: /

But all other URLs start to give "Whitelabel Error Page"!

Upvotes: 2

Views: 6669

Answers (2)

Ankur Jain
Ankur Jain

Reputation: 1404

I tired the following way in application-context.xml, and it worked absolutely fine.

<mvc:resources mapping="/robots.txt" location="/custom/bots/robots.txt" />
<mvc:resources mapping="/favicon.ico" location="/custom/images/main_logo_pink.png" />

I have put just underneath, like the other static files in my project.

<mvc:annotation-driven />

<mvc:resources mapping="/custom/styles/**" location="/custom/styles/" />
<mvc:resources mapping="/custom/scripts/**" location="/custom/scripts/" />
<mvc:resources mapping="/custom/images/**" location="/custom/images/" />

Upvotes: 1

Chloe
Chloe

Reputation: 26294

I gave up on WebMvcConfigurerAdapter. I think controller requests have priority over resource handlers. I just used this in the controller. 

@RequestMapping(value = "/robots.txt")
public void robots(HttpServletRequest request, HttpServletResponse response) {
    try {
        response.getWriter().write("User-agent: *\nDisallow: /\n");
    } catch (IOException e) {
        CustomLogger.info(TAG, "robots", "robots(): "+e.getMessage());
    }
}

I might have had trouble because there was a @RequestMapping(value = "/*") path.

Upvotes: 3

Related Questions