CODEWITHSUNDEEP
fastapirobots.txtnoindex
tomo
tomo

Reputation: 101

FastAPI, robots.txt and noindex

Does FastAPI need robots.txt and the tag noindex? I am creating business api app which shouldn't be called by anonymous. So I wonder whether I have to prepare robots.txt and the tag noindex in order to avoid any crawler's action or not.

I made robots.txt router as follows:

@router.get('/robots.txt')
def robots():
    data = """
    User-agent: *
    Disallow: /
    """
    return Response(content=data, media_type='text/plain')

Do I need another change?

Upvotes: 10

Views: 1677

Answers (1)

Welpike
Welpike

Reputation: 61

I tried this and it works

from fastapi.responses import PlainTextResponse
@app.get('/robots.txt', response_class=PlainTextResponse)
def robots():
    data = """User-agent: *\nDisallow: /"""
    return data

Upvotes: 6

Related Questions