CODEWITHSUNDEEP
pythonapicurluser-agent
Jonathan
Jonathan

Reputation: 11365

How to detect if a GET request is from a browser or not

I'm looking for a way to show html to a user if they call from a browser or just give them the API response in JSON if the call is made from an application, terminal with curl or generally any other way.

I know a number of APIs do this and I believe Django's REST framework does this.

I've been able to fool a number of those APIs by passing in my browser's useragent to curl so I know this is done using useragents, but how do I implement this? To cover every single possible or most useragents out there.

There has to be a file/database or a regex, so that I don't have to worry about updating my useragent lists every few months, and worrying that my users on the latest browsers might not be able to access my website.

Upvotes: 2

Views: 3608

Answers (2)

John Stone
John Stone

Reputation: 307

I know this post is a few years old, but since I stumbled upon it...

tldr; Do not use the user agent to determine the return format unless absolutely necessary. Use the Accept header or (less ideal) use a separate endpoint/URL.

The standard and most future-proof way to set the desired return format for a specific endpoint is to use the Accept header. Accept is explicitly designed to allow the client to state what response format they would like returned. The value will be a standard MIME type.

Web browsers, by default, will send text/html as the value of the Accept header. Most Javascript libraries and frontend frameworks will send application/json, but this can usually be explicitly set to something else (e.g. text/xml) if needed. All mobile app frameworks and HTTP client libraries that I am aware of have the ability to set this header if needed.

There are two big problems with trying to use user agent for simply determining the response format:

  1. The list will be massive. You will need to account for every possible client which needs to be supported today. If this endpoint is used internally, this may not be an immediate problem as you might be able to enforce which user agents you will accept (may cause its own set of problems in the future, e.g. forcing your users to a specific version of Internet Explorer indefinitely) which will help keep this list small. If this endpoint is to be exposed externally, you will almost certainly miss something you badly need to accept.
  2. The list will change. You will need to account for every possible client which needs to be supported tomorrow, next week, next year, and in five years. This becomes a self-induced maintenance headache.

Two notes regarding Accept:

  1. Please read up on how to use the Accept header before attempting to implement against it. Here is an actual example from this website: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9. Given this, I would return back HTML.
  2. The value of the header can be */*, which basically just says "whatever" or "I don't care". At that point, the server is allowed to determine the response format.

Upvotes: 5

Jonathan
Jonathan

Reputation: 11365

Using @reevkandari's pastebin, the following Python code works very well in production for me

browser_useragents = ["ABrowse", "Acoo Browser", "America Online Browser", "AmigaVoyager", "AOL", "Arora", "Avant Browser", "Beonex", "BonEcho", "Browzar", "Camino", "Charon", "Cheshire", "Chimera", "Chrome", "ChromePlus", "Classilla", "CometBird", "Comodo_Dragon", "Conkeror", "Crazy Browser", "Cyberdog", "Deepnet Explorer", "DeskBrowse", "Dillo", "Dooble", "Edge", "Element Browser", "Elinks", "Enigma Browser", "EnigmaFox", "Epiphany", "Escape", "Firebird", "Firefox", "Fireweb Navigator", "Flock", "Fluid", "Galaxy", "Galeon", "GranParadiso", "GreenBrowser", "Hana", "HotJava", "IBM WebExplorer", "IBrowse", "iCab", "Iceape", "IceCat", "Iceweasel", "iNet Browser", "Internet Explorer", "iRider", "Iron", "K-Meleon", "K-Ninja", "Kapiko", "Kazehakase", "Kindle Browser", "KKman", "KMLite", "Konqueror", "LeechCraft", "Links", "Lobo", "lolifox", "Lorentz", "Lunascape", "Lynx", "Madfox", "Maxthon", "Midori", "Minefield", "Mozilla", "myibrow", "MyIE2", "Namoroka", "Navscape", "NCSA_Mosaic", "NetNewsWire", "NetPositive", "Netscape", "NetSurf", "OmniWeb", "Opera", "Orca", "Oregano", "osb-browser", "Palemoon", "Phoenix", "Pogo", "Prism", "QtWeb Internet Browser", "Rekonq", "retawq", "RockMelt", "Safari", "SeaMonkey", "Shiira", "Shiretoko", "Sleipnir", "SlimBrowser", "Stainless", "Sundance", "Sunrise", "surf", "Sylera", "Tencent Traveler", "TenFourFox", "theWorld Browser", "uzbl", "Vimprobable", "Vonkeror", "w3m", "WeltweitimnetzBrowser", "WorldWideWeb", "Wyzo", "Android Webkit Browser", "BlackBerry", "Blazer", "Bolt", "Browser for S60", "Doris", "Dorothy", "Fennec", "Go Browser", "IE Mobile", "Iris", "Maemo Browser", "MIB", "Minimo", "NetFront", "Opera Mini", "Opera Mobile", "SEMC-Browser", "Skyfire", "TeaShark", "Teleca-Obigo", "uZard Web", "Thunderbird", "AbiLogicBot", "Link Valet", "Link Validity Check", "LinkExaminer", "LinksManager.com_bot", "Mojoo Robot", "Notifixious", "online link validator", "Ploetz + Zeller", "Reciprocal Link System PRO", "REL Link Checker Lite", "SiteBar", "Vivante Link Checker", "W3C-checklink", "Xenu Link Sleuth", "EmailSiphon", "CSE HTML Validator", "CSSCheck", "Cynthia", "HTMLParser", "P3P Validator", "W3C_CSS_Validator_JFouffa", "W3C_Validator", "WDG_Validator", "Awasu", "Bloglines", "everyfeed-spider", "FeedFetcher-Google", "GreatNews", "Gregarius", "MagpieRSS", "NFReader", "UniversalFeedParser", "!Susie", "Amaya", "Cocoal.icio.us", "DomainsDB.net MetaCrawler", "gPodder", "GSiteCrawler", "iTunes", "lftp", "MetaURI", "MT-NewsWatcher", "Nitro PDF", "Snoopy", "URD-MAGPIE", "WebCapture", "Windows-Media-Player"]
    if any(browser_useragent in request['userAgent'] for browser_useragent in browser_useragents):
        return "<html>Website HTML</html>"
else:
    return api JSON or XML

Upvotes: 0

Related Questions