Understanding How to Store Web Push Endpoints

Question

I'm trying to get started implementing Web Push in one of my apps. In the examples I have found, the client's endpoint URL is generally stored in memory with a comment saying something like:

In production you would store this in your database...

Since only registered users of my app can/will get push notifications, my plan was to store the endpoint URL in the user's meta data in my database. So far, so good.

The problem comes when I want to allow the same user to receive notifications on multiple devices. In theory, I will just add a new endpoint to the database for each device the user subscribes with. However, in testing I have noticed that endpoints change with each subscription/unsubscription on the same device. So, if a user subscribes/unsubscribes several times in a row on the same device, I wind up with several endpoints saved for that user (all but one of which are bad).

From what I have read, there is no reliable way to be notified when a user unsubscribes or an endpoint is otherwise invalidated. So, how can I tell if I should remove an old endpoint before adding a new one?

What's to stop a user from effectively mounting a denial of service attack by filling my db with endpoints through repeated subscription/unsubscription?

That's more meant as a joke (I can obvioulsy limit the total endpoints for a given user), but the problem I see is that when it comes time to send a notification, I will blast notification services with hundreds of notifications for invalid endpoints.

---

I want the subscribe logic on my server to be:

1. Check if we already have an endpoint saved for this user/device combo
2. If not add it, if yes, update it

The problem is that I can't figure out how to reliably do #1.

Answer 1

I'm guessing that in the real world, most users don't unsubscribe and re-subscribe very often. However, Push subscriptions getting stale can be a problem. And users with more than one device can get notified on the wrong device if your server is using the most recent Push subscription and the user has meanwhile returned to a device with a subscription which is older, but still valid.

One way to deal with this, which I have considered and dismissed, would be to re-subscribe the user every time they fire up a client. However, I could find no information from the Apple/Google/etc (the big browser push service providers) as to whether there are any limits on this or whether it's a bad practice. Don't want to get black-listed.... [BTW, Apple gives errors to push requests as JSON in the response content, whereas Google sends plain text. Broke my server code with Chrome client after writing it while testing on Safari.]

So, rather than simply inserting the latest subscription into, say, a column in a users db table, perhaps one should insert it into a new table with a primary key consisting of user_id /and/ browser_id. I'll show an example of what browser_id might be below. Then, when the user comes from a particular browser on a particular device, look up the appropriate subscription and fire away at the endpoint.

[Another option would be to store the complete subscription JSON in a cookie or localStorage and send it to your server very often, but a few hundred extra bytes on every HTTP request seems like a waste. Thoughts anyone?]

Anyway, here's my take on making a smaller browser_id and using it along with user_id to look up the subscription JSON on the server side (the comment is a little redundant; you may just want to look at the code):

    /*
        This is to help the server use the right web Push API subscription if the user switches
        between different browsers; e.g. say a user is on browser A and the server caches a push
        subscription for them, and then they go to browser B and we cache a new subscription for
        them, and then they go back to browser A and we don't cache a new subscription for them
        because browser A already has a subscription going because the service worker is still
        running from their previous session; then the server wants to send them a notification
        and it uses the browser B subscription instead of the browser A one; clearly the 
        subscriptions need to be cached per user, per browser -- but how to identify each browser
        uniquely?  Upon first visiting the site, we'll create a browser_id that combines the
        users initial IP address with a timestamp; if the visitor is later issued a user_id, 
        we will store any Push subscriptions in a db table with primary key of user_id and 
        browser_id.     Sample browser_id: 10.0.0.255-1680711958431
    */
    async function set_browser_id() {

        let browser_id = Cookies.get('browser_id') || localStorage.browser_id;

        if ( ! browser_id) {

            /*
            const response = await fetch("https://api.ipify.org?format=json");
            const jsonData = await response.json();
            const ip_addr = jsonData.ip;
            */
            const timestamp = Date.now();

            browser_id = `${ip_addr}-${timestamp}`; // server set JS const ip_addr earlier

            console.log(`NEW browser_id: ${browser_id}`);
        }
        else { console.log(`EXISTING browser_id: ${browser_id}`); }

        // set whether new or existing to keep it from ever expiring
        Cookies.set('browser_id', browser_id, 
                { path: '/', domain: `.${hostname}`, expires: 365, secure: true });
        
        localStorage.browser_id = browser_id; // backup in persistent storage
    }

    window.onload = set_browser_id; // do it after everything else

The browser_id should be pretty persistent given the localStorage backup of the cookie (and might be useful for analytical purposes). Now your server can get a short little browser_id cookie with every request and can use it to look up the longer Push subscription JSON. In fact, you could probably skip the user_id in the db table since the chance of a browser_id namespace collision is infinitesimal. Of course, whenever the user generates a new subscription, you'll need the client to send that over so the server can update the db table.

I'd be curious what others think about this plan; I'm implementing it presently. And also about any security concerns with passing the complete subscription JSON around in cookies repeatedly -- your private VAPID key is needed to send a notification, right? But....

p.s. I'm on about this now since Apple /finally/ turned on the Push API with iOS 16.4 -- yay! About time.

**EDIT: So, to determine the same browser/device/user I make up a browser_id (I've now just moved to a timestamp of fine enough granularity) and store in in a cookie and in localStorage. Unless the user deletes the localStorage, it'll stay the same for months or longer. Send it to the server with each subscription, and if it matches user_id/browser_id in your db, overwrite the old sub. The harder thing for me was the user switching between different devices/browsers, while also possibly having multiple games in play. I log the user out of a specific game session if they start playing the same game in a different place (got that idea from bridgebase.com), but it's still possible that you end up sending a notification to the user for a specific game session to the wrong device/browser. I don't know if there's a perfect solution to this problem. My solution is to update the subscription in the user_id/browser_id row in the db every time the user goes back to a specific game, even if subscription hasn't changed, just so I can update another column in the table that holds a Unix epoch time. Then when the server is not sure where to send a notification for a certain game, since that's not a column in the table, since it doesn't make sense, I grab the most recent subscription for a user_id using something like this:

        SELECT push_sub_json FROM user_x_browser_id
        WHERE user_id = n
        ORDER BY last_update_time DESC
        LIMIT 1

Not a 100%, but if someone is playing multiple games on multiple browser/device combos, they can deal with it. I do often have the browser_id too, if the client has sent it recently, in which case I am nearly 100%. I could have the client/server channel send browser_id all time, but that would be a waste of effort. I'll call this good 'nuff. **

Answer 2

Another way is to have a keep alive field on you server and have your service worker update it whenever it receives a push notification. Then regularly purge endpoints which haven't been responded to recently.

Answer 3

I will just add a new endpoint to the database for each device the user subscribes with

The best approach is to have a table like this:

endpoint | user_id

• add an unique constraint (or a primary key) on the endpoint: you don't want to associate the same browser to multiple users, because it's a mess (if an endpoint is already present but it has a different user_id, just update the user_id associated to it)
• user_id is a foreign key that points to your users table

if a user subscribes/unsubscribes several times in a row on the same device, I wind up with several endpoints saved for that user (all but one of which are bad).

Yes, unfortunately the push API has a wild unsubscription mechanism and you have to deal with it.

The endpoints can expire or can be invalid (or even malicious, like android.chromlum.info). You need to detect failures (using the HTTP status code, timeouts, etc.) when you try to send the push message from your application server. Then, for some kind of failures (permanent failures, like expiration) you need to delete the endpoint.

What's to stop a user from effectively mounting a denial of service attack by filling my db with endpoints through repeated subscription/unsubscription?

As I described above, you need to properly delete the invalid endpoints, once you realize that they are expired or invalid. Basically they will produce at most one invalid request. Moreover, if you have high throughput, it takes only a few seconds for your server to make requests for thousands of endpoints.

My suggestions are based on a lot of experiments and thinking done when I was developing Pushpad.