How to store third-party credentials (no api, no OAuth) for automatic reuse?

Question

I've read several Stack Overflow threads, I still can't decide what is the best option for my case. And the most secure one.

Here is the story. My webapp is to help users automatically get an overview of some of their data available in some third-party website. I need to store for each user some third-party credentials. Each night or so, my server will connect to the third-party services on the users' behalf and retrieve the required data.

Most of those third-party sites do not implement any API or OAuth mechanism, so I was thinking to do some web scraping.

I've read in many places that storing the credentials in the DB is not a good idea - especially because my app needs access to the password (so it has to be encrypted in such a way I can easily reuse it).

So, I have two options left:

1. Whenever I access (via webscraping) the third-party service, I store on the server the cookies issued by that service, for future reuse. I encrypt them and keep them encrypted in a DB, and decrypt them only when I need them. The problem is that the cookie can be denied or expired after a while, and so the automatic process wouldn't work any more.
2. I store the credentials in the environment variables. I will be on Node.js and Heroku. That's an idea I found in another SO thread. But I'm wondering about the security of this idea. Is it really safe? No one can access them but me? And what about if I reach many users. Like 1000 users, with 10 services. That's 10000 credentials to store in the env variables. That doesn't seem like a good idea.

I found two interesting questions on Stack Overflow but they don't fit 100% with my use case.

• Security model: log in to third-party site with user's credentials (that gave me the idea in point 1)
• Rails storing third party credentials.. Anyone know best practice? (gave me the idea in point 2).

Answer 1

I add another answer because maybe this one will do the trick for you. You said the main goal of your website is to have an overview of third party applications. But what if instead of updating this overview every night, you update it when the user logs in ? It changes everything, because you could use the user's password (of your website) as master password to encrypt (using AES) all the others.

If you do that, the communications between your server and the clients have to be encrypted with SSL pinning, because an attacker could perform a MITM, get the master password and all the others stored in the DB... (Even if in practice it's very hard because you need to hack the client AND the server)

Answer 2

Storing users credentials in a reversible way looks like a terrible idea anyway. But if you really want to store them, I suggest you to use the environment variables solution. But you can improve it. To limit the amount of data you store and don't have 1000000 variables as you said, you can just store an AES encryption key, store all credentials in a DB encrypted with this key, and you just have to get this key (which is in memory) and decrypt the DB. But there is another problem with this solution. As I said, this is stored in RAM memory, so it's not persistent, imagine your server has to reboot for X or Y reason... You will lose the AES key and also the credentials of your users... Moreover, if the attacker performs a memory dump, he will have access to the AES key...

I think the better idea is to store the cookies (in an encrypted way) and when this one expires, you alert the user (by mail, phone, notifications, ...) and ask him to fill his credentials again. But it's not a perfect solution ! Indeed the cookies are a type of credentials and shouldn't be stored either...

Answer 3

Plan for the attacker gaining admin access to the server. Your site will be very attractive to attackers, kind of a one-stop-shop for user credentials so you will need very good security of the login credentials.

There are more than two options for storing the credentials:

1. Use an HSM for the storage or individual credential encryption keys.

2. Keep the credentials on another dedicated server with no Internet access, 2-factor authentication and limit admin personal. Rate limit the access to this server and add rate alarms. Access this server on a per user credential basis over a non-Internet connection. The credentials will only be available to the Internet connected server in memory as used, not at-rest in a file.

Answer 4

Storing a lot of data that changes and grows in environment variables will never be practical, no matter if it's secure or not so this is pretty much out of the question, unless if you have a small fixed number of users.

Not storing credentials in the database is a very good advice, but the cookies are credentials and even if you store them encrypted, your app needs to be able to encrypt it to use it. (This is unlike the situation with verifying passwords of your users when you don't need to ever encrypt them, you only need to see if the provided passwords hash to the same values that you have stored).

This is a hard problem because to make it work you need to have some form of credentials (whether those are passwords or cookies) stored and ready to be used unencrypted (even if they are stored encrypted, you need to store the keys to encrypt it as well).

Also, what you are trying to do can be illegal. Mayke sure that you follow the TOC of every service that you're using or otherwise you may face legal trouble.