Why is it safe to store API keys in DigitalOceans API "Environment Variable" option when using "App Platform"?

Question

I am setting up a NodeJS backend that needs to hold a few API keys to contact among others Firebase to authenticate users.

I have been able to publish the backend and succesfully get a HTTP reply from an API endpoint on it. I use Express to run the NodeJS API.

Now I wish the NodeJS API to act as my user and database contactpoint. But I am unsure where to safely store the API keys for the project. As I am relatively new to this type of security.

So, I have Googled around and found that people say that .env files are NOT safe places to store API Keys. Yet on the DigitalOcean documentation it says the following:

Environment variables are excellent places to store sensitive information (such as API keys), or information that needs to be globally accessible from anywhere in the app, (such as version numbers or hardcoded paths).

So my question is, what is the reason that DigitalOcean would say it is safe to store an API key in the environment variables when so many other sources say that it is not. Is this because the danger lies in the accessibility of the file and that DigitalOcean then somehow secures the file? I have noticed they have a "secret" boolean box you can tick, that says it will make it not appear in consoles. But will it also be completely inaccessible for people in general?

My main concern is to prevent hackers or people with bad intentions from accessing the API key. I am not concerned that people with legitimate access will abuse it, only people that dont have legitimate access.

Looking forward to your insights.

Answer 1

I will rank the security of storing secrets in the following orders (from less secure to the most):

1. Storing them directly in your code (really, really bad)
2. Store them in environment variables
3. Storing them in files on disks
   • Access permission can be set, e.g., who can read it
   • If you load the content into env vars, it will be no different from Point 2
4. Storing them in object stores (AWS S3, Google Cloud Storage)
5. Storing them using secrets managers (AWS, GCP Secret Manager)
6. Using IAM (not really a place to store secrete but a way to control who can access the secrets using the methods above).

Eloboration:

Point 2 & 3:

The difference between points 2 and 3 becomes obscure if your system is compromised. In this case, if the attacker has the root access, he/she can read/write/delete/change everything, so point 3 will fail. For point 2, the attacker can access environment variables via 3 methods:

According to @phmmmer in this post

1. The running environment of the process When the process is running, the environment variables of that process can be > > accessed through /proc/$PID/environ. However, only the user who owns the process, or root, can access that file.

2. The source of the environment variables If you're using an init script, and the variables are stored in that init script, the variables can, of course, be obtained by reading that script.

Or if the environment variables are coming from somewhere else, then wherever that is.

3. 'ps' output Yeah, I know I said 2, and in any decent system, it will be 2. However, if the admin doesn't know what he's doing, it's possible to open up the 3rd avenue.

If the process is launched via something like sh -c 'cd /foo/bar; POP=tart /my/executable', then that sh process will be visible in ps:

$ ps ax | grep POP phemmer   3085  14   5  0.0  0.0 SN         00:00
sh -c cd /; POP=tart sleep 10```

Point 4:

You can store secrets as objects in object stores and download them during deployment or during boot time. Most of the object stores offer logging functionalities, so you can also monitor how your secrets are being used and who is using them. Note:Point 4 is good, if and only if you properly configure IAM permissions. Otherwise, it will be no different from Point 2, albeit now you are getting the file from a remote location.

Point 5:

Your application can call the secret managers' APIs in order to consume the secrets. Your secrets are not stored anywhere locally except in the memory. And, Yes. You can encrypt your memory as well, and you can read more in this post

Point 6:

IAMs or identity and access management are used to grant the right access to the right resource to the right person/user/application at the right time. It is not a storage for secrets, but it can be used to manage access to secret storage.

So, to answer your question:

"Environment variables are excellent places to store sensitive information" might be an overstatement from DO. But they do offer something more than just storing your sensitive information in environment variables and just leaving them there.

In the next post "How to Use Environment Variables in App Platform" of the documentation that you mentioned in your question, DO does offer some levels of access control and encryption regarding environment variables:

1. Runtime variables
   • During app creation
   • Afer app creation
2. Build time variables
3. Bindable Variables within Environment Variables
   • App-wide variables
   • Component-specific variables

So when people are saying it is not safe to store sensitive information using environment variables, most of them are talking about vanilla env vars with no access control and encryption. And in this case, it is definitely not safe to store your API Keys there. Plain, vanilla environment variables should be used to store information like port numbers, production environments, or other non-sensitive information.

Here are some posts that I used as references for this answer:

1. 5 ways to manage serverless secrets, ranked best to worst
2. A comprehensive guide to managing secrets in your Terraform code
3. DigitalOcean: How to Use Environment Variables in App Platform

Answer 2

Got to agree with the previous post. A good practice is to think "if your system is compromised, they are running as ROOT on your system". If there is anything you can do as sudo - assume they can too.

One item that I would like to add is to get into the habit of using unique API keys for dev / test / prod / and each local developer. I know it seems obvious but I have seen to many cases where people do great things in prod but re-use the same API key in dev or even allow a developer to use the same prod API key on their local machine where it isn't protected with the same rigor.

Also using IAM as indicated above is great to make sure to limit access, but if you go down this route, make sure you are willing to make it harder for yourself to move between Azure, AWS, GCP environments.