Managing ‘secrets’ is a pain

Credentials - API Keys, database credentials and certificates - are widely used in many applications we create at In The Pocket. Projects tend to get bigger and more complex year after year. This results in a growing ecosystem of connected managed services, each requiring its own credentials to protect the data they are storing and processing. Securing and managing those credentials becomes more and more challenging over time: as your project tends to get bigger, so does the amount of secrets you’ll need to store.

Recently Google announced its Secret Manager, a new Google Cloud service that aims to centralize, manage and secure sensitive information in a convenient and secure way. You could see this as a vault on steroids, where only the people who know the code have access to the contents of it.

So this looks really promising and it should make engineers’ lives better, but does it really? We've been using Google's Secret Manager for the past few months and in this post, you'll learn what our opinion is so far!

What is a secret manager?

A secret manager is a service that helps you protect access to your applications, services, and IT resources. This service enables you to easily rotate, manage, and retrieve database credentials, API keys, TLS certificates and other secrets needed by an application at runtime.

These are a few of our favorite things

When Google announced its new service, we were all very excited about it! We've been using Berglas or Sops as our go-to solution for managing secrets. But implementing those services is always a time-consuming task, so we were curious if this was the case for the secret manager as well.

Managed service

At In the Pocket, we tend to go for managed services as much as possible. There are many reasons why you should go for managed services, but we'll discuss them in another blog post. A few of our main reasons is we can stay focused on the core business, reduce the risk of outdated software, wrong configuration, security issues, ... and they are quickly implemented (which will be discussed later on).

There are many other solutions that provide managing secrets like Vault or Berglas. In the case of Vault, a team has to set up and manage the whole infrastructure. This means more budget and more time for managing something that could be done for you. Berglas on the other hand provides secret management via Google Cloud KMS. If you're already working with Berglas, you can easily integrate it with Google Secret Manager. When starting up a new project however, you should just implement Google Secret Manager from the get-go and omit Berglas altogether, as adding Berglas to the integration provides no additional benefits.

It's flexible

How you integrate Google Secret Manager in your application is totally up to you. You could see it as a fully decoupled service where you manage everything yourself. Another approach is to integrate it with your deploy pipeline, where secret versions could be created when building and deploying the application.

A secret could be a text string or a binary blob, so this means it could hold the value for one secret or a whole JSON file that contains all secrets for the application. How many secrets you’ll use will depend on which permissions are needed and your personal preference.

Secure access to your secrets

Google Secret Manager uses the principle of least privilege. By default, only project owners have permission to access any secret. Access control is granted through Cloud IAM where you are able to grant read & admin access to other members and make sure they have the permissions that they actually need. Permissions can be granted to individual secrets too and can be disabled easily.

A big advantage of keeping your secrets in Google Cloud is that not everyone has access to them. First you need access to Google Cloud console and on top of that you need permissions for viewing or managing secrets. Secrets could hold information that gives someone access to a service, database, ... and those could hold a lot of sensitive information. When developing an application that works with medical data, you don't want everyone from your organization to have access to it. Google Cloud and the Secret Manager helps you to prevent that! So this is a big thumbs up!

And another big plus is that audit logging comes for free when Cloud Audit Logging is enabled. For every interaction with the Secret Manager, an audit entry will be created. This could help you to identify abnormal access and potential security leaks.

Why encrypted secrets are so important

A common method for injecting configuration and secrets is the use of environment variables. Almost every application, service or platform is able to read an environment variable. This is a simple and straightforward approach, but it has one major drawback: the secrets exist in the environment in plaintext.

Any other process, library, dependency in your environment has access to the environment variables. A malicious library could result in a major security leak. Environment variables are great to store configuration settings, but not secrets!

With the help of Google's Secret Manager, our secrets are stored in a central place, are encrypted by default and are only accessible if you have the right permissions. The encrypted part here is one of the most important ones: secrets are protected when at rest and also when they are in transit by moving them to and from Google Cloud.

When at rest, Secret Manager supports both Google-managed encryption keys (GMEKs) and Customer-managed encryption keys (CMEKs).

Versioning

The data of each secret is immutable, so you're forced to create a new version any time the content of secret changes. From within your application, you can access a secret from a specific version (e.g. 12) or get the latest version with the 'latest' alias. Google advises to use the latest version only in development and staging and use a specific version for production.

“Production deployments should always be pinned to a specific secret version. Updating a secret should be treated in the same way as deploying a new version of the application.”

A major plus about versioned secrets is that it supports a gradual roll out of a new version of your application or that you’re able to do a rollback when something goes wrong.

Easy peasy integration

When Google announced his Secret Manager we created a proof of concept (POC) to see how much effort it is to integrate their new service. For this POC we used Cloud Run, Fastify as our Node.js framework and Node Config to handle configuration.

We extended the configuration with our secret (e.g. ORM Config) that is stored in the Google Secret Manager. First, we created a script that communicates with Google Secret Manager. You probably wonder where the credentials are to access the service? Access is done with Cloud IAM where Cloud Run has the permissions to access secrets.
‍

‍

Now the only thing that is left to do is create a ‘production.js’ file in the config directory so our 'orm_config' secret will be added to the existing configuration. Here we chose to use one secret to hold all the ORM configuration.
‍

‍

That's all! Wow, that wasn't hard right? Now in order to let this work with Fastify, there is one additional thing that should happen. When starting our Fastify server we need to resolve the config first so we are sure it's loaded.
‍

‍

Beside Node.js Google provided a library for many other programming languages like GO, .NET Core, PHP, ...

Conclusion

When Google announced their Secret Manager we created a POC first to test this new service. We were really surprised how smooth the integration went and certainly when you compare it with alternatives such as Berglas or Vault. It integrates really well with all other services (Cloud Run, Kubernetes, …) we use within Google Cloud, so we can only advise using Google Cloud Secret Manager!

If you're not using Google Cloud, there are other secret managers out there like AWS Secret Manager, Vault, ... It's just so important that you don't store your sensitive information in plain text! Using a Secret Manager will prevent access to your secrets by any installed package and these secrets are not visible to anyone that has access to the application’s infrastructure.