Jan 12, 2022

Kubernetes Secrets Best Practices

Every day more and more companies are moving their applications from Monoliths to Microservices. In many scenarios, it is required to decouple sensitive information or credentials from the application itself - essentially passing credentials during "runtime". It could be things like -

• Database user and passwords.
• SSH/API Keys and tokens to access external services
• TLS/SSL Certificates for secure communication.
• Auxillary services credentials such as RabbitMQ, Redis, etc.

In the rest of the post, let's refer to all the above as "Secrets". ?

Kubernetes "In-build" Secrets Support

Storing as Environment Variables(Skippable)

In Kubernetes, a very crude way to perform this is by providing this information as Environment Variables. Example -

env:
- name: database__client
  value: mysql
- name: database__connection__user
  value: root
- name: database__connection__password
  value: password
- name: database__connection__host
  value: limosyn-com-mysql
- name: database__connection__database
  value: mysql

However, this raises various security concerns, such as -

1. These passwords will be stored in Plaintext hence they'll be visible to anyone with cluster access.
2. These have a nasty habit of getting recorded in Source Control like git.
3. No Rotation Policy / Auto-Resets for passwords.
4. A Single change will have to be duplicated in all the applications which are using these credentials.

Storing via Opaque Secrets API (Skippable)

Kubernetes also supports storing SIs as Base64 values. These are a step-up from storing as Plain text but still suffer from a multitude of flaws. They are just configMaps specifically to store secrets. From Kubernetes Official website -

Creating a Kubernetes Secret

VIA kubectl CLI

# Syntax
kubectl create secret generic NAME [--type=string] [--from-file=[key=]source]
[--from-literal=key1=value1] [--dry-run=server|client|none] [options]


# Example Usage
kubectl create secret generic SecretName --from-literal=key1=value --from-literal=key2=value2You must be connected to the cluster before executing these commands

VIA Declarative Way

You can create a file - secret.yaml containing the secrets then execute - kubectl apply -f secret.yaml.

apiVersion: v1
kind: Secret
metadata:
  name: secretName
data:
  key1: value1
  key2: value2
immutable: true

A good practice, in this case, is to add a immutable: true field that prevents someone from accidental updates that could cause application outages. Also, this removes load on Kube-apiserver by closing watches for secrets marked immutable.

Consuming a Kubernetes Secret

By Mounting them as a Volume

...
spec:
  volumes:
  - name: secret-volume
    secret:
      secretName: key-secret
  containers:
  - name: test-container
    image: busybox:latest
    volumeMounts:
    - name: secret-volume
      readOnly: true
      mountPath: "/etc/secret-volume"
...

By loading them as Environment Variables

...
containers:
- name: mycontainer
  image: redis
  env:
  - name: SECRET_USERNAME
    valueFrom:
      secretKeyRef:
        name: mysecret
        key: username
  - name: SECRET_PASSWORD
    valueFrom:
      secretKeyRef:
        name: mysecret
        key: password
...

Secrets Management Best Practices

Not exactly an exhaustive list, but below are some patterns that you should remember while selecting your secret management -

1. A clear secrets access policy - you should have a well-defined policy as to which containers and people are going to have access to the secrets.
2. Having a centralized secrets store makes it easier to maintain, audit, and access control the full Kubernetes security landscape.
3. In case of breaches, it should help in identifying bad actors.
4. Should support RBAC - Role-based access control.
5. At last but not the least - support rotation and changing of secrets.

In the rest of this post, I'll discuss the various third-party tools that more or less fulfill the above salient features.

Third-Party Secret Management tools

There are both paid as well free tools to manage your secrets keeping in mind the above discussed best practices.

Cloud Provider tools

These management systems are provided by various cloud providers. The choice depends on the Cloud you are using as they don't support interplatform applications.

These Cloud Native Systems check most of our Best Practices requirements and provide great performance. But they are paid and you get vendor locked.

• Cloud KMS Offered by Google Cloud Platform
• Azure Key Vault - Offered by Azure Cloud Services
• AWS Secret Manager - Offered by Amazon Web Services

OpenSource tools

HashiCorp Vault

So far the best option in my opinion. It is the most widely used, popular, and feature-rich open-source secrets manager. It offers more fine-grained control than the cloud Provider tools and is completely FREE!

Some advantages offered by HashiCorp Vault -

1. It is cloud-agnostic and can be deployed on any cloud platform. It works perfectly with EKS, GKE, and on-prem clusters.
2. It supports dynamically generated users and passwords for databases. This shields your organization from leaked credentials.
3. Has multiple interfacing mediums - API, UI, and CLI.
4. It can be used to encrypt application data and generate PKI Certificates

The main downside is that it's harder to set up and configure. However, this can be alleviated using the built-in automation via vault CLI and various open-source helm scripts. Also, secrets are stored in a File System like structure which is not always intuitive.

You can find the official Vault Image on Dockerhub.

Sealed Secrets

Bitnami's Secrets Controller is an excellent tool to store secrets safely in public or private Git repositories. The sealed secrets can be decrypted only by the controller running in your cluster and no one else can access the secret, not even the author!

This can be used to create secure GitOps based workflows.

Sealed Secrets is comprised of two components:

• A cluster-side Kubernetes controller/operator
• A client-side utility called kubeseal

kubeseal uses asymmetric cryptography to achieve encryption. Only the controller has the private key for decryption.

Finishing Up...

So in this post, we discussed the requirements of a  good Secret Management System(SMS). The Native Secrets API provided by Kubernetes is great for College projects but definitely should be avoided by companies.

If you have a small to medium-scale application and already have your technology stack deployed on any of the cloud platforms, it makes sense to use the SMSs provided by them.

However, if you have an on-prem deployment or do not wish to get vendor locked, you can self deploy any of the open-source SMSs.

If you need version control and wish to maintain all configurations in a single place, go with Bitnami's sealed secrets. If not, go with HashiCorp Vault.

That's it for this post... Good Bye! Adios ✌?