This is part three of a five-part series on Kubernetes Fundamentals. Check back for new parts in the following weeks.
Containerized applications running in Kubernetes almost always need access to external resources that usually require secrets, passwords, keys, or tokens to gain access. Kubernetes Secrets lets you securely store these items, removing the need to store them in Pod definitions or container images.
In this post, we’ll discuss the various ways to create and use secrets in Kubernetes, with an aim to help you select the best approach for your environment.
Use cases for Kubernetes Secrets
Kubernetes secrets are designed to manage sensitive information, and they have various use cases to enhance the security and functionality of your applications. Here are some common scenarios where Kubernetes secrets are particularly useful:
- Storing API keys: Store third-party API keys securely and inject them into your application at runtime.
- Database credentials: Keep database usernames and passwords secure and automatically populate them in your application's configuration.
- SSL/TLS certificates: Store SSL/TLS certificates and keys to enable HTTPS for your applications.
- SSH keys: Securely manage SSH keys for accessing other services or Git repositories.
- OAuth tokens: Store OAuth tokens to allow your application to authenticate with external services.
- Service account tokens: Use Secrets to store service account tokens for internal communication between Kubernetes services.
- Encryption keys: Store encryption keys that your application uses to encrypt/decrypt data.
- Configuration files: Some applications require configuration files that may contain sensitive information. Secrets can be used to manage these files.
- Environment-specific configurations: Store environment-specific configurations like URLs for development, staging, and production environments.
- Docker registry credentials: Store credentials for private Docker registries to pull images securely.
- Cluster credentials: Store credentials required for cluster components to talk to the API server.
- Filesystem paths: Use Secrets to store paths to sensitive files or directories that your application needs to access.
- Seed data: Store initial data for applications like initial admin usernames and passwords.
By using Kubernetes secrets, you can keep sensitive information out of your source code and configuration files, thereby enhancing the security posture of your applications.
Types of Kubernetes Secrets
Kubernetes secrets come in various types to handle different types of sensitive data. Here are some of the most commonly used types:
- Opaque secrets: General-purpose secrets for storing arbitrary data.
- Docker registry secrets (kubernetes.io/dockerconfigjson): Stores credentials for private Docker registries.
- Basic authentication secrets (kubernetes.io/basic-auth): Holds credentials like username and password for basic HTTP authentication.
- SSH authentication secrets (kubernetes.io/ssh-auth): Stores SSH private keys.
- TLS secrets (kubernetes.io/tls): Holds TLS certificates and keys.
- Service account tokens (kubernetes.io/service-account-token): Contains tokens for interacting with the Kubernetes API.
- Bootstrap token secrets (bootstrap.kubernetes.io/token): Used for the bootstrap process of new cluster nodes.
- Ceph storage secrets (Custom): Stores the Ceph client admin key.
- Cloud provider secrets (Custom): Holds credentials for cloud provider APIs.
- Custom or user-defined secrets: User-defined types for specific needs.
Each type serves a specific purpose and helps in securely managing different kinds of sensitive data.
How to create Kubernetes Secrets
Kubernetes Secrets offers three methods to create and store secrets:
- Via the command line
- In a configuration file
- With a generator
Let’s take a look at creating some secrets with these methods.
Creating Kubernetes Secrets from the command line
You can create a secret via the Kubernetes administrator command line tool,
kubectl. This tool allows you to use files or pass in literal strings from your local machine, package them into secrets, and create objects on the cluster server using an API. It’s important to note that secret objects must be in the form of a DNS subdomain name.
For username and password secrets, use this command line pattern:
kubectl create secret generic <secret-object-name> <flags>
For secrets using TLS from a given public/private key pair, use this command line pattern:
kubectl create secret tls <secret-object-name> --cert=<cert-path> --key=<key-file-path>
You can also create a generic secret using a username and password combination for a database. This example applies the
literal flag to specify the username and password at the command prompt:
kubectl create secret generic sample-db-secret --from-literal=username=admin --from-literal=password=’7f3,F9D^LJz37]!W’
The command creates a new secret called sample-db-secret, with a username value of admin and a password value of
7f3,F9D^LJz37]!W. It is worth noting that strong, complex passwords often have characters that need to be escaped. To avoid this, you can put all your usernames and passwords in text files and use the following flags:
kubectl create secret generic sample-db-secret --from-file=username.txt --from-file=password.txt
This command drops the username and password keys as it provides the name of a file containing this information. You can add this back into the
--from-file switch the same way as the
--from-literal switch if the key is different than the file name.
kubectl create secret generic sample-db-secret --from-file=username=123.txt --from-file=password=xyz.txt
Setting Kubernetes Secrets in a configuration file
Another option is to create your secret using a JSON or YAML configuration file. A secret created in a configuration file has two data maps:
stringData. The former requires your values to be base64-encoded, whereas the latter allows you to provide values as unencoded strings.
The following template is used for secrets in YAML files:
apiVersion: v1 kind: Secret metadata: name: <secret name> type: Opaque data: <key>: <base64 Value> stringData: <key>: <string value>
You apply the template using the
kubectl apply -f ./<filename>.yaml command. As an example, here is a YAML file for an application that requires a number of secret values:
apiVersion: v1 kind: Secret metadata: name: my-example-app type: Opaque data: app-user: YWRtaW5pc3RyYXRvcg== app-password: cGFzc3dvcmQ= stringData: Dbconnection: Server=tcp:myserver.database.net,1433;Database=myDB;User ID=mylogin@myserver;Password=myPassword;Trusted_Connection=False;Encrypt=True; config.yaml: |- LogLevel: Warning API_TOKEN: NcNIMcMYMAMg.MGwjPnPfEBgqMl8Q API_URI: https://www.myapp.com/api
The above YAML file contains a number of values, including:
- The secret name (my-example-app)
config.yamlfile with data
This is a good way for packaging up many secrets and potentially sensitive configuration information into a single configuration file.
Creating Kubernetes Secrets with a generator
The third option for creating secrets is to use Kustomize, a standalone tool for customizing Kubernetes objects using a configuration file called
Kustomize allows you to generate secrets in the fashion similar to the command line by specifying secrets in files (with a key/value pair on each line), or as literals within the configuration file.
For secrets, the following structure is used within a kustomization.yaml file:
secretGenerator: name: <secret-name> files: <filename> literals: <key>=<value>
When you have created the kustomization.yaml file and included all the linked files in a directory, you can use the kubectl kustomize <directory> command, then apply the configuration using the kubectl apply -k <directory> command.
The following example
kustomization.yaml file creates a secret with two literal key/values (API_TOKEN and API_URI), as well as a config.yaml file:
secretGenerator: name: example-app-secrets files: passwords.txt literals: API_TOKEN: NcNIMcMYMAMg.MGwjPnPfEBgqMl8Q API_URI: https://www.myapp.com/api
config.yaml file referenced in the above example could be the config file for an application, for instance.
Which method for creating Kubernetes Secrets is the best?
Each of the methods we’ve discussed is “the best” under specific circumstances.
The command line is most useful when you have one or two secrets you want to add to a Pod—for instance a username and password—and you have them in a local file or want to pass them in as literals.
Configuration files are great when handling a handful of secrets that you want to include in a Pod all at one time.
The Kustomize configuration file is the preferred option when you have one or more configuration files and secrets you want to deploy to multiple Pods.
Once you have created your secrets, you can access them in two main ways:
- Via files (volume mounted)
- Via environment variables
The first option is similar to accessing configuration files as part of the application process. The second option loads the secrets as environment variables for the application to access. We are going to explore both methods.
Accessing volume mounted Kubernetes Secrets
To access secrets loaded in a volume, first you need to add the secret to the Pod under
spec.volumes.secret.secretName. You then add a volume to each container
under spec.containers.volumeMounts, where the name of the volume is the same as that of the secret, and where readOnly is set to
There are also a number of additional options you can specify. Let’s have a look at an example Pod with a single container:
apiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - name: myapp image: ubuntu volumeMounts: - name: secrets mountPath: "/etc/secrets” readOnly: true volumes: - name: secrets secret: secretName: mysecret defaultMode: 0400 items: - key: username path: my-username
This configuration file specifies a single Pod (
mypod) with a single container (
myapp). In the volumes section for the Pod, you have a volume named secrets, which is shared by all containers. This volume is of type secret, and it loads the secret called mysecret. It loads the volume using the Unix file permissions
0400, which gives read access to the owner (
root), and no access to other users.
The secret also contains the items list, which casts only specific secret keys (in this case,
username) and an appended path (
my-username). In your container (
myapp), you map the volume in volumeMounts (name is
secrets) to the mountPath as read only.
Because you’re casting the username to my-username, the directory
/etc/secrets in the container will look something like this:
lrwxrwxrwx 1 root root 2 September 20 19:18 my-username -> ..data/username
Because you’re casting a single value and changing the path, the key has changed. If you follow symlink, you will see that the permissions are set correctly:
-r-------- 1 root root 2 September 20 19:18 username
All the files that reside on the secret volume will contain the base64-decoded values of the secrets.
The advantage of loading your secrets into a volume is that, when the secrets are updated or modified, the volume is eventually updated as well, allowing your applications to re-read the secrets. It also makes parsing secret files (such as sensitive configuration files), as well as referencing multiple secrets, a lot easier.
How to use 'secretKeyRef' to set environmental variables
You can project your secrets into a container using environment variables. To do this, you add an environment variable for every secret key you wish to add using
env.valueFrom.secretKeyRef. Let’s have a look at an example Pod specification:
apiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - name: myapp image: ubuntu env: - name: USERNAME valueFrom: secretKeyRef: name: mysecret key: username - name: PASSWORD valueFrom: secretKeyRef: name: mysecret key: password
This configuration file passes two keys (
password) from the mysecret secret to the container as environment variables. These values are the base64-decoded values of the secrets.
If you logged into the container and ran the echo
$USERNAME command, you would get the decoded value of the
username secret (for example, “admin”).
One of the biggest advantages of casting secrets this way is that you can be very specific with the secret value. Besides, for some applications, reading environment variables is easier than parsing configuration files.
Best practices for Kubernetes Secrets management
Version control: Keep your Kubernetes manifests in version control systems like Git.
Documentation: Document your cluster configurations and deployments.
RBAC: Use Role-Based Access Control to limit permissions.
Secrets management: Never store secrets in source code; use Kubernetes Secrets or third-party tools like HashiCorp Vault.
Network policies: Implement network policies to restrict communication between pods.
Pod security policies: Use pod security policies to restrict pod capabilities.
API throttling: Limit API requests to protect against DDoS attacks.
Resource limits: Set CPU and memory limits and requests for containers.
Quality of service: Use quality of service (QoS) classes to ensure resource availability.
Node affinity: Use node affinity and anti-affinity rules for better resource allocation.
Immutable infrastructure: Treat your containers as immutable; rebuild and redeploy instead of modifying.
Rolling updates: Use rolling updates for zero-downtime deployments.
Health checks: Implement liveness and readiness probes for self-healing applications.
Monitoring and logging
Monitoring tools: Use monitoring tools for real-time metrics.
Logging: Centralize logging using solutions like New Relic.
Alerting: Set up alerts for key performance and health metrics.
Backup and recovery
Data backup: Regularly back up your etcd data.
Disaster recovery plan: Have a disaster recovery plan and regularly test it.
Horizontal scaling: Use horizontal pod autoscalers for dynamic scaling based on metrics.
Cluster autoscaler: Use a cluster autoscaler to add or remove nodes as needed.
By adhering to these best practices, you can ensure a more secure, efficient, and manageable Kubernetes environment.
Alternatives to Kubernetes Secrets
While secret management in a Kubernetes cluster is relatively simple, fairly secure, and can meet most requirements, it does have some downsides. In particular, secrets use namespaces like Pods, so if secrets and Pods are in the same namespace, all Pods can read the secrets.
The other major downside is that keys are not rotated automatically. You need to manually rotate secrets.
To address these issues and provide a more centralized secret management, you can use an alternative configuration, such as:
- Integrating a cloud vendor secrets management tool, such as Hashicorp Vault or AWS Secrets Manager. These tools typically use Kubernetes Service Accounts to grant access to the vault for secrets and mutating webhooks to mount the secrets into the Pod.
- Integrate a cloud vendor Identity and Access Management (IAM) tool, such as AWS Identity and Access Manager. This type of integration uses a method similar to OpenID Connect for web applications, which allows Kubernetes to utilize tokens from a Secure Token Service.
- Run a third party secrets manager, such as Conjur loaded into Pods as a sidecar.
Secure storage of secrets is critical to running containers in Kubernetes because almost all applications require access to external resources—databases, services, and so on. Using Kubernetes Secrets allows you to manage sensitive application information across your cluster, minimizing the risks of maintaining secrets in a non-centralized fashion.
Ready for a deep dive into Kubernetes monitoring? Check out A Complete Introduction to Monitoring Kubernetes with New Relic.
New Relic is a proud Platinum Sponsor at this year's KubeCon + CloudNativeCon Virtual Conference taking place on November 17-20. Stop by our virtual booth located in the Platinum Sponsor Hall and chat with one of our developer experts
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.