Understanding Kubernetes secrets in a Kubernetes environment

As we know, in today’s era, most applications are deployed using Kubernetes. So that applications can function properly, and the users can use the applications without any issues. The applications sometimes require access to external resources, services or databases for processing or storing data. One of the most efficient ways of accessing sensitive data from other services is the secret object of the Kubernetes environment.

Using the secrets management part, one can easily manage and distribute secret values or sensitive information across the Kubernetes cluster. The secrets include database passwords, OAuth Tokens, Image registry keys, SSH keys or API keys. Sometimes, secrets as environment variables are also passed in the configuration files.

What are Kubernetes secrets?

A Kubernetes secret is a container that contains sensitive information such as usernames, passwords, tokens, and keys. Secrets are generated by the system during app installation or by users when they need to store sensitive information and make it available to a pod. Secrets are also used as secure storage.

Passwords, tokens, or keys were previously part of a pod definition or container image. They might be inadvertently exposed during Kubernetes operations because they were simply a part of it. As a result, the most basic authentication and essential function of the Secret is to guarantee that sensitive information isn’t accidentally released while still allowing it to be accessed by the person who needs it.

Why are Kubernetes secrets important?

While working in a distributed cloud environment, all the applications deployed in the containers must remain transitory and isolated by not sharing the Kubernetes resources associated with them with other pods. This is particularly valid regarding public key certificates and other private resource pods required to access external resources. Applications, therefore, require a technique for externally querying their authentication mechanisms, independent of the application itself.

The solution for this follows the path of least privilege, offered by Kubernetes. To provide credentials to the application for access to external resources, Kubernetes Secrets function as independent objects that may be requested by the application Pod. Secrets can only be accessible by Pods if they are specifically included in a volume that has been mounted or when the Kubelet is pulling the image that will be used for the Pod.

Types of Secrets in Kubernetes

In this module, we will have a look at different built-in secret types in Kubernetes. Understanding all the secret types can help you manage secrets efficiently. Following are the various types of secrets in Kubernetes:

Opaque secrets

This is the default secret type in Kubernetes. This sort of Secret is thought to apply to all secrets whose configuration file lacks a type statement. The purpose of opaque secrets is to store any user data.

Service account token secrets

Tokens used to identify service accounts are kept in service account token secrets. Kubernetes automatically generates this Secret and associates it with the Pod when a pod is created, enabling safe access to the API. It is possible to stop this behaviour. This can be applied to the existing service account.

Docker config secrets

Valid Docker credentials are necessary to access a registry for Docker images. A serialised version of the traditional /.dockercfg format used for Docker command-line setup is stored in this kind of Secret. The. dockercfg key is base64-encoded in it.

Basic authentication secret

Used to keep the credentials necessary for fundamental authentication. The “username” and “password” keys must be present in the Secret’s “data” field when utilising this particular Secret type.SSH Authentication secrets

SSH Authentication secret

used to store the associated key information needed for SSH authentication. You must enter an’ssh-privatekey’ key-value pair as the SSH credential to utilise when utilising this Secret type in the ‘data’ (or’stringData’) field.

TLS secrets

for keeping a certificate and the public key certificate name that goes with it, usually used for TLS. Although it can be utilised with other resources or directly by a workload, this data is primarily used with TLS termination of the Ingress resource. When utilising this kind of Secret, the data (or “stringData”) field of the Secret configuration must have both the “tls.key” and “tls.crt” keys.

Bootstrap token secrets

These tokens are used during the node bootstrap process. It keeps track of the signing tokens for well-known ConfigMaps.

Usage of Kubernetes secret

After creating the Secret, it needs to be referenced by a pod so that the Pod can use that Secret. There are various ways in which a pod can use or access secrets loaded or created. Following are some of how one can use Kubernetes secret:

As a file, mount the Secret in a volume accessible to as many containers as in the Pod.

The Secret should be imported into a container as an environment variable.

Use the imagePullSecrets field and the Kubelet.

How to create a secret in Kubernetes?

One can create a secret differently, like using the command line or creating a secret file. This module will discuss some of the most used, efficient and helpful ways to access secrets.

Using kubectl for creating secrets

To create a secret using the kubectl command line tool, we first need to create files containing the secret data. Let us make it more clear with an example.

In this example, we will store the username and the password in two files named username.txt and password.txt, respectively.

echo -n 'admin' > ./username.txt
echo -n '1f2d1e2e67df' > ./password.txt

You can use this command to add whitespace at the end of a text file. The -n option instructs echo not to append a new line after the string. Because it is also regarded as a character, it would be encoded with the other characters, resulting in a different decoded value.

Now that you have used all the files from the previous step, use kubectl to create a secret. To do so, utilise the generic subcommand to create Opaque secrets using the opaque data. In addition, include the option –from-file for each file:

kubectl create secret generic db-user-pass 
  --from-file=./username.txt 
  --from-file=./password.txt

You should receive an output saying the secret/secret name secret created like this:-

secret/db-user-pass created

Creating Kubernetes secret object using config file

To create Kubernetes secrets, one can also use the secret configuration file. The configuration files are JSON or YAML files containing the secret metadata. There are two significant fields in secret objects. Those maps are:-

• data:- This stores the arbitrary data in base64 encoded format.
• stringData:- This lets you supply Secret info as unencoded strings.

Data and stringData’s keys must comprise the alphanumeric characters -, _, or.

Let us make it more apparent with the help of an example. Here, there will be two strings in the same Secret. One will be the username, and the other will be the password. We need to encode both strings in base64 format. One can use the below-mentioned commands:-

echo -n 'admin' | base64

The result should look something like this:-

YWRtaW4=

Now, for encoding the password in base64 format:-

echo -n '1f2d1e2e67df' | base64

For this, the result should look something like this:-

MWYyZDFlMmU2N2Rm

Now, one can create Kubernetes secrets using the data and the stringData. The final secret object YAML file should look like this:-

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

Instead of using the kubectl create secret command now, one should use the following command to create a secrete using the above YAML file:-

kubectl create -f mysecret.yaml

The above command creates a secret using the mentioned YAML file for a secret object.

Note:- If we have to create secrets or other objects using a YAML manifest file, we use a declarative command specifying the local file path for creating the object. The above command was declarative because we were using a YAML file.

But while creating Kubernetes secrets using the kubectl create secret generic command, we were not specifying the YAML file, so that was the imperative command.

Creating Secret using Kustomize

By declaring a secretGenerator in a kustomization.yaml file that references other existing files, you may also create a Secret object. The following customisation file, for instance, refers to the./username.txt and./password.txt files:-

secretGenerator:
- name: db-user-pass
  files:
  - username.txt
  - password.txt

After that, one can use the apply command specifying the kustomization.yaml file path to create a secret. That will look like this:-

kubectl apply -k .

The result will look something like this:-

secret/db-user-pass-96mffmfh4k created

How to edit a secret in Kubernetes?

To edit an existing secret in Kubernetes, the below-mentioned command can be used:-

kubectl edit secrets mysecret

In the above example, we have created a new secret named mysecret. Therefore, in the edit command, we have mentioned mysecret name. One can edit all the existing files of secrets using the command.

Once you hit the enter button after entering the command, the default editor will open the YAML file, and there, one can change the secret key-value pairs. The output of the command will be like this:-

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: { ... }
  creationTimestamp: 2016-01-22T18:41:56Z
  name: mysecret
  namespace: default
  resourceVersion: "164619"
  uid: cfee02d6-c137-11e5-8d73-42010af00002
type: Opaque

How to use secrets?

One can use secrets in various ways, like mounting secret volume or using a container inside a pod as an environment variable. Without being directly revealed to the Pod, secrets can potentially be exploited by other system components. Secrets, for instance, may contain credentials that other system components can use to communicate with external systems on your behalf.

Usage of secrets as environment variables

If you want to use a secret in a Pod’s environment variable, you should do the following:-

• First, you need to create a secret, or you can also use an existing secret.
• Change your Pod definition in each container where you want to use an environment variable to store the value of a secret key. The Secret’s name and key should be filled up in env by the environment variable that uses the secret key env[].valueFrom.secretKeyRef.
• Environment variables can be set to store data and influence the program’s behaviour. To make the program look for values in these environment variables, change your image and/or command line.

Let us make it more clear to understand with the help of an example:-

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: username
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: mysecret
            key: password
  restartPolicy: Never

Usage of Immutable Secret metadata

In Kubernetes, you can set individual secrets to pod metadata immutable. The following benefits come from not allowing modifications to data for clusters that heavily utilise Secrets (at least tens of thousands of distinct Secret to Pod mounts):-

• Guards against unintentional (or undesired) changes that might result in the interruption of applications.
• Increases the efficiency of your cluster by minimising the burden on the kube-apiserver and by shutting off watches for secrets that have been declared immutable.

ImmutableEphemeralVolumes feature gate controls this feature. Since the release of Kubernetes Version v1.19, this comes by default. To create an immutable secret volume, all you need to do is to set the immutable field to true.

You can understand this more with the help of the below-mentioned example:-

apiVersion: v1
kind: Secret
metadata:
  ...
data:
  ...
immutable: true

How are Kubernetes secrets stored?

Kubernetes Secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, as can anyone with access to etcd. Additionally, anyone authorised to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access, such as the ability to create a Deployment.

Conclusion

Kubernetes Secrets provides a way to store and manage sensitive data without including that data in your application code or on disk. This can help reduce the risk of confidential information being exposed. Kubernetes and applications that run in your cluster can also take additional precautions with Secrets, such as avoiding secret writing data to nonvolatile storage.

Consider validating the security of your Kubernetes cluster and its implementation using our Kubernetes penetration testing service.