I want to authenticate to Vault and then generate dynamic Kubernetes service accounts and associated roles.
- I explored the use of PKI secrets engine first, where I thought that I could create an intermediate CA from the Kubernetes root CA.
- I would be able to issue user certificates to allow for user logins easily through Vault.
- I would be creating Kubernetes groups which would map to role bindings.
- However quickly realized it is more complex than necessary and very brittle.
- I explored the use of JWT/ODIC method, it seemed interesting at first but then quickly realized its increased complexity and unnecessary abstractions.
- The Kubernetes secret engine in Vault was most straight forward, though its examples and usage documentation wasn't very good.
- A working Kubernetes cluster.
- Two DNS names are needed. This is an important implementation detail.
- A DNS name that leads direct to the Kubernetes API server, without any SSL termination mechanism in between. This is needed between the Vault server and the Kubernetes API server.
- A DNS name exposed to end users, where SSL termination can be fine. This is preferable for clients, and works well with kubectl. I should be able to use kubectl across the Internet if I have my SSL termination setup correctly.
Kubernetes setup for Vault
We need to create a long living service account on Kubernetes. This particular service account will be used by Vault to then create the dynamic Kubernetes service accounts.
In the above, I made this service account live in the kube-system namespace.
We need to then create a secret that will hold the service accounts token.
I found helpful details on the token setup here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#manually-create-an-api-token-for-a-serviceaccount
We need to create a Kubernetes role binding for this
The above role example I found here: https://developer.hashicorp.com/vault/docs/secrets/kubernetes#setup
The important detail here is that this role can allow for Vault to create dynamic roles that can limit the scopes to namespaces.
For instance, I could create a dynamic role that is equivalent to the default "admin" Kubernetes role but then further limit its functionality to a specific namespace.
Then we bind the ClusterRole to the service account using a Kubernetes ClusterRoleBinding.
To apply the above Kubernetes manifest its simple as:
The last step is to pull out this Vault trust/admin token.
In the above secret manifest, I placed it in the kube-system namespace. Retrieve the base64 value of the
Remember that Kubernetes secrets are just base64 encoded, so perform a base64 decode on the value to expose its literal value.
Its best to use the Vault CLI for the next set of steps.
|1||Perform a |
Enable the Kubernetes secret engine
Determine the needed Kubernetes configuration. The API docs mention the needed variables: https://developer.hashicorp.com/vault/api-docs/secret/kubernetes#write-configuration
The above contents can be placed in a JSON file.
Write the Kubernetes configuration.
Writing a dynamic role in Vault. Below I'm creating a role called
There are some pre-existing ClusterRoles that could be useful when creating generic dynamic roles
Generating a dynamic service account.
A Vault write operation is needed, and this endpoint can also allow for further namespace limitation and TTL limits.
The important returns value are the
Setup kubectl to use the service account
Create the Kubernetes context.
The user is the
Setup the endpoint of the cluster.
Setup the credentials to the cluster.
The username is the service account name and the token value is the service account token.
After setup of the Kubernetes config. You should be able to use the