Background
I want to authenticate to Vault and then generate dynamic Kubernetes service accounts and associated roles.
Exploration
- I explored the use of PKI secrets engine first, where I thought that I could create an intermediate CA from the Kubernetes root CA.
- I would be able to issue user certificates to allow for user logins easily through Vault.
- I would be creating Kubernetes groups which would map to role bindings.
- However quickly realized it is more complex than necessary and very brittle.
- I explored the use of JWT/ODIC method, it seemed interesting at first but then quickly realized its increased complexity and unnecessary abstractions.
- The Kubernetes secret engine in Vault was most straight forward, though its examples and usage documentation wasn't very good.
Pre-requisites
- A working Kubernetes cluster.
- Two DNS names are needed. This is an important implementation detail.
- A DNS name that leads direct to the Kubernetes API server, without any SSL termination mechanism in between. This is needed between the Vault server and the Kubernetes API server.
- A DNS name exposed to end users, where SSL termination can be fine. This is preferable for clients, and works well with kubectl. I should be able to use kubectl across the Internet if I have my SSL termination setup correctly.
Steps
Kubernetes setup for Vault
| Steps | |
|---|---|
| 1 | We need to create a long living service account on Kubernetes. This particular service account will be used by Vault to then create the dynamic Kubernetes service accounts. apiVersion: v1 kind: ServiceAccount metadata: name: vault-trust namespace: kube-system In the above, I made this service account live in the kube-system namespace. |
| 2 | We need to then create a secret that will hold the service accounts token. apiVersion: v1
kind: Secret
metadata:
name: vault-trust-secret
namespace: kube-system
annotations:
kubernetes.io/service-account.name: vault-trust
type: kubernetes.io/service-account-token
I found helpful details on the token setup here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#manually-create-an-api-token-for-a-serviceaccount |
| 3 | We need to create a Kubernetes role binding for this apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: vault-trust-cluster-role rules: - apiGroups: [""] resources: ["namespaces"] verbs: ["get"] - apiGroups: [""] resources: ["serviceaccounts", "serviceaccounts/token"] verbs: ["create", "update", "delete"] - apiGroups: ["rbac.authorization.k8s.io"] resources: ["rolebindings", "clusterrolebindings"] verbs: ["create", "update", "delete"] - apiGroups: ["rbac.authorization.k8s.io"] resources: ["roles", "clusterroles"] verbs: ["bind", "escalate", "create", "update", "delete"] The above role example I found here: https://developer.hashicorp.com/vault/docs/secrets/kubernetes#setup The important detail here is that this role can allow for Vault to create dynamic roles that can limit the scopes to namespaces. For instance, I could create a dynamic role that is equivalent to the default "admin" Kubernetes role but then further limit its functionality to a specific namespace. |
| 4 | Then we bind the ClusterRole to the service account using a Kubernetes ClusterRoleBinding. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: vault-trust-cluster-role-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: vault-trust-cluster-role subjects: - kind: ServiceAccount name: vault-trust namespace: kube-system |
| 5 | To apply the above Kubernetes manifest its simple as: |
| 6 | The last step is to pull out this Vault trust/admin token. Example [email protected]:~# kubectl get secrets -n kube-system NAME TYPE DATA AGE vault-trust-secret kubernetes.io/service-account-token 3 148m [email protected]:~# kubectl get secrets -n kube-system -o yaml vault-trust-secret apiVersion: v1 data: ca.crt: ... namespace: a3ViZS1zeXN0ZW0= token: ... kind: Secret metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{"kubernetes.io/service-account.name":"vault-trust"},"name":"vault-trust-secret","namespace":"kube-system"},"type":"kubernetes.io/service-account-token"} kubernetes.io/service-account.name: vault-trust kubernetes.io/service-account.uid: f6c5af17-be29-43db-95e6-3a2df065ea02 creationTimestamp: "2022-12-18T17:05:44Z" name: vault-trust-secret namespace: kube-system resourceVersion: "1082656" uid: 8aaa1b8f-78c5-43ae-b587-23d2147c8c39 type: kubernetes.io/service-account-token In the above secret manifest, I placed it in the kube-system namespace. Retrieve the base64 value of the |
| 7 | Remember that Kubernetes secrets are just base64 encoded, so perform a base64 decode on the value to expose its literal value. Example decode base64 [email protected]:~# echo aGVsbG93b3JsZAo= | base64 -d helloworld |
Vault setup
Its best to use the Vault CLI for the next set of steps.
| Steps | |
|---|---|
| 1 | Perform a vault login . You will need root access since we will create a new secrets engine. |
| 2 | Enable the Kubernetes secret engine vault secrets enable kubernetes |
| 3 | Determine the needed Kubernetes configuration. The API docs mention the needed variables: https://developer.hashicorp.com/vault/api-docs/secret/kubernetes#write-configuration {
"kubernetes_host": "https://homelab-k8s-cluster:6443",
"service_account_jwt": "..."
}
The above contents can be placed in a JSON file. |
| 4 | Write the Kubernetes configuration. cat kubernetes_config.json | vault write -f kubernetes/config - |
| 5 | Writing a dynamic role in Vault. Below I'm creating a role called vault write -f kubernetes/roles/admin-role - <<eof
{
"allowed_kubernetes_namespaces": "*",
"kubernetes_role_type": "ClusterRole",
"kubernetes_role_name": "admin"
}
eof
There are some pre-existing ClusterRoles that could be useful when creating generic dynamic roles NAME admin cluster-admin edit view |
| 6 | Generating a dynamic service account. A Vault write operation is needed, and this endpoint can also allow for further namespace limitation and TTL limits. [email protected]:~# vault write -f kubernetes/creds/admin-role - <<eof { "kubernetes_namespace": "default", "ttl": "8h" } eof Key Value --- ----- lease_id kubernetes/creds/admin-role/OB5ZgA9fsQ7JT2SKvC76sVe3 lease_duration 8h lease_renewable false service_account_name v-root-admin-ro-1671393013-... service_account_namespace default service_account_token ... The important returns value are the |
Setup kubectl to use the service account
| Steps | |
|---|---|
| 1 | Create the Kubernetes context. # create a new context kubectl config set-context homelab-k8s-cluster --cluster=homelab-k8s-cluster --user=v-root-admin-ro-1671393013-... # use the newly created context kubectl config use-context homelab-k8s-cluster The user is the |
| 2 | Setup the endpoint of the cluster. # setup the cluster endpoint kubectl config set-cluster homelab-k8s-cluster --server=https://k8s.tenzin.io |
| 3 | Setup the credentials to the cluster. # link the username to the token kubectl config set-credentials v-root-admin-ro-1671393013-... --token=... The username is the service account name and the token value is the service account token. |
| 4 | After setup of the Kubernetes config. You should be able to use the Example # kubectl get all NAME READY STATUS RESTARTS AGE pod/my-nginx-778975588f-n2v8m 1/1 Running 0 3d17h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.11.0.1 <none> 443/TCP 6d17h service/my-nginx LoadBalancer 10.11.46.172 192.168.200.50 80:30050/TCP 3d17h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/my-nginx 1/1 1 1 3d17h NAME DESIRED CURRENT READY AGE replicaset.apps/my-nginx-778975588f 1 1 1 3d17h |