Background

I want to authenticate to Vault and then generate dynamic Kubernetes service accounts and associated roles.

Exploration

  • I explored the use of PKI secrets engine first, where I thought that I could create an intermediate CA from the Kubernetes root CA.  
    • I would be able to issue user certificates to allow for user logins easily through Vault.
    • I would be creating Kubernetes groups which would map to role bindings.
    • However quickly realized it is more complex than necessary and very brittle.
  • I explored the use of JWT/ODIC method, it seemed interesting at first but then quickly realized its increased complexity and unnecessary abstractions.
  • The Kubernetes secret engine in Vault was most straight forward, though its examples and usage documentation wasn't very good.

Pre-requisites

  • A working Kubernetes cluster.
  • Two DNS names are needed.  This is an important implementation detail.
    • A DNS name that leads direct to the Kubernetes API server, without any SSL termination mechanism in between.  This is needed between the Vault server and the Kubernetes API server.
    • A DNS name exposed to end users, where SSL termination can be fine.  This is preferable for clients, and works well with kubectl.  I should be able to use kubectl across the Internet if I have my SSL termination setup correctly.

Steps

Kubernetes setup for Vault


Steps
1

We need to create a long living service account on Kubernetes.  This particular service account will be used by Vault to then create the dynamic Kubernetes service accounts.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-trust
  namespace: kube-system

In the above, I made this service account live in the kube-system namespace.

2

We need to then create a secret that will hold the service accounts token.

apiVersion: v1
kind: Secret
metadata:
  name: vault-trust-secret
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: vault-trust
type: kubernetes.io/service-account-token

I found helpful details on the token setup here:  https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#manually-create-an-api-token-for-a-serviceaccount 

3

We need to create a Kubernetes role binding for this vault-trust service account.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vault-trust-cluster-role
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["serviceaccounts", "serviceaccounts/token"]
  verbs: ["create", "update", "delete"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["rolebindings", "clusterrolebindings"]
  verbs: ["create", "update", "delete"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["roles", "clusterroles"]
  verbs: ["bind", "escalate", "create", "update", "delete"]

The above role example I found here: https://developer.hashicorp.com/vault/docs/secrets/kubernetes#setup

The important detail here is that this role can allow for Vault to create dynamic roles that can limit the scopes to namespaces.

For instance, I could create a dynamic role that is equivalent to the default "admin" Kubernetes role but then further limit its functionality to a specific namespace.

4

Then we bind the ClusterRole to the service account using a Kubernetes ClusterRoleBinding.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-trust-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vault-trust-cluster-role
subjects:
- kind: ServiceAccount
  name: vault-trust
  namespace: kube-system
5

To apply the above Kubernetes manifest its simple as:  kubectl apply -f <manifest.yaml> 

6

The last step is to pull out this Vault trust/admin token.

Example
[email protected]:~# kubectl get secrets -n kube-system
NAME                 TYPE                                  DATA   AGE
vault-trust-secret   kubernetes.io/service-account-token   3      148m

[email protected]:~# kubectl get secrets -n kube-system -o yaml vault-trust-secret
apiVersion: v1
data:
  ca.crt:  ...
  namespace: a3ViZS1zeXN0ZW0=
  token: ...
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{"kubernetes.io/service-account.name":"vault-trust"},"name":"vault-trust-secret","namespace":"kube-system"},"type":"kubernetes.io/service-account-token"}
    kubernetes.io/service-account.name: vault-trust
    kubernetes.io/service-account.uid: f6c5af17-be29-43db-95e6-3a2df065ea02
  creationTimestamp: "2022-12-18T17:05:44Z"
  name: vault-trust-secret
  namespace: kube-system
  resourceVersion: "1082656"
  uid: 8aaa1b8f-78c5-43ae-b587-23d2147c8c39
type: kubernetes.io/service-account-token

In the above secret manifest, I placed it in the kube-system namespace.  Retrieve the base64 value of the data.token field.

7

Remember that Kubernetes secrets are just base64 encoded, so perform a base64 decode on the value to expose its literal value.

Example decode base64
[email protected]:~# echo aGVsbG93b3JsZAo= | base64 -d
helloworld

Vault setup

Its best to use the Vault CLI for the next set of steps.


Steps
1Perform a vault login .  You will need root access since we will create a new secrets engine.
2

Enable the Kubernetes secret engine

vault secrets enable kubernetes
3

Determine the needed Kubernetes configuration.  The API docs mention the needed variables:  https://developer.hashicorp.com/vault/api-docs/secret/kubernetes#write-configuration

{
  "kubernetes_host": "https://homelab-k8s-cluster:6443",
  "service_account_jwt": "..."
}


  1. The kubernetes_host URL needs to be the direct endpoint to the Kubernetes API server.  There can't be any middle-man LB in between, or you will encounter some x509: certificate signed by unknown authority.
  2. The service_account_jwt is the token that Kubernetes generated for the service account created prior.

The above contents can be placed in a JSON file.

4

Write the Kubernetes configuration.

cat kubernetes_config.json | vault write -f kubernetes/config -
5

Writing a dynamic role in Vault.  Below I'm creating a role called admin-role which will be bound to the Kubernetes role called admin and scoped the allowed namespaces.

vault write -f kubernetes/roles/admin-role - <<eof
{
  "allowed_kubernetes_namespaces": "*",
  "kubernetes_role_type": "ClusterRole",
  "kubernetes_role_name": "admin"
}
eof

There are some pre-existing ClusterRoles that could be useful when creating generic dynamic roles

NAME
admin
cluster-admin
edit
view
6

Generating a dynamic service account.  

A Vault write operation is needed, and this endpoint can also allow for further namespace limitation and TTL limits.

[email protected]:~# vault write -f kubernetes/creds/admin-role - <<eof
{
  "kubernetes_namespace": "default",
  "ttl": "8h"
}
eof
Key                          Value
---                          -----
lease_id                     kubernetes/creds/admin-role/OB5ZgA9fsQ7JT2SKvC76sVe3
lease_duration               8h
lease_renewable              false
service_account_name         v-root-admin-ro-1671393013-...
service_account_namespace    default
service_account_token        ...

The important returns value are the service_account_name and service_account_token .

Setup kubectl to use the service account


Steps
1

Create the Kubernetes context.

# create a new context
kubectl config set-context homelab-k8s-cluster --cluster=homelab-k8s-cluster --user=v-root-admin-ro-1671393013-...

# use the newly created context
kubectl config use-context homelab-k8s-cluster

The user is the service_account_name given by Vault.

2

Setup the endpoint of the cluster.

# setup the cluster endpoint
kubectl config set-cluster homelab-k8s-cluster --server=https://k8s.tenzin.io
3

Setup the credentials to the cluster.

# link the username to the token
kubectl config set-credentials v-root-admin-ro-1671393013-... --token=...

The username is the service account name and the token value is the service account token.

4

After setup of the Kubernetes config.  You should be able to use the kubectl  get and other operations.

Example
# kubectl get all                                                                                                                                                                                                                       
NAME                            READY   STATUS    RESTARTS   AGE
pod/my-nginx-778975588f-n2v8m   1/1     Running   0          3d17h

NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
service/kubernetes   ClusterIP      10.11.0.1      <none>           443/TCP        6d17h
service/my-nginx     LoadBalancer   10.11.46.172   192.168.200.50   80:30050/TCP   3d17h

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-nginx   1/1     1            1           3d17h

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/my-nginx-778975588f   1         1         1       3d17h
  • No labels