Using HashiCorp Vault as Certificate Issuer on a Kubernetes Cluster

In this article, we will set up HashiCorp vault as a certificate issuer using the HashiCorp Vault PKI engine and cert-manager as the management of certificates. We will demonstrate how the certificate issuing and automatic renewal works when we deploy an example web application with ingress resources and TLS enabled. This can simplify the usual manual process of certificate requesting, signing, and renewal. The following figure is the high-level view of the implementation.

Installing HashiCorp Vault

To install HashiCorp Vault on a Kubernetes cluster, we will add the Vault chart to our Helm repository and retrieve the update from the remote repository. We then deploy Vault using Helm with the injector disabled.

helm repo add hashicorp https://helm.releases.hashicorp.com

helm repo update

helm install vault hashicorp/vault --set "injector.enabled=false"

After installation, the vault pod created will not be ready since the vault is not initialized and unsealed yet. We will have to initialize and unseal the vault to get the master key which gives access to the data stored in the vault by encryption/decryption.

kubectl get po

Initializing and unsealing the vault

When the vault is initiated, it will be in sealed mode, meaning it can access the storage layer but cannot decrypt the content. So, we will initialize the vault and distribute the unseal key using Shamir’s Secret algorithm with one key share and one key threshold. The resulting unseal key and initial root token is written to init-keys.json.

kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > init-keys.json

We can view the unseal key in init-keys.json using the following commands and save it in an environment variable. We then exec into vault-0 to unseal the vault running on it with VAULT_UNSEAL_KEY.

cat init-keys.json | jq -r ".unseal_keys_b64[]"

VAULT_UNSEAL_KEY=$(cat init-keys.json | jq -r ".unseal_keys_b64[]"

kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY

After the unsealing process is complete, we can check that the vault-0 pod appears in a ready state.

kubectl get po

Configuring the Vault PKI engine as a Certificate Authority

The init-keys.json also contains the initial root token we can use to log in to configure the vault. We will retrieve it and set it in an environment variable called VAULT_ROOT_TOKEN to log in to the vault and configure the PKI engine as a certificate authority.

cat init-keys.json | jq -r ".root_token"
VAULT_ROOT_TOKEN=$(cat init-keys.json | jq -r ".root_token")

After logging into the vault with the root token, we can enable the PKI engine, and configure ttl of the PKI engine to 8760 hours.

kubectl exec vault-0 -- vault login $VAULT_ROOT_TOKEN
kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh
vault secrets enable pki
vault secrets tune -max-lease-ttl=8760h pki

To configure the PKI engine as CA, we will generate a self-signed root certificate for signing certificates and set its ttl to 8760 hours, and write to a file called demo-root-ca.json.

vault write -format=json pki/root/generate/internal \
    common_name="Demo Root Certificate Authority" > /tmp/demo-root-ca.json
cat /tmp/demo-root-ca.json

vault write pki/root/generate/internal \
    common_name=example.com ttl=8760h

We will then configure the PKI engine certificate issuing and certificate revocation list (CRL) endpoints of the vault services in the default namespace.

vault write pki/config/urls \
    issuing_certificates="http://vault.default:8200/v1/pki/ca" \
    crl_distribution_points="http://vault.default:8200/v1/pki/crl"

According to the architecture shown at the beginning of the article, we need to create a role that enables the creation of the certificates under the condition for example.com domain with any subdomains, and a policy that defines finer-grained permissions.

vault write pki/roles/example-dot-com \
    key_type=any \
    allowed_domains=example.com \
    allow_subdomains=true \
    max_ttl=5m

Once we have the role created, we can proceed to make a policy named pki for the corresponding vault PKI role.

vault policy write pki - <<EOF
path "pki*" { capabilities = ["read", "list"] }
path "pki/roles/example-dot-com"   { capabilities = ["create", "update"] }
path "pki/sign/example-dot-com"    { capabilities = ["create", "update"] }
path "pki/issue/example-dot-com"   { capabilities = ["create"] }
EOF

Enable the Kubernetes authentication method

To simplify how applications interact with Vault, we will use Kubernetes auth method. Kubernetes auth method makes use of JWT associated with Kubernetes Service Account.

To configure the auth method, we will use the local token and CA certificate created when the vault pod is started, which is located on the default mount folder /var/run/secrets/kubernetes.io/serviceaccount/ . Vault will periodically re-read the files in this folder to support short-lived tokens.

When an application tries to interact with Vault, Vault uses this configuration to verify and retrieve the application’s identity with the Kubernetes API server and its TokenReview API.

vault auth enable kubernetes

vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

We will then create a Kubernetes authentication role named issuer that binds the pki the policy defined earlier with a Kubernetes service account name and namespaces.

vault write auth/kubernetes/role/issuer \
    bound_service_account_names=issuer \
    bound_service_account_namespaces=cert-manager,default \
    policies=pki \
    ttl=20m

exit

After the above configuration, the Vault with the PKI engine can act as CA and for the application (cert-manager) to interact with Vault and authenticate through Kubernetes auth method (TokenReviewer API).

Upon authentication, Vault will retrieve the policy defined for the role of the service account issuer. Vault can then issue certificates according to the permissions in the policy.

Installing cert-manager, configuring issuer, and creating a certificate

We will install Cert-Manager to interact with the vault PKI to issue certificates. After installation, we can check the custom resource definitions and pod created.

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
    cert-manager jetstack/cert-manager \
    --namespace cert-manager --create-namespace \
    --version v1.11.0 --set installCRDs=true
kubectl get crds
kubectl get po -n cert-manager

We will then create a service account called issuer and service account token as a Secret resource. This secret will be referenced in the cert-manager Issuer resource to authenticate with Vault via Kubernetes auth method when generating and issuing certificates.

kubectl create serviceaccount issuer
kubectl get sa
kubectl apply -f secret.yaml
kubectl get secrets
kubectl describe secret issuer-token
kubectl get secret issuer-token -o jsonpath={.data.token} | base64 -d

apiVersion: v1
kind: Secret
metadata:
  name: issuer-token
  annotations:
    kubernetes.io/service-account.name: issuer
type: kubernetes.io/service-account-token

We can now create a cert-manager Issuer resource that references the issuer token and role for authentication against Vault, Vault PKI certificate issuing endpoint, and Vault server URL, as in the following.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    server: http://vault.default:8200
    path: pki/sign/example-dot-com
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: issuer
        secretRef:
          name: issuer-token
          key: token

kubectl apply -f issuer.yaml
kubectl get issuer
kubectl describe issuer vault-issuer

Finally, we can create a cert-manger Certificate resource with a created secret containing a certificate that is issued by Vault. The Certificate resource is managed by the cert-manager.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: demo-example-com
  namespace: default
spec:
  secretName: example-com-tls
  issuerRef:
    name: vault-issuer
  commonName: demo.example.com
  dnsNames:
  - demo.example.com

kubectl apply -f certificate.yaml
kubectl get certificate
kubectl describe certificate demo-example-com

kubectl get secrets example-com-tls
kubectl describe secrets example-com-tls

Install ingress-nginx controller

To demonstrate applications deployed on the Kubernetes cluster to make use of Vault and cert-manager to issue and manage certificates, we will deploy a demo web application and enable TLS through an ingress resource. First, we will deploy an ingress-nginx controller to the Kubernetes cluster using Helm.

helm upgrade --install ingress-nginx ingress-nginx \
    --repo https://kubernetes.github.io/ingress-nginx \
    --namespace ingress-nginx --create-namespace

We then create a deployment and a service for the web application.

kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
kubectl expose deployment web --port=8080
kubectl get svc web

Within the ingress manifest, we enable TLS by including a tls section in the manifest file to reference the secret containing the certificate issued.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - demo.example.com
    secretName: example-com-tls
  rules:
  - host: demo.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web
            port:
              number: 8080

kubectl apply -f ingress.yaml
kubectl get ingress

To simulate the domain name resolution on the local machine. We can edit the hosts file with sudo vi /etc/hosts .

127.0.0.1 demo.example.com

We can test the application by browsing https://demo.example.com and verifying the certificate according to your browser.

We can also verify and show TLS handshake details by running the following command. From the output of this command, we can see the start date, expire date, and issuer info of the issued certificate is correct based on our configuration (valid for 5 minutes).

curl -kivL https://demo.example.com

Either from the browser window or curl command, we can also demonstrate the automatic certificate renewal by refreshing the browser or rerunning the curl command after 5 minutes according to our setup. We can see that it has automatically renewed values for start date and expire date.

Conclusion

The Kubernetes auth method simplifies how an application (cert-manager) uses Kubernetes Service Account to authenticate and interact with HashiCorp Vault. HashiCorp Vault PKI engine act as a Certificate Authority to simplify the issuing process of certificates which can be managed by the cert-manager automatically and efficiently, such as certificate renewal.

The are many other use cases from HashiCorp Vault in cloud-native platforms or applications that we can apply, such as application secret injection and management.

Thanks for reading.