Introduction:
Pods such as contexts-service can get stuck in a CrashLoopBackOff status when failing to get a root token from vault. This can be verified by checking the logs for the affected service.
Prerequisites
Access to your Kubernetes cluster is needed in order to run the kubectl commands below.
Instructions
Step 1 - Verifying the pod logs
First, run a command like the following to get the most recent logs for the affected pod(s).
kubectl -n <namespace> get logs -l app=contexts-service
Logs such as the following indicate an issue connecting to Vault:
Using Vault encryption backend Retrieving vault root token from url "http://vault:8201/client-token" ... ... failed to retrieve root token. Retrying 9 more times ... ... failed to retrieve root token. Retrying 8 more times ... ... failed to retrieve root token. Retrying 7 more times ... ... failed to retrieve root token. Retrying 6 more times ... ... failed to retrieve root token. Retrying 5 more times ... ... failed to retrieve root token. Retrying 4 more times ... ... failed to retrieve root token. Retrying 3 more times ... ... failed to retrieve root token. Retrying 2 more times ... ... failed to retrieve root token. Retrying 1 more times ... ... failed to retrieve root token.
Step 2 - Rolling vault-0 pod
You will need to roll the vault-0 pod. This can be done by deleting the pod with the following command, after which the pod will automatically be recreated.
kubectl delete pods -n <namespace> -l app=vault
Step 3 - Rolling contexts-service pods
You will need to roll the contexs-service pods as well. This can be done by deleting the pod with the following command, after which the pods will automatically be recreated.
kubectl delete pods -n <namespace> -l app=contexts-service
Outcome:
Builds will start running normally, and there will no longer be failed to retrieve root token generated in the pod logs.