One of the most tantalizing promises of Kubernetes is that it reinforces our applications' resiliency by letting them heal themselves. It also gives us tools to prevent our applications from ever getting unhealthy in the first place. Probes are central to both of these capabilities.

They come in three flavors:

  • Startup probes
  • Liveness probes
  • Readiness probes

Startup probes

Recall from the last exercise on the page on Helm that we had deployed an update to our API which, through misconfiguration, had broken all its functionality. This was an intentionally fabricated example, but issues like it legitimately crop up from time to time. Passwords might have been changed but not updated in the application. Connection details may have been mixed up between environments. The failed deployment due to bad database configuration is an unfortunate fixture of API development.

The probe that best protects against this is the startup probe.

  • This probe defines a check that the cluster will do when attempting to startup a container.
  • A container that passes this check is flagged as Ready.
  • If it fails, it is given a specified number of retries. And if it fails all of these tries, the container is terminated and the cluster will try to startup the container again.

That's the idea. Let's see how it works in practice.

In your ShoppingCart Helm chart, go to values-prod.yaml and:

  • Correct any intentional error you made to the Redis connection details
  • Reset shoppingCartConfigMapVersion to prod-0.0.1

Then, in your template file, find the part of the template that defines the Deployment, and change it to the below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shopping-cart-dep
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      depLabel: shopping-cart
  template:
    metadata:
      name: shoping-cart-pod
      labels:
        depLabel: shopping-cart
        component: shopping-cart
    spec:
      containers:
      - name: shopping-cart-ctr
        image: {{ .Values.imageName }}:{{ .Values.imageVersion }}
        startupProbe:          httpGet:            path: /probe/startup            port: {{ .Values.serverPort }}          periodSeconds: {{ .Values.startupPeriodSeconds }}          failureThreshold: {{ .Values.startupFailureThreshold }}        ports:
        - containerPort: {{ .Values.serverPort }}
        volumeMounts:
        - name: properties-volume
          mountPath: /usr/local/lib/override.properties
          subPath: override.properties
        - name: passwords-volume
          mountPath: /usr/local/lib/passwords.properties
          subPath: passwords.properties
      volumes:
      - name: properties-volume
        configMap:
          name: shopping-cart-configmap-{{ .Values.shoppingCartConfigMapVersion }}
          items:
          - key: my-conf
            path: override.properties
      - name: passwords-volume
        secret:
          secretName: {{ .Values.redisPwSecretName }}
          items:
          - key: {{ .Values.redisPwSecretPath }}
            path: passwords.properties

The only part of this that is new is the startupProbe and the five lines underneath it.

  • The check that this startup probe makes is an HTTP request to the API's GET /probe/startup endpoint. In this case, the endpoint makes a lightweight call to Redis (specifically, a PING command) to ensure the Redis connection details are correct. But in your other projects you can write whatever logic you want to check whether your API is ready and working.
    • The cluster will pass this check if the HTTP request returns an acceptable status code such as 200, and will fail it if it doesn't return, or returns a status code of 400 or greater.
  • periodSeconds defines how often the check is attempted. This is one of the things we've parameterized so as to be able to change it from a Helm values file, but the value we'll be choosing is 10. So, it'll try calling the GET /probe/startup endpoint every 10 seconds.
  • failureThreshold is the number of failed checks in a row after which the container will be deemed unhealthy and recreated. We'll be putting 12 here, so it'll make up to 12 checks.

Finally, in values.yaml:

  • change replicaCount to 3 (both because it's a good practice to have at least 3 replicas, and because this will help demonstrate a point later in this exercise)
  • add startupPeriodSeconds: 10 and startupFailureThreshold: 12

So when you're done values.yaml should look like this:

imageName: bmcase/shopping-cart
imageVersion: 0.0.2

replicaCount: 3

startupPeriodSeconds: 10
startupFailureThreshold: 12

Deploy the chart with helm upgrade --install --atomic --timeout 3m shopping-cart ShoppingCart -f ShoppingCart/values.yaml,ShoppingCart/values-prod.yaml

You'll notice it takes several seconds longer for Helm to return with the results of the upgrade. This is because it is now waiting for the check on the startup probe to be sent and to pass.

This API has been written so that it logs info pertaining to its probes. Find the name of any of the pods with kubectl get pods, and then use kubectl logs <pod-name>. You'll see a log line with "Received request to startup probe" at the end, followed by one with "Startup check passed."

When the startup probe fails

Now go to values-prod.yaml and do the following:

  • Again do the change to redisHost so as to make it invalid.
  • Increment the 0.0.1 at the end of the value for shoppingCartConfigMapVersion to be 0.0.2.

Then run helm upgrade again just like above.

Helm will take much longer to return. While it is running, open another command line or terminal window and run kubectl get pods to observe what is happening.

$ kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
shopping-cart-dep-85978cf6d8-7q8ft    1/1     Running   0          3m11s
shopping-cart-dep-85978cf6d8-qzhht    1/1     Running   0          3m11s
shopping-cart-dep-85978cf6d8-vm2ls    1/1     Running   0          3m11s
shopping-cart-dep-6876c8bbff-mdj6b    0/1     Running   0          45s

Since it's doing a rolling update, Pods from the previous, successful deployment still remain. But it is failing to create Pods with the updated configuration. You can confirm using kubectl describe pod <pod-name> and kubectl logs <pod-name> that this is because the startup probe is encountering an error. After 3 minutes of trying (due to the --timeout 3m option set for helm upgrade) it will stop and rollback the release.

You may wonder if, during this deployment attempt, the Service would route any incoming requests to the failing Pod. But no requests will be sent to a Pod unless that Pod is designated as "Ready". So, in the above example, requests will only be routed to the first three Pods.

Liveness and readiness probes

The other two flavors of probes check for the continued health of the Pod once it's on the cluster.

  • Liveness: If it fails a specified number of checks in a row, the Pod will be restarted.
  • Readines: If it fails a specified number of checks in a row, Services will no longer route traffic to the Pod until it passes a check.

Add these probes by modifying the Deployment to add a livenessProbe and a readinessProbe right below where the startupProbe was added. Note that different values will be provided for PeriodSeconds and FailureThreshold, so don't just copy there what you used for the startup probe.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shopping-cart-dep
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      depLabel: shopping-cart
  template:
    metadata:
      name: shoping-cart-pod
      labels:
        depLabel: shopping-cart
        component: shopping-cart
    spec:
      containers:
      - name: shopping-cart-ctr
        image: {{ .Values.imageName }}:{{ .Values.imageVersion }}
        startupProbe:
          httpGet:
            path: /probe/startup
            port: {{ .Values.serverPort }}
          periodSeconds: {{ .Values.startupPeriodSeconds }}
          failureThreshold: {{ .Values.startupFailureThreshold }}
        livenessProbe:          httpGet:            path: /probe/live            port: {{ .Values.serverPort }}          periodSeconds: {{ .Values.livenessPeriodSeconds }}          failureThreshold: {{ .Values.livenessFailureThreshold }}        readinessProbe:          httpGet:            path: /probe/ready            port: {{ .Values.serverPort }}          periodSeconds: {{ .Values.readinessPeriodSeconds }}          failureThreshold: {{ .Values.readinessFailureThreshold }}        ports:
        - containerPort: {{ .Values.serverPort }}
        volumeMounts:
        - name: properties-volume
          mountPath: /usr/local/lib/override.properties
          subPath: override.properties
        - name: passwords-volume
          mountPath: /usr/local/lib/passwords.properties
          subPath: passwords.properties
      volumes:
      - name: properties-volume
        configMap:
          name: shopping-cart-configmap-{{ .Values.shoppingCartConfigMapVersion }}
          items:
          - key: my-conf
            path: override.properties
      - name: passwords-volume
        secret:
          secretName: {{ .Values.redisPwSecretName }}
          items:
          - key: {{ .Values.redisPwSecretPath }}
            path: passwords.properties

Then add livenessPeriodSeconds, livenessFailureThreshold, readinessPeriodSeconds, and readinessFailureThreshold to your values.yaml:

imageName: bmcase/shopping-cart
imageVersion: 0.0.2

replicaCount: 3

startupPeriodSeconds: 10
startupFailureThreshold: 12

livenessPeriodSeconds: 10
livenessFailureThreshold: 6

readinessPeriodSeconds: 10
readinessFailureThreshold: 1

The rationales behind the different failure thresholds are:

  • startupFailureThreshold: 12 because the Pod may take a long time to start. We want to have a high threshold to account for the failures that will occur before the Pod is actually initialized and ready to receive requests.
  • livenessFailureThreshold: 6 because we don't want to be restarting Pods on a whim. You want to be sure that failing the chosen number of liveness checks is actually indicative that the Pod needs to be restarted. So keep this in mind both when creating the logic behind the endpoint and when specifying the probe's failure threshold.
  • readinessFailureThreshold: 1 so as to reduce the impact to the user of failing requests. If the Ready designation is removed from one Pod, there will still be others to handle the requests until it gets healthy.

In values-prod.yaml, correct any errors you made in the Redis connection details, and increment shoppingCartConfigMapVersion again. Then use helm upgrade to replace the current Pods with ones that use liveness and readiness probes. You can confirm that these probe endpoints are being hit by running kubectl logs <pod-name>.

Images with failing probe endpoints

There are three variants of the bmcase/shopping-cart:0.0.2 image. Each corresponds to a probe endpoint that is set to always fail when using that image:

  • bmcase/shopping-cart:0.0.2-fail-startup
  • bmcase/shopping-cart:0.0.2-fail-live
  • bmcase/shopping-cart:0.0.2-fail-ready

You can try swapping in each of these image versions. See if you can predict what will happen, and then observe the result.

Common practices relating to probe endpoints

The API used in this example had separate endpoints for each probe. But it is a common practice to have all your probes use the same endpoint, and this will be fine most of the time.

You're likely not going to want to expose your probe endpoints outside of the cluster, since they provide no benefit to your users and there's no reason for them to access them. Outside access to your probe endpoints can be prevented by using the routing capabilities of your Ingress, or by using an API gateway or reverse proxy.

A demonstration of the importance of versioning the names of ConfigMaps

The example above had you increment the version of shoppingCartConfigMapVersion before introducing the error to the Redis connection details. But what if you hadn't done that?

To setup:

  • Use helm uninstall shopping-cart to remove the API and give yourself a clean slate.
  • Correct any errors you made to the Redis connection details in values-prod.yaml, and reset the version at the end of shoppingCartConfigMapVersion to be 0.0.1.
  • Then install the API again with helm upgrade --install --atomic --timeout 3m shopping-cart ShoppingCart -f ShoppingCart/values.yaml,ShoppingCart/values-prod.yaml
  • Since it should be configured correctly now, the installation should succeed.

What we'll do now is again change redisHost to something invalid, but leave shoppingCartConfigMapVersion as is. Do this and once again run the same helm upgrade command to upgrade the release.

You'll notice no apparent change to the API. If you run kubectl get pods, you'll see, as indicated by the "AGE" column, that the pods were never brought down. That's because the cluster detected no change to the Pod configuration, and so it saw no need to restart any pods.

But now a silent-but-deadly threat lurks in your cluster: the erroneous configuration actually was applied to your ConfigMap. If any Pods go down, they will fail when the cluster tries to restart them. And if all Pods go down, none will be able to restart, and your application will experience unexpected downtime. This is likely to happen, too. Many cluster administrators will perform routine maintenance which, as a by-product, will result in rolling restarts of Pods.

You can simulate this with kubectl scale deployment shopping-cart-dep --replicas=0 followed by kubectl scale deployment shopping-cart-dep --replicas=3.

If you use kubectl get pods after doing this, you'll see the Deployment's Pods are unable to start. This can be confirmed by running kubectl describe pod <pod-name> on any of the Pods.

$ kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
shopping-cart-dep-658959d669-cbvqd    0/1     Running   0          105s
shopping-cart-dep-658959d669-4tb8q    0/1     Running   0          105s
shopping-cart-dep-658959d669-h4d6k    0/1     Running   0          105s

Changing the ConfigMap name avoids this problem because that name is part of the Pod definition, in its volumes section.

  • The cluster will see that the upgrade changes the Pod definition, and so it will attempt a rolling restart of the Pods.
  • Some of the previous Pods will remain up during this, so the application will not go down.
  • But the new Pods will fail to start.
  • After trying this for long enough, Helm's timeout duration will have elapsed and it will rollback this deployment attempt.

The end result is that you're immediately made aware of a problem in your configuration, instead of leaving it to blow up at some point down the line.

Probes using checks other than HTTP

In the examples on this page, the probes have performed checks by making HTTP requests to endpoints. But probes can also perform checks by running commands in the containers. See an example of that here.