Each container has its own file system. The files within only last for the lifetime of the container. When Kubernetes terminates a Pod, the files in its containers will be lost.

For many applications, this is nothing to worry about. A common Kubernetes use case is a microservice that receives requests, makes calls to an external database or API, and then returns the results. The data may change, but the API itself won't do anything differently from previous requests. There's no need to store any state for later reference; there's no need to persist.

For other applications, the threat of sudden loss of a container's previously written-to files is a problem. Some applications might be databases themselves. Others might have a cache, or maintain the state of a session. For these, we'll need PersistentVolumes.

To demonstrate we'll deploy a small API.

  • The API, called message-holder, will expose a couple REST endpoints.
  • The first endpoint will let us send it a message, which it'll store in a file.
  • The second endpoint will return the contents of the file.

Create the objects needed for this with the following manifests (and delete your previous Ingress if using Rancher Desktop):

apiVersion: v1
kind: Service
metadata:
  name: msg-holder-service
spec:
  ports:
  - port: 8082
  selector:
    component: msg-holder
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: msg-holder-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: msg-holder-service
            port:
              number: 8082
apiVersion: apps/v1
kind: Deployment
metadata:
  name: msg-holder-dep
spec:
  replicas: 1
  selector:
    matchLabels:
      depLabel: msg-holder
  template:
    metadata:
      name: msg-holder-pod
      labels:
        depLabel: msg-holder
        component: msg-holder
    spec:
      containers:
      - name: msg-holder
        image: bmcase/message-holder
        ports:
        - containerPort: 8082

Once it's up and running, you can make requests to its REST endpoints via the below curl commands:

curl -X GET 'localhost/api/v1/m'
curl -X PUT 'localhost/api/v1/m' \
--header 'Content-Type: text/plain' \
--data-raw 'Hello, World!'

(If you're using Windows, you can try using curl for Windows, or recreating these commands in a REST client like Postman.)

Try out the endpoints to gain familiarity with how the API is supposed to work.

Since the messages are being saved to the container's file system, they will not survive when the container is brought down and then brought back up. You can confirm this by using kubectl scale deployment --replicas=0 msg-holder-dep followed by kubectl scale deployment --replicas=1 msg-holder-dep, which will remove the Pod and then start a new one. Any previously saved message will no longer be there.

Let's have the Pod use a PersistentVolume so that the message may be retained even when the Pod is recreated. To do this, we'll actually need to create two objects: the PersistentVolume, and a PersistentVolumeClaim.

  • You can think of a PersistentVolume as another piece of the raw materials, like nodes or load balancers or system resources, that the Kubernetes cluster must use in order to put together your Deployments, Services, etc. If you're an application developer, you might not ever even create a PersistentVolume in practice. Instead it would be done by the cluster administrator. But if you're using a local cluster like Rancher Desktop or Minikube you'll have to do it yourself.
  • A PersistentVolumeClaim is a Pod's way of defining what kind of PersistentVolume it needs. The Pod doesn't choose a certain PersistentVolume, but rather describes its requirement to the cluster, which then provides the Pod with one.

Use kubectl apply to add the below PersistentVolume manifest:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: app-pv
spec:
  capacity:
    storage: 256M
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: app-sc
  hostPath:
    path: "/tmp"

and the below PersistentVolumeClaim manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-pvc
spec:
  storageClassName: app-sc
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 64M

It bears repeating that the PersistentVolumeClaim describes a need. In this case:

  • It needs a ReadWriteMany access mode. (More on this later.)
  • It needs the volume to have at least 64 megabytes of storage.
  • The PersistentVolume that satisfies this claim must be designated with the app-sc storage class.

Storage classes are created by the cluster administrators, and are not always needed in PersistentVolumeClaims. Cluster admins use them because sometimes cluster tenants (that's you if you're an application developer) have various capabilities that they need from their storage, and storage classes are a way of separating the volumes and designating what kind of capabilities they offer. If a PersistentVolumeClaim doesn't specify a storage class, the cluster may satisfy it with any class that fits the claim's other requirements.

(It's worth noting that individual clusters will commonly have rules governing the use of PersistentVolumeClaims, and one of these may be that a storage class needs to be specified. So the field will then be mandatory.)

Finally, you need to connect your PersistentVolumeClaim to your Pod, so apply the below manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: msg-holder-dep
spec:
  replicas: 1
  selector:
    matchLabels:
      depLabel: msg-holder
  template:
    metadata:
      name: msg-holder-pod
      labels:
        depLabel: msg-holder
        component: msg-holder
    spec:
      containers:
      - name: msg-holder
        image: bmcase/message-holder
        ports:
        - containerPort: 8082
        volumeMounts:        - name: message-storage          mountPath: /app/message-folder      volumes:      - name: message-storage        persistentVolumeClaim:          claimName: app-pvc

Note specifically the last 7 lines:

  • At the very bottom, a volumes section was added to the Pod spec.
    • The name field is something you create for use in the containers section.
    • The persistentVolumeClaim.claimName field refers to the name you gave your PersistenVolumeClaim in its manifest.
  • To the container's spec was added a volumeMounts section.
    • The name here refers to the one from the Pod's volumes section.
    • The mountPath is the actual absolute path of the folder within the container. In our example, this is /app/message-folder because that's where the container stores the file containing the message.

Use kubectl apply with this, then use the API to add a message. When you want to test whether the message is being persisted through Pod restarting, use kubectl scale twice again in the same way as before. Afterward you should be able to immediately get the same message despite the Pod having been recreated.

Access modes and sharing volumes between Pods

Right now there's only 1 replica of the API Pod. It's a good practice to use at least 3 replicas of your servers and microservices so as to better guarantee application resiliency. Use kubectl scale deployment --replicas=3 msg-holder-dep to increase the number of Pods to 3.

Once these Pods are all ready, run curl -X GET 'localhost/api/v1/m' 10 times in succession. Then use kubectl get pods, followed by kubectl logs <pod-name> for each of the 3 pods.

$ curl -X PUT 'localhost/api/v1/m' \
--header 'Content-Type: text/plain' \
--data-raw 'Hello, World!!!'
$ kubectl scale deployment --replicas=3 msg-holder-dep
deployment.apps/msg-holder-dep scaled
$ curl -X GET 'localhost/api/v1/m'
Hello, World!!!
$ curl -X GET 'localhost/api/v1/m'
Hello, World!!!
$ curl -X GET 'localhost/api/v1/m'
Hello, World!!!
$ curl -X GET 'localhost/api/v1/m'
Hello, World!!!
$ curl -X GET 'localhost/api/v1/m'
Hello, World!!!
$ curl -X GET 'localhost/api/v1/m'
Hello, World!!!
$ curl -X GET 'localhost/api/v1/m'
Hello, World!!!
$ curl -X GET 'localhost/api/v1/m'
Hello, World!!!
$ curl -X GET 'localhost/api/v1/m'
Hello, World!!!
$ curl -X GET 'localhost/api/v1/m'
Hello, World!!!
$ kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
msg-holder-dep-695576f76b-xdgf9       1/1     Running   0          4m28s
msg-holder-dep-695576f76b-t582p       1/1     Running   0          76s
msg-holder-dep-695576f76b-9dcpf       1/1     Running   0          76s
$ kubectl logs msg-holder-dep-695576f76b-xdgf9
2024/06/02 02:39:57 Received message to store
2024/06/02 02:47:09 Received request for message
2024/06/02 02:47:14 Received request for message
2024/06/02 02:47:16 Received request for message
$ kubectl logs msg-holder-dep-695576f76b-t582p
2024/06/02 02:46:49 Received request for message
2024/06/02 02:47:13 Received request for message
2024/06/02 02:47:16 Received request for message
2024/06/02 02:47:18 Received request for message
$ kubectl logs msg-holder-dep-695576f76b-9dcpf
2024/06/02 02:47:12 Received request for message
2024/06/02 02:47:15 Received request for message
2024/06/02 02:47:17 Received request for message

Notice the following:

  • The message you had previously set should've returned successfully all 10 times when using curl.
  • The logs indicate that the requests to get the message were spread out among the 3 Pods. This is normal behavior of ClusterIP Services—to balance requests among the matching Pods.
  • You only ever set the message once, in a request to one of the Pods, before the other 2 Pods were even created. But all 3 Pods are still able to serve the correct message, and so evidently all 3 have access to the same file in the volume.

The Pods are able to all read and write to the same files due to the ReadWriteMany access mode. There are several access modes, and a guide to all of them is beyond the scope of this document, but you can read more here. ReadWriteMany is often the best access mode to use for API deployments. But not all clusters support it, and which access mode you end up using may in practice be determined by rules set for the cluster.