This is a short article with a little k8s persistent storage nugget that I recently uncovered. FWIW, I've only checked this while using the AWS EBS Container Storage Interface (CSI) driver, but the article should apply to any cluster using CSI drivers to mount volumes.

Background

After a PersistentVolume (PV) is created and bound to a Persistent Volume Claim (PVC), a Pod can mount the claim as a volume to access its data. While the Pod is being provisioned, the k8s storage controller creates a VolumeAttachment, which declares its intent to physically attach the volume to the Pod's Node. The storage driver running on the Node will be notified when this VolumeAttachment is created, recognize that it needs to mount a volume to its host, and perform the volume mount, thus making the volume available for the Pod to use.

Lets say that you want to check if a particular volume is actually attached to a Node. For example, maybe you want to ensure that a volume is unmounted before destroying a Node to ensure data integrity of a high-throughput database? So how can you find a PV's corresponding VolumeAttachment to determine if its backing volume is unmounted?

Querying VolumeAttachments

The VolumeAttachment Spec contains nodeName and source fields, which as expected, correspond to the Node's name and PV's name respectively. But these are not valid fieldSelector fields, so you cannot query the API server for a VolumeAttachment with a matching Node or PV name. Instead, you need to LIST all VolumeAttachments in the cluster and iterate over them to find the one that you want. In a large cluster, there could be hundreds (or even thousands) of VolumeAttachments, so this operation could take quite a while to run inside of a controller reconcile loop.

Maybe we could somehow craft a GET request to find a PV's matching VolumeAttachment? Unfortunately, the "name" field, at least when using a supported k8s CSI driver, is a vague string that looks something like csi-e52f481e06b9cf4dc9d95a56d1788026b4707cdce2e105d4444f0be9d6206a09. Where does this name come from? Can we use this to check if a PV is mounted to a particular Node?

VolumeAttachment Naming

After a bit of digging, I found the answer in the k8s csi source code

// getAttachmentName returns csi-<sha256(volName,csiDriverName,NodeName)>
func getAttachmentName(volName, csiDriverName, nodeName string) string {
	result := sha256.Sum256([]byte(fmt.Sprintf("%s%s%s", volName, csiDriverName, nodeName)))
	return fmt.Sprintf("csi-%x", result)
}

The attachment suffix is a sha256 sum of the names of the following components:

  1. The volume handle (part of the CSI Spec)
  2. The driver doing the mounting, and
  3. The target Node

With this discovery, we can now determine if a specific PV is mounted to a Node by using a GET request based on the imputed VolumeAttachment name.

Here's a small function that I wrote to generate a VolumeAttachment name based on a PV and Node:

package csi

import (
	"crypto/sha256"
	"fmt"

	v1core "k8s.io/api/core/v1"
)

func GetVolumeAttachmentName(pv *v1core.PersistentVolume, n *v1core.Node) string {
	if pv.Spec.CSI == nil {
		return ""
	}

	var (
		volName       = pv.Spec.CSI.VolumeHandle
		csiDriverName = pv.Spec.CSI.Driver
		nodeName      = n.Name
	)

	result := sha256.Sum256([]byte(fmt.Sprintf("%s%s%s", volName, csiDriverName, nodeName)))
	return fmt.Sprintf("csi-%x", result)
}

This function accepts a PV and a Node, and sha256sums pieces of their metadata to construct a valid VolumeAttachment name. Now, you can now use this function to generate an input to check if a PV is mounted to a Node, all in a single API request!


func IsPvAttached(pv *v1core.PersistentVolume, n *v1core.Node) bool {
	va := &v1storage.VolumeAttachment{}
	return !apierrors.IsNotFound(
		client.Get(
			ctx,
			types.NamespacedName{
				Name: GetVolumeAttachmentName(pv, n),
			},
			va,
		),
	)
}

The storage controller will delete a VolumeAttachment when a PV is unmounted from a Node, so we can simply check for the existence of a VolumeAttachment with the PV's name & driver and the Node's name by using our GetVolumeAttachmentName function from above. If the VolumeAttachment does not exist, we can definitively say that the volume is unmounted from the Node, and safely delete the node without any data corruption on the volume.

Hopefully this demystifies those strange VolumeAttachment names, and unlocks some new ideas when writing your next k8s controller.