How to exchange storage classes in kubernetes

Aloha people,

this tutorial will explain how to migrate data from one StorageClass to another.

Here is the scenario: You have a certain setup with some prometheus pods. These pods have a PersitanceVolumeClaim (pvc) of 10Gi and a StorageClass (sc) called "gp2" with an PersistanceVolume (pv) attached to it. Now the thing is the sc was manually configured to use 'retain' as the reclaimPolicy. This will be resettet back to 'delete' with deploying the same cluster again. We now want to move the persistence volumes to a new defined sc with the reclaimPolicy set to 'retain'. The important fact is that we don't want to lose the data of these volumes.

So here without further ado… here… we… go.

Step 1: List your pods and find out which pods to scale down

kubectl get pods -n namespace

Step 2: List your deployments, replicasets or statefulsets

kubectl get statefulset -n namespace

Step 3: Scale down the pods which are writing to the volumes

kubectl scale statefulset --replicas 0 -n namespace name_of_statfulset

Step 4: List your storage classes

kubectl get sc -n namespace

Step 5: Create a new storage class

kubectl apply -f storage.yml

Here is a example configuration for a storage class:

[here is your current storage class...]

---

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: [NAME_OF_SC]
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
mountOptions:
  - debug

Step 6: tell the persistence volume to use the new storage class

1. go into the pv configuration on kubernetes

   kubectl edit pv -n namespace pvc-uuid_of_pv

2. change field 'storageClassName' to the name of the new sc

Step 7: create local file with configuration of pvc and delete all unneccessary lines

kubectl edit pvc -n namespace name_of_pvc

Here is a sample configuration:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: prometheus
    prometheus: k8s
  name: [NAME_OF_PVC]
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  dataSource: null
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp2-retain
  volumeName: [NAME_OF_PV]
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  phase: Bound

Step 8: delete the pvc which you want to switch but remember the corresponding name_of_pv

kubectl delete pvc -n namespace name_of_pvc

Step 9: create new pvc which is using the new sc

kubectl apply -n namespace -f the_config_file_of_the_old_pvc

If you now query the pvc the newly created one should display as being lost. This is fine because now we need to connect the pv which is already connected to the sc with it's corresponding pvc.

Step 10: Connect pv with corresponding PersitanceVolumeClaim

1. Open the config of the newly created pvc

   kubectl edit pvc -n namespace name_of_pvc

2. copy the uid of the pvc

   labels:
   app: ...
   prometheus: ...
   name: name_of_pvc
   resourceVersion: ...
   selflink: ...
   uid: 871e49b9-3dbe-48d3-b1e8-c2439ec1fb78 

3. Edit the pv and change the uid of the corresponding pvc in the config

   kubectl get pv -n namespace
   kubectl edit pv -n namespace name_of_pv
   
   Search and change this:
   claimRef:
   apiVersion: ...
   kind: PersitanceVolumeClaim
   name: name_of_pvc
   namespace: namespace
   resourceVersion: ...
   uid: [uuid_of_pvc]

Step 11: Scale up your pods again

  kubectl scale statefulset --replicas 2 -n namespace name_of_statfulset

AAAND now you are done :)