Adding vSphere/vSAN Storage Class to Kubernetes
I am a HELM lover. At first, I always searched for a helm solution. If I do not find any, I will write them myself. But this time, I found it. Thanks to “Stefan van Gastel”
https://github.com/stefanvangastel/vsphere-cpi-csi-helm.git
You must download the code to your Bastion host or local PC.
git clone https://github.com/stefanvangastel/vsphere-cpi-csi-helm.git
I want to mention my system in order to have my aim understood. I have a 5-node vSphere cluster (VMWare ESXi, 7.0.3) and a vSAN cluster. I have 5 VMs for the RKE2/Rancher Kubernetes cluster on the vSAN cluster. Before, I installed Longhorn on RKE2 nodes and used it as persistent storage. But I had some problems and thought multiple replica storage was the cause.
Before installation, you should check a setting for your vSphere. Your vSphere version must be 7.0 or higher, and your Kubernetes cluster version must be 1.16 or higher for CSI 2.0.0. Moreover, you need extra controls for your VMs. Your Kubernetes node VM version must be 15 or higher, and UUID configuration options must be enabled.
How do you check it or do it?
- Log in to the ESXi or vCenter UI.
- After power-off, right-click the virtual machine, and choose Edit Settings.
- Click the VM Options tab, and select Advanced.
- Click Edit Configuration in Configuration Parameters.
- Click Add parameter.
- In the Key column, type “disk.EnableUUID”
- In the Value column, type “TRUE”
- Click OK and click Save.
- Power on the virtual machine
Check if it is working.
# kubectl get nodes -o json | grep providerID
Output must be like that below: ( 5 Nodes )
“providerID”: “vsphere://421bf484–6a67-d982-e12d-0b6a7a4d0fd8”
“providerID”: “vsphere://421b8453–1a6c-be78–49e9-c392a0d621db”
“providerID”: “vsphere://421b94e4–1d52–3706-ae38–17697276fe03”
“providerID”: “vsphere://421b9151–828a-f013–9e8f-336734c2c6b8”
“providerID”: “vsphere://421be6ee-c163–13c1-eb72-fa75aff2deaf”
This environment plays a crucial role. If you do not forget to do it, your PVC objects will be in “Pending Status.”
You should find your “datastoreurl” in vSAN from vCenter or the ESXi UI. You can click the storage icon from the left-side icon menu.
I needed a private values file because I use RKE2 and Rancher uses “control-plane” label instead of “controlplane”. You can see that from https://github.com/stefanvangastel/vsphere-cpi-csi-helm/blob/master/charts/vsphere-cpi-csi/v2.3.0/values.yaml
So I need to change the nodeSelector label.
nodeSelector:
node-role.kubernetes.io/control-plane: "true"
Meanwhile, I used v2.3.0, which is the last one now.
That is my custom “values.yaml”
cloudProvider:
nodeSelector:
node-role.kubernetes.io/master: "true"
csiController:
nodeSelector:
node-role.kubernetes.io/master: "true"
I decided to use the “master” value, because my nodes are masters.
We prepared our helm, but we need a CSIDriver object ( CRD ). if you have not installed it yet. After this operation, we will have a CSI storage class. So if you have an error about CSIDriver, check that https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/docs/install-csi-driver-master.md
You can check your “CSIDrivers” with the command below.
kubectl get csidrivers
Output:
error: the server doesn't have a resource type "csidrivers"
So you need to install it like that.
curl -skSL https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/deploy/install-driver.sh | bash -s master --
After all of that, you can run your helm command. That is my example command:
cd vsphere-cpi-csi \
helm upgrade - install vsphere-cpi-csi \
- namespace kube-system \
./charts/vsphere-cpi-csi/v2.3.0 \
- set vcenter.insecurehost=true \
- set vcenter.port=443 \
- set vcenter.host="vcenter.mylocal.net" \
- set vcenter.username="administrator@vsphere.local" \
- set vcenter.password="XXXXXXXX" \
- set storageclass.datastoreurl="ds:///vmfs/volumes/vsan:52a88bd9638b1b83-b0bce0d4cf62b81c/" \
- set vcenter.datacenter="vSAN Datacenter" \
-f values.yaml
Check the installation status now.
# kubectl get pods -n kube-system | grep vsphere
vsphere-cloud-controller-manager-6g6mk 1/1 Running 0 2d4h
vsphere-cloud-controller-manager-824zd 1/1 Running 0 2d4h
vsphere-cloud-controller-manager-gjtm9 1/1 Running 0 2d4h
vsphere-cloud-controller-manager-hk9tt 1/1 Running 0 2d4h
vsphere-cloud-controller-manager-tdl97 1/1 Running 0 2d4h
vsphere-csi-controller-5fbf876877-6cq8c 6/6 Running 0 2d4h
vsphere-csi-node-8hbfr 3/3 Running 0 2d4h
vsphere-csi-node-dfk96 3/3 Running 0 2d4h
vsphere-csi-node-n7gd6 3/3 Running 0 2d4h
vsphere-csi-node-sfn44 3/3 Running 0 2d4h
vsphere-csi-node-zx8hl 3/3 Running 0 2d4h
Everything is perfect. If things don’t go well, you can inspect them from the “vsphere-csi-controller-*****-*****” pod.
# kubectl logs -f vsphere-csi-controller-5fbf876877-6cq8c -n kube-system
For example, you can see that “vsphere-csi”: rpc error: code = Internal desc = failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager”. It means that “spec.providerID” is null. That is a “disk.enableUUID” problem.
Now you can see your CSIDriver and Storage Class.
root@bastion:~# kubectl get csidrivers
NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE
csi.vsphere.vmware.com true false false <unset> false Persistent 2d9h
nfs.csi.k8s.io false false false <unset> false Persistent 3d10h
root@bastion:~# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-csi nfs.csi.k8s.io Delete Immediate true 3d10h
vsphere-csi (default) csi.vsphere.vmware.com Delete Immediate false 2d9h
I am putting in a simple “statefulset” Nginx example yaml to check your success.
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx
replicas: 1 #
minReadySeconds: 10
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: registry.k8s.io/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www-data
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "vsphere-csi"
resources:
requests:
storage: 1Gi
I wish the data sources that there is no data loss. :)