Data Volume IO Throttling¶
In HwameiStor, it allows users to specify the maximum IOPS and throughput for volumes on a Kubernetes cluster.
Follow these steps to create a volume with maximum IOPS and throughput and create a workload to use it.
Prerequisites¶
-
The cluster has already installed HwameiStor
-
cgroup v2 requirements:
- The operating system distribution enables cgroup v2
- Linux kernel is 5.8 or higher
For more information, see Kubernetes official documentation.
Create a New StorageClass with IOPS and Throughput Parameters¶
-
Go to the Container Management module, find the cluster with HwameiStor installed in the cluster list, and click the cluster name.
-
In the left navigation bar, select Container Storage -> Storage Pools (SC), and click the Create Storage Pool (SC) button in the upper left corner.
-
Enter the create storage pool interface, paying special attention to filling in the following parameters. Other parameters can refer to form creation.
- Name: For this example, enter
hwameistor-storage-lvm-hdd-sample
. - Storage System: Select HwameiStor.
- Storage Type: Supports LVM Type and Bare Metal Type. In this example, select LVM Type.
- Disk Type: Supports HDD and SSD. In this example, select HDD.
-
Custom Parameters: Fill in the following four parameters,
poolType: REGULAR
: Specify the storage pool type, currently onlyREGULAR
is supported.csi.storage.k8s.io/fstype: xfs
: Specify the required file system type, if not defined, the default isext4
.provision-iops-on-creation: "100"
: Specify the maximum IOPS of the volume at creation.provision-throughput-on-creation: 1Mi
: Specify the maximum throughput of the volume at creation.
- Name: For this example, enter
-
Click OK . After the creation is successful, return to the SC list interface.
Create a PVC Using the StorageClass¶
-
In the left navigation bar, select Container Storage -> Persistent Volume Claims (PVC), and click the Create Persistent Volume Claim (PVC) button in the upper left corner.
-
Enter the create persistent volume claim interface, and fill in the following parameters.
- Name: For this example, enter
pvc-hwameistor-sample
. - Storage Pool: Select the SC created above, named
hwameistor-storage-lvm-hdd-sample
. - Capacity: For this example, enter
10
. - Access Mode: The default is
ReadWriteMany
.
- Name: For this example, enter
-
Click Confirm. After the creation is successful, return to the SC list interface. After completion, you can create a Deployment to use the PVC.
Create a Deployment with PVC¶
-
In the left navigation bar, select Workloads -> Stateless Workloads, and click the Create from Image button in the upper left corner.
-
Pay attention to the following parameters:
- Image: For this example, enter
daocloud.io/daocloud/testtools:latest
. -
Data Storage:
- Type: Select
Persistent Volume Claim (PVC)
. - Persistent Volume Claim (PVC): Select
pvc-hwameistor-sample
. - Container Path (mountPath): Enter
/data
.
- Type: Select
Other parameters have no special requirements.
- Image: For this example, enter
-
After creating the Deployment, click Console in the details interface, and execute the following command to test the IOPS and throughput of the volume:
fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4K -size=50M -numjobs=1 -runtime=600 -group_reporting -filename=/data/file.txt -name=Rand_Write_IOPS_Test
It is expected to output the following:
Note
Due to cgroupv1 limitations, the maximum IOPS and throughput settings may not take effect for non-direct IO.
Change the Maximum IOPS and Throughput of a Volume¶
The maximum IOPS and throughput are specified in the StorageClass parameters, and you cannot change it directly as it is currently immutable.
Unlike other storage vendors, HwameiStor is a Kubernetes-based storage solution that defines a set of operational primitives based on Kubernetes CRD. This means you can modify the relevant CRD to change the actual maximum IOPS and throughput of the volume.
The following steps show how to change the maximum IOPS and throughput of a volume.
Find the LocalVolume CR Corresponding to the Given PVC¶
$ kubectl get pvc pvc-sample
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
demo Bound pvc-c354a56a-5cf4-4ff6-9472-4e24c7371e10 10Gi RWO hwameistor-storage-lvm-hdd 5d23h
pvc-sample Bound pvc-cac82087-6f6c-493a-afcd-09480de712ed 10Gi RWO hwameistor-storage-lvm-hdd-sample 5d23h
$ kubectl get localvolume
NAME POOL REPLICAS CAPACITY USED STATE RESOURCE PUBLISHED FSTYPE AGE
pvc-c354a56a-5cf4-4ff6-9472-4e24c7371e10 LocalStorage_PoolHDD 1 10737418240 33783808 Ready -1 master xfs 5d23h
pvc-cac82087-6f6c-493a-afcd-09480de712ed LocalStorage_PoolHDD 1 10737418240 33783808 Ready -1 master xfs 5d23h
According to the printed output, the LocalVolume CR for the PVC is pvc-cac82087-6f6c-493a-afcd-09480de712ed
.
Modify the LocalVolume CR¶
In the editor, find the spec.volumeQoS
section and modify the iops
and throughput
fields. By the way, empty values mean no limit.
Finally, save the changes and exit the editor. The settings will take effect in a few seconds.
Note
In the future, once Kubernetes supports this feature, we will allow users to directly modify the maximum IOPS and throughput of the volume.
Check the Actual IOPS and Throughput of a Volume¶
HwameiStor uses cgroupv1 to limit the IOPS and throughput of the volume, so you can use the following command to check the actual IOPS and throughput of the volume.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 160G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 159G 0 part
├─centos-root 253:0 0 300G 0 lvm /
├─centos-swap 253:1 0 7.9G 0 lvm
└─centos-home 253:2 0 101.1G 0 lvm /home
sdb 8:16 0 100G 0 disk
├─LocalStorage_PoolHDD-pvc--cac82087--6f6c--493a--afcd--09480de712ed
253:3 0 10G 0 lvm /var/lib/kubelet/pods/3d6bc980-68ae-4a65-a1c8-8b410b7d240f/v
└─LocalStorage_PoolHDD-pvc--c354a56a--5cf4--4ff6--9472--4e24c7371e10
253:4 0 10G 0 lvm /var/lib/kubelet/pods/521fd7b4-3bef-415b-8720-09225f93f231/v
sdc 8:32 0 300G 0 disk
└─sdc1 8:33 0 300G 0 part
└─centos-root 253:0 0 300G 0 lvm /
sr0 11:0 1 973M 0 rom
$ cat /sys/fs/cgroup/blkio/blkio.throttle.read_iops_device
253:3 100
$ cat /sys/fs/cgroup/blkio/blkio.throttle.write_iops_device
253:3 100
$ cat /sys/fs/cgroup/blkio/blkio.throttle.read_bps_device
253:3 1048576
$ cat /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device
253:3 1048576