Providing Sufficient PVCs for Your StatefulSets: Creating New Volumes Larger Than the PVCTemplate

Providing Sufficient Storage Capacity for Your StatefulSets: Creating New Volumes Larger Than the PVCTemplate Size Kaoru Esashika, Cybozu, Inc.

Introduction • • • Name: Kaoru Esashika Cloud Infrastructure Division, Cybozu, Inc. A maintainer of some OSS products • • • TopoLVM pvc-autoresizer pvc-autoresizer provides the ability to automatically expand PVCs. 3

Request from the Database Team According to the database team: “When an instance that makes up a MySQL cluster is recreated, it crashes due to insufficient capacity and cannot return to the cluster.” Database Cluster In Normal Operation: Node Node Broken Node Node A Database Instance A Database Instance A Database Instance New Database Instance Migrate 4

Request from the Database Team According to the database team: “When an instance that makes up a MySQL cluster is recreated, it crashes due to insufficient capacity and cannot return to the cluster.” Database Cluster Broken Node A Database Instance Migrate Node New Database Instance Failed to start: Insufficient free capacity! Why? 5

Investigation 1. Older instances had repeated auto-expansion of capacity due to long term use. 2. When an instance is re-created, the capacity indicated by initial PVC template is requested. 3. An instance with a small volume is created. 4. MySQL tries to start synchronizing but gives up because the capacity is too small. database instance grew larger over time database instance database instance recreate with smaller size 7

Improvements • • • Our Improved Group Provisioning rewrites the PVC dynamically by using Mutating Webhook as new feature of pvc-autoresizer. Volume size of rewrote PVC has same capacity as other instances, even if the volume is originally requested in a smaller capacity. As a result, instances can begin synchronizing with sufficient capacity. instance instance instance original PVC mutated PVC same size as another instance 8

Improvements (cont.) • • Annotations are used to define the instances included in the group. Even if there are similar clusters in the same namespace, they can be separated by annotation. PVCTemplate in StatefulSet are immutable. annotated instance annotated instance annotated instance instance original PVC mutated PVC annotated instance same size as another instance 9

Extensibility provided by Mutating Webhook Controller Controller create/update Kubernetes resources. • • Mutating Webhook Mutating Webhook intercept and modify resources. Kubernetes Resources • Kubernetes has various extension points. Mutating Webhooks is a very powerful point that can change its behavior without directly modifying an existing controller. In this case, we extended the behavior of K8s, but it may have other applications. 11