Rook: Intro and Deep Dive With Ceph

1

Rook: Intro and Deep Dive With Ceph Satoru Takeuchi, Cybozu, Inc. June 17th, 2025 2

Agenda ● Introduction to Rook and Ceph ● Block and Filesystem Storage ● Object Storage ● Other Features ● Project Health Title 3

Introduction to Rook and Ceph 4

What is Rook? ● An open source K8s operator to manage Ceph storage ● For Admins ○ Deploy, manage, upgrade Ceph cluster by CR ● For Users ○ Consume Ceph by PVC and OBC CR Title 5

What is Ceph? ● All-in-one open source distributed storage platform Title Name Type RBD Block Storage CephFS Large-scale share filesystem storage RGW S3-compatible object storage CephNFS Export CephFS and S3 object as NFS 6

Remote replications Storage Type Feature RBD RBD mirroring CephFS CephFS mirroring RGW RGW multisite Region B Region A Ceph cluster Replicate Ceph cluster 7

Ceph’s Architecture ● OSD daemons ○ Manage data ● MON daemons ○ Manage cluster’s state ● MGR daemons ○ Provide additional features Network storages Storage (e.g.RBD) Storage (e.g.RBD) (e.g. RBD) Ceph Cluster Mons MGR Storage pool Mons Mons MON OSD OSD … OSD disk disk … disk 8

Ceph’s Architecture ● OSD daemons ○ Manage data ● MON daemons ○ Manage cluster’s state ● MGR daemons ○ Provide additional features Network storages Storage (e.g.RBD) Storage (e.g.RBD) (e.g. RBD) Ceph Cluster Mons MGR Storage pool Mons Mons MON OSD OSD … OSD disk disk … disk 9

Ceph’s Architecture ● OSD daemons ○ Manage data ● MON daemons ○ Manage cluster’s state ● MGR daemons ○ Provide additional features Network storages Storage (e.g.RBD) Storage (e.g.RBD) (e.g. RBD) Ceph Cluster Mons MGR Storage pool Mons Mons MON OSD OSD … OSD disk disk … disk 10

Ceph’s Architecture ● OSD daemons ○ Manage data ● MON daemons ○ Manage cluster’s state ● MGR daemons ○ Provide additional features Network storages Storage (e.g.RBD) Storage (e.g.RBD) (e.g. RBD) Ceph Cluster Mons MGR Storage pool Mons Mons MON OSD OSD … OSD disk disk … disk 11

Ceph’s Characteristics ● High scalability ○ Real example: ~1800 OSDs, over 5 PiB ● High durability ○ Replication or Erasure Coding ○ Configurable failure domains (e.g. rack) ● High availability ○ e.g. Add/remove/replace OSDs online Title 12

Rook’s Architecture ● Rook operator Rook Ceph CSI operator ○ Manage Rook/Ceph clusters manage manage ○ Provision Pod for each Ceph daemons ● Ceph CSI Storage (e.g.RBD) Network storages Storage (e.g.RBD) ○ A CSI Driver for Ceph (e.g. RBD) ○ Provisions storage from Ceph Rook/Ceph Cluster Title MGR Mons MGR Pods Pod Storage pool Mons Mons MON Pod OSD Pod OSD Pod … OSD Pod disk disk … disk 13

Example: Provisioning and expanding cluster 1. Deploy a minimum Ceph cluster 2. Expand the cluster Rook operator disk0 disk1 14

Step1: Deploy a Minimum Cluster 1. Create Rook operator disk0 disk1 Admin kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count: 1 15

Step1: Deploy a Minimum Cluster Admin 3. Create 2. Detect Rook/Ceph cluster (0 OSD) Rook disk0 disk1 kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count: 1 16

Step1: Deploy a Minimum Cluster Admin Rook/Ceph cluster (1 OSD) Rook operator 4. Create kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count: 1 OSD0 Pod disk0 disk1 17

Step2: Expand the cluster 1. Update Rook/Ceph cluster (1 OSD) Rook operator Admin kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count: 2 OSD0 Pod disk0 disk1 18

Step2: Expand the cluster Admin 2. Detect Rook/Ceph cluster (2 OSDs) Rook operator 3. Create OSD0 Pod disk0 kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count: 2 OSD1 Pod disk1 19

Block and Filesystem Storage 20

Supported Configurations Storage Volume Mode Access Mode RBD Block, Filesystem RWO, RWOP, ROX CephFS Filesystem RWX, RWO, ROX, RWOP CephNFS The same as above The same as above 21

Additional Features Title Storage Volume Expansion, snapshot, and cloning Static provisioning QoS RBD ✅ ✅ ✅ CephFS ✅ ✅ CephNFS ✅ 22

Example: Consuming a Block Volume 1. Create an RBD pool 2. Consume a block volume Rook/Ceph cluster Ceph CSI Rook operator 23

Step1: Create an RBD Pool Admin 1. Create Rook/Ceph cluster Ceph CSI Rook operator kind: CephBlockPool metadata: name: my-pool spec: replicated: size: 3 — kind: StorageClass metadata: name: ceph-block parameters: pool: my-pool 24

Step1: Create an RBD Pool Admin Rook/Ceph cluster Ceph CSI RBD pool (3 replicas) 2. Detect 3. Create Rook operator kind: CephBlockPool metadata: name: my-pool spec: replicated: size: 3 — kind: StorageClass metadata: name: ceph-block parameters: pool: my-pool 25

Step2: Consume a Block Volume User 1. Create Rook/Ceph cluster Ceph CSI RBD pool (3 replicas) kind: PersistentVolumeClaim metadata: name: my-pvc spec: resources: requests: storage: 1Gi storageClassName: ceph-block Rook operator 26

Step2: Consume a Block Volume User PV Rook/Ceph cluster 4. Create 2. Detect Ceph CSI RBD pool (3 replicas) 3. Create kind: PersistentVolumeClaim metadata: name: my-pvc spec: resources: requests: storage: 1Gi storageClassName: ceph-block block volume Rook operator 27

Object Storage 28

How to provision and consume RGW? ● Use ObjectBucket(OB) and ObjectBucketClaim(OBC) CR ○ Similar to PV and PVC for block and filesystem storage Create Create OBC PVC User PV OB Block or filesystem storage RGW bucket User 29

Example: Consuming a Bucket 1. Create an RGW pool 2. Create a bucket 3. Consume the created bucket Rook/Ceph cluster Rook operator 30

Step1: Create an RGW Pool Admin 1. Create kind: CephObjectStore metadata: name: my-store spec: dataPool: replicated: size: 3 Rook/Ceph cluster … Rook operator — apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rgw-bucket parameters: objectStoreName: ceph-store 31

Step1: Create an RGW Pool Admin kind: CephObjectStore metadata: name: my-store spec: dataPool: replicated: size: 3 Rook/Ceph cluster RGW pool (3 replicas) … 3. Create 2. Detect Rook operator — apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-sc parameters: objectStoreName: ceph-store 32

Step2: Create a Bucket User 1. Create Rook/Ceph cluster Rook operator kind: ObjectBucketClaim metadata: name: my-bucket spec: bucketName: my-bucket storageClassName: ceph-store RGW pool (3 replicas) 33

Step2: Create a Bucket User Rook/Ceph cluster 3. Create RGW pool (3 replicas) Rook operator 2. Detect kind: ObjectBucketClaim metadata: name: my-bucket spec: bucketName: my-bucket storageClassName: ceph-store 4. Create Bucket ConfigMap (URL) Secret (Access Key Secret Key) 34

Step3: Consume the Created Bucket User 1. Create Rook/Ceph cluster Rook operator RGW pool (3 replicas) Bucket ConfigMap (URL) Secret (Access Key Secret Key) apiVersion: v1 kind: Pod metadata: name: my-app spec: containers: - envFrom: - configMapRef: name: my-bucket - secretRef: name: my-bucket 35

Step3: Consume the Created Bucket User Rook operator Rook/Ceph cluster 3. Access RGW pool (3 replicas) My-app pod 2. Use as envvars Bucket ConfigMap (URL) Secret (Access Key Secret Key) apiVersion: v1 kind: Pod metadata: name: my-app spec: containers: - envFrom: - configMapRef: name: my-bucket - secretRef: name: my-bucket 36

Another Interface to Access RGW ● OB and OBC are not the K8s official way ● Rook supports Container Object Storage Interface (COSI) ○ The K8s official way ○ Similar to CSI for block and filesystem storage ● COSI will replace OB and OBC in the future 37

Other Features 38

External Cluster ● Consume external Ceph clusters from a Kubernetes cluster Non-K8s environment K8s cluster PVC, OBCs Ceph cluster Application Pods Rook&Ceph CSI Other K8s clusters Rook/Ceph cluster 39

Remote Replications Ceph Feature Custom Resource RBD mirroring CephRBDMirror CephFS mirroring CephFilesystemMirror RGW multisite CephObjectRealm K8s cluster in region A Rook/Ceph cluster RBD pool kind: CephRBDMirror … Replicate K8s cluster in region B Rook/Ceph cluster RBD pool kind: CephRBDMirror … 40

Managed PDB Configuration ● Create PDBs for each failure domains ● Only one failure domain is allowed to be down at once ● e.g: when the failure domani is “node” 1. Create Rook Admin apiVersion: policy/v1 kind: PodDisruptionBudget spec: maxUnavailable: 1 selector: matchLabels: app: rook-ceph-osd … 2. Drain OSD0 Pod node0 3. Drain (blocked) OSD1 Pod node1 41

Administration Tools ● Toolbox Pod ○ A Pod for running arbitrary Ceph commands ● Kubectl rook-ceph krew plugin ○ Running handy Ceph operations Title By Rook’s CRs All Ceph features Features covered by Rook Admin By toolbox pod & kubect rook-ceph 42

Project Health 43

Philosophy ● Support latest Ceph and K8s ● Make Ceph the best storage platform for K8s! Title 44

Stability ● Marked as stable 6 years ago ● Many upstream users running in production ● Many downstream deployments running in production Title 45

Release Cycle ● Major version: Always “1” for now ● Minor version: Once per 4 months ● Patch version: Biweekly or on demand Title 46

Active Community ● GitHub and slack channel ● 500+ contributors to the GitHub project ○ e.g. Clyso, Cybozu, IBM/Red Hat, and Upbound ● Monthly community meeting ● CNCF Graduated project Title GitHub https://github.com/rook/rook Slack https://slack.rook.io 47

• https://github.com/rook/rook
• https://slack.rook.io

Try Rook! Containers and Helm charts Docker Hub, Quay.io, GitHub Container Registry (GHCR) Website and Docs https://rook.io 48

• https://rook.io

Questions? 49