Ceph 101
Ceph is a distributed storage system built that consists of a few daemon processes.
- OSDs - object storage daemons are the workers that actually store bytes, one per disk. your data gets broken into objects and spread across OSDs
- monitors (mons) - hold the cluster map and know which OSDs exist, which are up, where data lives.
- RADOS - reliable autonomic distributed object store. the object layer that the ceph interfaces (RBD, CephFS, RGW) is built on
- RBD images - RADOS block device images. Ceph's underlying object store. this is what backs PersistentVolumeClaims when using the RBD storage class
- manager (mgr) - runs cluster-wide services: metrics, the dashboard, and the modules that back RBD's higher-level features
- pools - named buckets of objects with a replication policy attached. nothing is stored in ceph without landing in a pool
Container Storage Interface (CSI) is basically a gRPC interface between Kubernetes and any storage system. Whether its Ceph, AWS EBS, Google Persistent Disk, Azure Disk, AWS EFS, or PortWorx, as long as they follow the CSI spec, they can integrate with Kubernetes.
To implement a CSI you have to create certain gPRC Services (functions that are called by a client, remotely). The CSI services can be broken into two sides:
- Controller services include
CreateVolume,DeleteVolume,ControllerPublishVolume,ControllerUnpublishVolume- usually served by a driver Pod (Deployment with replicas and leader election) - Node services include
NodeStageVolumeandNodePublishVolume. these are per-node operations run by a driver pod on every node (DaemonSet)
What is the client that is calling these services, you ask?
Sidecar containers running alongside the CSI Controller pod(s) and the CSI Node pod(s).
Creating a ceph-rook cluster:
Well, first, what is Rook? Rook is the Operator. It looks for and manages CephClusters.
You can get a storage cluster up and running with a couple helm charts (rook-ceph and rook-ceph-cluster)
Install the Rook operator:
helm repo add rook-release https://charts.rook.io/release
helm repo update
helm install --create-namespace --namespace rook-ceph rook-ceph rook-release/rook-ceph --version v1.15.7
This gives you:
$ k get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
rook-ceph-operator 1/1 1 1 14m
And its kinda sitting there waiting for a CephCluster.
Here's a basic CephCluster:
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
image: quay.io/ceph/ceph:v19.2.0
dataDirHostPath: /var/lib/rook
mon:
count: 3
storage:
useAllNodes: true
useAllDevices: true
You can use that manifest to install the ceph cluster helm chart:
helm install --namespace rook-ceph rook-ceph-cluster rook-release/rook-ceph-cluster --version v1.15.7 -f ceph-cluster.yaml