Using ceph storage directly for docker volumes

Question

Does Docker or Kubernetes support Ceph storage interfaces (objects, red, cephfs)?

Answer 1

You can easily find the answer to this question with a quick google, but I will give the answer here anyway, for future readers.

There is a good page on ceph's website describing how to do this for docker. To quickly summarise, you need to your distributions versions of golang, librados-dev, librbd-dev, ceph-common and xfsprogs, set up go, run go get github.com/yp-engineering/rbd-docker-plugin, run rbd-docker-plugin --create --user=<ceph-user> --pool=<ceph-pool> as root, and then you can use the rbd volume driver.

Kubernetes also has a page on volume mounting. I used that and this

Run the following on a ceph monitor:

ceph auth get-key client.admin | base64 # admin key

ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=<poolname>'
ceph auth get-key client.kube | base64 # client key

On a kubernetes admin, run the following

kubectl create secret generic ceph-admin-secret --type="kubernetes.io/rbd" \
--from-literal=key='<admin key>' \
--namespace=kube-system

kubectl create secret generic ceph-client-secret --type="kubernetes.io/rbd" \
--from-literal=key='<client key>' \
--namespace=kube-system

cat << EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: <storage class name you want>
provisioner: kubernetes.io/rbd
parameters:
  monitors: <ceph monitor ip>:<ceph monitor port>
  adminId: client.admin
  adminSecretName: ceph-admin-secret
  adminSecretNamespace: kube-system
  pool: kube
  userId: client.kube
  userSecretName: ceph-client-secret
  fsType: <filesystem type>
  imageFormat: <image format>
  imageFeatures: <image features>
EOF