1

Ceph documentation (http://docs.ceph.com/docs/jewel/start/hardware-recommendations/) puts an emphasis on the need for more powerful hardware as the cluster scales.

For instance hardware specs for ceph-mon call for:

Processor

1x 64-bit AMD-64/i386
1x 32-bit ARM dual-core or better
1x i386 dual-core

RAM

1 GB per daemon

Disk Space

10 GB per daemon

Network

2x 1GB Ethernet NICs

I assume that the per daemon figure means per OSD. If I have several JBODs with 12 bays at 4TB each this number gets large fairly quickly.

If I wanted to use a cluster of RPIs for the task, could I distribute the load across many to match this requirement or is the requirement that each ceph-mon node be able to match these minimums?

ceph-mds has even more stringent requirements, could it also be served by many smaller boards (such as an odroid XU4 possibly)?

1 Answer 1

0

I can't say how a RPi will work in a Ceph cluster, but if it helps I can give you some numbers from a production cluster. It has 3 MONs, 25 OSDs (4 to 6 OSDs per node). The MONs are deployed on OSD nodes, as are two of three MDS (we are currently outsourcing the MDS to dedicated servers).

Although I don't have performance data from a recovery (very I/O intense) available right now I have data from a healthy cluster. The MON values are quite stable over time, 751804 B (VIRT) and 339848 B (RSS). So the memory consumption is not very high, 1 GB of RAM should be sufficient, even a RPi would probably be able to handle that.

Looking into MDS, that's hard to tell and totally dependent on your requirements and workloads. With only a few clients 1 to 2 GB RAM should suffice, but this can increase rapidly. Just to give you a number: for 50 clients (or more precisely client connections) the active MDS has a mds_cache_memory_limit of 6 GB, this has been increased just recently from 4 GB. The current usage is at 7,7 GB (VIRT) 6,9 GB (RSS), MDS tends to use a little bit more than it's configured to. The MDS can be run with multiple active MDS daemons, read this for more information on multi-MDS setup. It says:

Adding more daemons may not increase performance on all workloads. Typically, a single application running on a single client will not benefit from an increased number of MDS daemons unless the application is doing a lot of metadata operations in parallel. Workloads that typically benefit from a larger number of active MDS daemons are those with many clients, perhaps working on many separate directories.

You can spread the load of your OSDs by running many nodes with few OSDs, but I think the RAM numbers for OSDs are not really suitable for RPi. On a Bluestore OSD (with journal on SSD) it consumes about 4 GB (VIRT) and 2.8 GB (RSS), so this doesn't cover the mentioned 1 GB per daemon; again, this is all dependent on the actual workload.

I would encourage you to build your cluster and watch the performance data while increasing workload, adding OSDs, adding CephFS clients etc. You'll get a feeling pretty quickly for the limits of your cluster and if tuning is possible.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .