Does that make sense and is it possible on a 3 node setup, how should the crush rule look like? Thats the current one. The idea behind this is to compensate multiple osd outages (for example 3 osds one per node will fail)

# rules
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit

1 Answer 1


Yes, this is possible with step chooseleaf firstn 0 type osd. If that makes sense is another question, for example your storage overhead would be very high. If you have the capacity you could do that, but ceph is designed as a highly scalable solution, with this setup you have kind of a corner case. Usually, host based replication is enough with min_size = 2 and size = 3. That way it doesn't matter how many OSDs fail on one host, and even a second host goes down your data won't be lost yet, just degraded until you bring OSDs back online. But also note that with only three hosts you wouldn't be able to recover a failed node, so the cluster would operate in a degraded state until the node is repaired or a new node joins the cluster.

You can edit your crush rule (or create a new one to test it first) so it chooses all available hosts first and then the number of OSDs, it could look like this:

step choose firstn 0 type host
step chooseleaf firstn 2 type osd

You should test it before applying with crushtool.

  • Thanks. @eblock how can I make sure that ceph does not place all 5 replicas on a single node? the rule sounds like it would just take random osds without look at which host they are.
    – jsterr
    Dec 9, 2021 at 13:08
  • whats the "2" for in the second line? @eblock Will I get 2 replicas per host then? That would be 6 replicas then on a 3 node cluster or how does it work?
    – jsterr
    Dec 9, 2021 at 13:56
  • 1
    It would try to place 6 replicas, yes, but if you set size to 5 it will stop after having placed 5 replicas. This would result in some nodes having two copies of each PG and others only one. But it would also mean that each host has at least one replica.
    – eblock
    Dec 9, 2021 at 14:01
  • Thank u very much for your help :-)
    – jsterr
    Dec 9, 2021 at 14:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .