Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
Does that make sense and is it possible on a 3 node setup, how should the crush rule look like? Thats the current one. The idea behind this is to compensate multiple osd outages (for example 3 osds one per node will fail)
# rules
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
Yes, this is possible with step chooseleaf firstn 0 type osd. If that makes sense is another question, for example your storage overhead would be very high. If you have the capacity you could do that, but ceph is designed as a highly scalable solution, with this setup you have kind of a corner case. Usually, host based replication is enough with min_size = 2 and size = 3. That way it doesn't matter how many OSDs fail on one host, and even a second host goes down your data won't be lost yet, just degraded until you bring OSDs back online. But also note that with only three hosts you wouldn't be able to recover a failed node, so the cluster would operate in a degraded state until the node is repaired or a new node joins the cluster.
You can edit your crush rule (or create a new one to test it first) so it chooses all available hosts first and then the number of OSDs, it could look like this:
step choose firstn 0 type host
step chooseleaf firstn 2 type osd
You should test it before applying with crushtool.
size to 5 it will stop after having placed 5 replicas. This would result in some nodes having two copies of each PG and others only one. But it would also mean that each host has at least one replica.