Recently I built ceph with two server configuration.

most uneasy thing is that i don't know how to check if two servers are replicated properly on ceph cluster.

Does anyone using ceph confirm that the data is replicated?


Just in case: Typically two nodes configuration is meant to be a test environment, not a production one. Two nodes cluster is more prone to such failures that might make you choose what you're going to lose: redundancy or uptime. They might also incur further data loss depending on pool's config.

Assuming a two-node cluster, you have to create pools to store data in it. There are some defaults preconfigured in ceph, one of them is your default pool size which reflects the replication size of your data. A pool size of 3 (default) means you have three copies of every object you upload to the cluster (1 original and 2 replicas). You can get your pool size with:

host1:~ # ceph osd pool get <POOL> size
size: 3
host1:~ # ceph osd pool get <POOL> min_size
min_size: 2

The parameter min_size determines the minimum number of copies in a pool and still be operable. For example, if you have both min_size and size 3, your cluster will stop the I/O to that pool if one object is in error state. If you have a configuration like above (min_size 2, size 3) your data will be processed even if one copy is not healthy. In your case you would need a pool size of 2 and min_size of 1, except if you decide to only allow writes to the pool if it's healthy, in that case 2 and 2 are recommended.

Now to validate if both your copies are alive (besides a cluster in HEALTH_OK state) you can check the following:

# Get PGs per pool
host1:~ # ceph pg ls-by-pool <POOL>
3.0          24                  0        0         0       0 100663296  84       84 active+clean 2018-09-24 10:00:31.274193   86'84   182:119 [5,7,0]          5 [5,7,0]              5      86'84 2018-09-23 10:39:06.518211             0'0 2018-09-18 14:41:06.260403 

# Get mapping of a PG
host1:~ # ceph pg map 3.0
osdmap e182 pg 3.0 (3.0) -> up [5,7,0] acting [5,7,0]

As you can see,this specific PG has three copies (size = 3) on OSDs 5, 7 and 0, and OSD.5 is the primary OSD, the one serving data to the client.

Did you build your cluster on filestore or bluestore? If you have a filestore cluster you can determine the file location of your objects within the filesystem on your server, see this section "Retrieve an object on the cluster" for an example how to retrieve that information, I don't have a filestore cluster at hand right now. On a bluestore cluster this won't work, though. It's not possible to browse through the files anymore.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.