0

I have a Ceph node with 2 disks (SSD 500G).

/dev/sda is used for the OS (Debian 11)

I want to install the second SSD as a OSD for Ceph. But Ceph does not recognize my 2nd disk.

Calling

$ sudo cephadm ceph-volume inventory

results in an empty list - no disks recognized. But the command lsblk shows that the disk exists:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:16   1 476.9G  0 disk 
├─sda1   8:17   1    32G  0 part [SWAP]
├─sda2   8:18   1     1G  0 part /boot
└─sda3   8:19   1 443.9G  0 part /
sdb      8:0    1 476.9G  0 disk

So it looks to me that something with my partitions or logical volumes is totally wrong.

How can I setup my second disk (/dev/sdb) to be recognized as a LVM by Ceph?

I verified the status in the following ways:

1.) check if mounted:

# umount /dev/sdb
umount: /dev/sdb: not mounted.

2.) check whether the device is using LVM or has a filesystem:

$ sudo fdisk /dev/sdb -l
Disk /dev/sdb: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: Micron_1100_MTFD
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
7
  • After a very quick scan it seems you haven't told Ceph about your available disk. I'm not familiar with Ceph but I'd suggest docs.ceph.com/en/quincy/cephadm/services/osd/… Dec 5, 2022 at 18:00
  • Running $ sudo ceph orch device ls does only show the disk with OS, not the other disk.
    – Ralph
    Dec 5, 2022 at 18:45
  • The very next step tells you how to provision a disk. It also lists the requirements for a new disk (including "The device must not have any LVM state"). Have you ever provisioned this new disk for Ceph? Dec 5, 2022 at 19:20
  • Can you please explain how you would verify if a Disk fulfills the requirements - e.g. no partitions, no LVM state, not contain a file system and not contain a Ceph BlueStore OSD?
    – Ralph
    Dec 5, 2022 at 20:43
  • I added into my question two additional checks to verify the status of the device.
    – Ralph
    Dec 5, 2022 at 21:01

1 Answer 1

1

The instructions for configuring Ceph storage state that a disk must not be in use. Specifically,

A storage device is considered available if all of the following conditions are met:

  • The device must have no partitions.
  • The device must not have any LVM state.
  • The device must not be mounted.
  • The device must not contain a file system.
  • The device must not contain a Ceph BlueStore OSD.
  • The device must be larger than 5 GB.

Ceph will not provision an OSD on a device that is not available.

So, you need to ensure your disk satisfies all these criteria. One way of doing this is to erase the partition table at the beginning of the disk:

# Destroy the partition table (all data!) on sdb
dd bs=1M count=1 </dev/zero >/dev/sdb

Now restart your machine and fix any errors that may be reported as a consequence of /dev/sdb now no longer being configured.

Finally you can follow the configuration instructions for Ceph regarding the set up of a new disk. This should show what would happen were you to add in /dev/sdb on the current host (remove --dry-run to execute):

ceph orch daemon add osd "$(hostname)":/dev/sdb --dry-run

Please note that I don't have Ceph installed. If this answer doesn't help you I'll delete it.

3
  • I tried the installation several times with different servers. It seems to be the SSD hardware which can not be detected by cephadmin for some reasons. I install a blank OS and installed just cephadm. And than wen I call cephadm ceph-volume inventory no devices are listed. On the other system I can see them. Normally I can see the devices even if they are in use and need to be zaped. In any case ceph-volume inventory should show the devices if they are not mounted. But this is not the case on some of my hardware. The hardware is hosted at hetzner.de but they did not support this issue :(
    – Ralph
    Dec 6, 2022 at 12:30
  • Have you erased the partition table on sdb? Dec 6, 2022 at 15:30
  • Yes I have erased the disk with zap and dd. It seems to be an issue between hardware and the ceph-volume tool. ceph-volume normally also shows other disks e.g. used for OS. But in my case, the tool doesn't show one device at all.
    – Ralph
    Dec 6, 2022 at 21:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .