LVM volume on USB drive fails: Read-only file system

Question

This is on a Raspberry Pi running Raspbian 8 (Linux kernel 4.9.) I have a 64 GB USB pen drive plugged (as /dev/sda) and configured as an LVM physical volume. There's only one volume group ("data") and one logical volume (also "data"), which is formatted as ext4. I automount this volume in fstab. (To be clear: /dev/sda is the physical volume, not /dev/sda1 - there are no partitions on the device.)

This has worked fine until recently, when I noticed that the logical volume was inaccessible. I've rebooted and conclude that LVM fails to start. systemctl status lvm2 reveals the following:

Aug 20 13:27:54 rasputin lvm[1747]: device-mapper: reload ioctl on  failed: Read-only file system_dmesg** from boot:

[    4.963170] sd 0:0:0:0: [sda] Attached SCSI removable disk
[    5.896509] systemd-udevd[126]: starting version 215
[    6.778455] sd 0:0:0:0: [sda] 126058496 512-byte logical blocks: (64.5 GB/60.1 GiB)
[    6.781190] sd 0:0:0:0: [sda] Write Protect is on
[    6.781219] sd 0:0:0:0: [sda] Mode Sense: 43 00 80 00
[    7.000522] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: [email protected]
[    7.139872] device-mapper: table: 254:0: linear: Device lookup failed
[    7.139890] device-mapper: ioctl: error adding target to table

My interpretation is that there is an issue with the /dev/sda physical volume, that has caused it to become read-only. From there on, I'm not very sure how to proceed, so it turns into experimentation.

I've tried fsck /dev/sda with no apparent improvement, which perhaps makes sense, since I suppose /dev/sda doesn't contain a filesystem in the normal sense. I've also tried forcing the device to turn off read-only mode with hdparm -r0 /dev/sda, which does not change the error message from trying to start the lvm2 service.

Trying to get into more LVM-specific tools, pvck seems to confirm the volume isn't doing great, but that's about it:

root@rasputin:~# pvck -v /dev/sda
    DEGRADED MODE. Incomplete RAID LVs will be processed.
    Scanning /dev/sda
  Found label on /dev/sda, sector 1, type=LVM2 001
  Found text metadata area: offset=4096, size=1044480

So that's where I am, basically. What else can I try to diagnosticise and/or fix tihs issue? The data on the disk is expendable, but I'd still like to recover it if possible, partly for convenience and partly to explore the available options for future use.

It's worth noting that (to my knowledge) I didn't do anything in particular to make this happen. Googling for the error messages throw up a lot of results where people tried to run LVM commands. I did not, it was working for maybe a month, over reboots, and then one day it wasn't.

I suppose the USB drive could be corrupted but it's almost brand new. Either way that would be the last assumption rather than the first.

Very grateful for any suggestions or assistance!

Answer 1

There is an issue with some pen drives (happened to me with a Sandisk) which when exposed to a power fluctuation go to a 'lockdown' which is intended (apparently) to protect the data.

Formatting the USB under Linux or Win was not possible.

I spoke to the manufacturer and they replaced the drive FoC.

Answer 2

It's rather peculiar that the dmsetup command fails; the 254:0 means it is your first device-mapper device. This command would include your origin device (/dev/sda) and a few parameters (block range of the linear target) which, if it is getting executed at all, would seem to indicate that your PV has sufficient configuration of its VG, which contains the entire configuration of all your LVs.

So, your backup in /etc/lvm/backup/data (== file) should contain your entire, text-based volume group configuration. If anything is actually corrupted, it should be corrupted there (You can generate a new one with vgcfgbackup) -- and older backups are supposed to be in /etc/lvm/archive/.

However since it is already trying to map your LV I suspect that the configuration is not corrupt, but you can check; and that the problem is hardware.

Regardless, your PV header is sitting in sector 2 of the USB stick with the VG header I believe down below, and something could have changed, but.........

Your safest bet is to copy everything off your stick using dd (or ddrescue) onto a different stick and see what happens. Don't insert them at the same time of course.