17

At this time no ansver for this problem.

Usually after some problems with readings or writings to block device, kernel decides to switch flag for WHOLE DEVICE as read-only. After this any writings to any partition / filesystem located on this device cause switch it as readonly together with device state, because any writings are impossible.

Example from dmesg, this is simulation for guest linux on windows8 using VirtualBox when defrag takes guests device image:

[11903.002030] ata3.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[11903.003179] ata3.00: failed command: READ FPDMA QUEUED
[11903.003364] ata3.00: cmd 60/08:00:a8:77:57/00:00:00:00:00/40 tag 0 ncq 4096 in
[11903.003385]          res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[11903.004074] ata3.00: status: { DRDY }
[11903.004248] ata3: hard resetting link
[11903.325703] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[11903.327097] ata3.00: configured for UDMA/133
[11903.328025] ata3.00: device reported invalid CHS sector 0
[11903.329664] ata3: EH complete
[11941.000472] ata3.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[11941.000769] ata3.00: failed command: READ FPDMA QUEUED
[11941.000952] ata3.00: cmd 60/08:00:c8:77:57/00:00:00:00:00/40 tag 0 ncq 4096 in
[11941.000961]          res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[11941.001353] ata3.00: status: { DRDY }
[11941.001504] ata3: hard resetting link
[11941.320297] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[11941.321252] ata3.00: configured for UDMA/133
[11941.321379] ata3.00: device reported invalid CHS sector 0
[11941.321553] ata3: EH complete
[11980.001746] ata3.00: exception Emask 0x0 SAct 0x11fff SErr 0x0 action 0x6 frozen
[11980.002070] ata3.00: failed command: WRITE FPDMA QUEUED
[11980.002255] ata3.00: cmd 61/18:00:28:23:59/00:00:00:00:00/40 tag 0 ncq 12288 out
[11980.002265]          res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
-------------------
There are many other errors, like "lost write page", "Journal has aborted", "Buffer I/O error", "hard resetting link" and many others.

After this, remount cause:

mount / -o remount,rw
mount: cannot remount block device /dev/sda1 read-write, is write-protected

because WHOLE device sda keeping rootfs sda1 is READONLY.

In my experience this occurs in situations:

  1. HDD is really damaged. Returned writing problems are depended on HDD condition
  2. Host machine is overloaded, then linux guest virtual HDD writings are timeouted
  3. FC cable or SAN device (array disks over Fibre Channel) is overloaded
  4. Momentary lost connection over FC or FCoE. Maybe lost/timeouted FC packet

At this situations device is really read-write, but linux kernel marks this device internally as read-only and is used as read-only. This is kernel functionality maked for damage prevention, but it is useable only at 1. point.

Question is. How to manually tell to kernel, hdd block device operates normally?

Witiout this, kernel serve device as read-only, like 'CD-ROM', and no other command has chance to works properly, including mount/remount -o read-write , fsck and others.

Unusable ansvers, really qualified as spam from people who wants to help, but doesn't understand about problem nature:

  1. Try remount as read-write (impossible, device is R-O)
  2. fsck this (what for? device is R-O, no repair is possible)
  3. 'I don't know' (first with sense, but unusable)
  4. 'Replace your device' *(usually the problem is something else)

Has anybody any formula for question above? Switch flag for writeable block device that reverts it from read-only to read-write state ? At this time it seems that no-one know how.

It is some workarounds, but usually semiusable or unusable:

  1. Remove module supports access to specified hdd or storage array. Unfortunately usually damaged device keeps rootfs, or driver keeps both damaged device and device that keeps rootfs
  2. Remove FC access to device and join this again (fctools), not allways possible, not allways works.
  3. Restart WHOLE machine. Usually only this is allways possible and we allways forced to.

At points 1. and 2. we tell to kernel that we completly disconnect device and connect to it again. Kernel recognized this as joining new properly operatings device. We can simulate this using USB device and momentary remove power. Point 3. is last chance and usually works. But why we should restart all? Unfortunately at all points we lost all journals updates and dirty buffers.

Notice, at the same situations I have no problems with Windows (desktop and server).

migrated from stackoverflow.com Apr 29 '13 at 18:40

This question came from our site for professional and enthusiast programmers.

  • Not an answer, but possibly related in case of #2 (high host load, guest hdd timeout): Increase the Linux hdd timeout to prevent filesystem corruption caused by hdd timeouts in guest system. – basic6 Aug 22 '16 at 10:57
  • @Znik, are these guest virtual machines running on Citrix XenServer? Or physical hardware? Our StorageServer bridges from the land of ethernet to land of mini-sas. When this bridge machine panics, it has to be forcefully rebooted. Windows guest VMs come back. Linux guest virtual machines exhibit the same exact problem you have. Nothing suggested here brings the mount points back to rw. – rjt Jul 23 '17 at 3:42
  • @rjt, this occurs in many situations. Main situation is where device is extremally slow down with any problem, like physical damage, device overload, cabling, external FC over Eth and eth is overloaded, sometimes switch reset when transfer block, timeout, lost packet etc. Device usually is still visible, but marked as readonly. Reboot is not resolution, it is workaround as I described at the main question / problem description. – Znik Aug 4 '17 at 7:29
12

try with blockdev --setrw or hdparm -r 0

  • thanks, this should be usefull. I'm waiting for any timeout on fc controller – Znik May 6 '13 at 12:47
  • An important part that needs to be added: Sometimes it is necessary to do a fsck on the read-only file system, before it can be mounted again. – Evi1M4chine Nov 20 '16 at 16:08
  • 3
    Diddnt work for me. i have similar problem – jonneymendoza Apr 4 '17 at 18:24
  • 1
    Did not work for me even with fsck. Citrix XenServer Linux guests. – rjt Jul 23 '17 at 3:52
  • Not Working ! This commands seem effective, but the dongle is still RO. (it is software, but from where???) If you want to try, take any Debian iso 9.4. – Sandburg May 20 '18 at 13:26
5

Like Jose Luis Martin suggested use blockdev, my 2cent is to do a remount rw and forcefsck

(assuming sda is your disk)

blockdev --setrw /dev/sda
mount /dev/sda -o remount,rw
touch /forcefsck
  • 1
    It makes more sense to just run fsck before the mount, as it will fail to mount without fsck. (At least in my case it did.) – Evi1M4chine Nov 20 '16 at 16:09
  • ` # blockdev --setrw /dev/xvda1 # # touch /tmp/date +%Y%m%d-%H%M%S touch: cannot touch ?/tmp/20170722-221904?: Read-only file system # # mount -o remount,rw /dev/xvda1 [137010.709883] EXT4-fs error (device xvda1): ext4_remount:4824: Abort forced by user mount: cannot remount block device /dev/xvda1 read-write, is write-protected ` – rjt Jul 23 '17 at 3:20
2

Check this wiki page, it explains the error thrown by libata :

https://ata.wiki.kernel.org/index.php/Libata_error_messages

From what I see above, you got a timeout issue and as per the document mentioned :

Controller failed to respond to an active ATA command. This could be any number of causes. Most often this is due to an unrelated interrupt subsystem bug (try booting with 'pci=nomsi' or 'acpi=off' or 'noapic'), which failed to deliver an interrupt when we were expecting one from the hardware.

You may want to disable ACPI ( check how to based on your distro) or check you kernel for known bugs and possibly update it if it is not the latest ( or downgrade it).

  • Yes, this is really timeout. Usually this occurs on FC controller when array device is overloaded. You're right, on local ATA subsystem this is usually any hardware bug or driver/chipset implementation – Znik May 6 '13 at 12:49
  • So it's a timeout? Well, what does sudo hdparm -I /dev/sdX | grep locked say? It must say: 'not locked'. It showed these enigmatic timeouts in the past here whenever a HDD was locked by ATA password (due to a previous security erase and a system crash later which caused the security pw not to be cleared again). This password stuff really has a huge impact, also on your nerves.:) Even standard tools shipped by your HD drive vendor behave crazily, as if the HDD is about to die when the password is active. The culprit for countless tufts of hair torn out through the years. – syntaxerror Dec 3 '14 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.