5

I've installed openSUSE 11.2 (64bit) a while ago, and am mainly using it as a dedicated Apache hosting machine.

Everything has been working properly for a while (~1 month), but as of a few days back, the filesystem seems to have been turned to read-only (!?).

Pretty much every task fails, be it starting vi or even man pages (can't create files on disk).

Has anyone seen something like this before ? If so, what did you do to recover? I tried googling for solution but failed to find a helpful answer.

Nota bene - no significant changes were made in this period, and machine load is very, very lightweight.

Improve this question
1
  • To answer such questions, more details are needed; otherwise it's just wild guessing. Typically remounting a filesystem read-only is a safety precaution when inconsistencies on the filesystem were found. The syslog will be explicit about that, but details are missing in the question. Maybe you just turned o ff the computer instead of shutting it down, or you have defective RAM, or whatever... More details are needed to answer!
    – U. Windl
    May 3 at 8:54

2 Answers 2

Reset to default
5

It could be a file system or disk failure. Check dmesg and system logs for any clues. Look there before you reboot if you haven't already. If you have rebooted does the system come up clean or warn you about file system problems?

You can remount the file system read-write using mount -o remount,rw / but I don't recommend doing that until you know why it mounted itself ro.

It possible to hit a keyboard shortcut that would remount the root file system read-only. Usually this is AltSysRequ. Did you or your cat perchance do that?

Did you initiate a shutdown request and then abort it? The shutdown scripts usually remount the system ro towards the end of the process, but I've seen them get borked and skip to the end :)

Improve this answer
7
  • Hi Caleb... Neither me, nor my cat hit the alt+sysreq+u. Nor did I initiate the shutdown and aborting it. Like I said, I pretty much did nothing on the server for the past month (just occasionally stopping and starting a web service thru Apache Tomcat manager). Also, invoking dmesg failed. Can you tell me why is remounting the filesystem to r-w not recommended at this stage?
    – Jas
    Jul 6, 2011 at 12:26
  • Invoking dmesg failed? That's really weird. What's the error message?
    – alex
    Jul 6, 2011 at 12:39
  • @alex - the error says : -bash: /bin/dmesg: Input/output error
    – Jas
    Jul 6, 2011 at 12:48
  • 3
    Then you have a failing disk drive most probably.
    – alex
    Jul 6, 2011 at 15:58
  • @Jas: I don't think there is much other sluth work you can do if you can't READ log files either which is what what seems to be happening. I would reboot this machine into a recovery disk of some kind and run a file system check, and also check for SMART messages or other indications of drive failure. Also turn OFF any backup systems that are not incremental because you may have seriously corrupt data already and you don't want to backup bad stuff over good.
    – Caleb
    Jul 6, 2011 at 18:22
1

I have seen this before. It has happened to us a few times.

We run VMware ESXi systems with back-ended to SAN storage. When we experienced high I/O on the SAN not related to that machine this increased the latency of the storage and essentially made the disc unavailable for a period of time. The system responded by making various volumes read-only. We could run a remount and it would complete successfully but, it would not actually remount the disk. The only way to get it back was to reboot and either go to single user mode and fsck the disk or sometimes the automatic fsck on reboot would catch it.

Our fix was to modify the disk timeouts for the block devices in /sys

# cat /sys/block/sd*/device/timeout

60

We increased the value of this number from 60 to 180 this modifies the time before a "timeout" occurs, essentially allowing you to suffer more latency. But, a side effect is that you can incur corruption in your filesystem especially those with high writes. This is also why if you separate your primary mounts (like we do) onto separate mounts or drives that volumes like /var will be read-only because there is more activity.

Improve this answer
1
  • Probably you'll have to do "root cause analysis", meaning: Don't fight the symptoms (filesystem being read-only), but fight the causes (I/O timeouts)!
    – U. Windl
    May 3 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.