Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've installed openSUSE 11.2 (64bit) a while ago, and am mainly using it as a dedicated Apache hosting machine.

Everything has been working properly for a while (~1 month), but as of a few days back, the filesystem seems to have been turned to read-only (!?).

Pretty much every task fails, be it starting vi or even man pages (can't create files on disk).

Has anyone seen something like this before ? If so, what did you do to recover? I tried googling for solution but failed to find a helpful answer.

Nota bene - no significant changes were made in this period, and machine load is very, very lightweight.

share|improve this question

2 Answers 2

up vote 2 down vote accepted

It could be a file system or disk failure. Check dmesg and system logs for any clues. Look there before you reboot if you haven't already. If you have rebooted does the system come up clean or warn you about file system problems?

You can remount the file system read-write using mount -o remount,rw / but I don't recommend doing that until you know why it mounted itself ro.

It possible to hit a keyboard shortcut that would remount the root file system read-only. Usually this is AltSysRequ. Did you or your cat perchance do that?

Did you initiate a shutdown request and then abort it? The shutdown scripts usually remount the system ro towards the end of the process, but I've seen them get borked and skip to the end :)

share|improve this answer
    
Hi Caleb... Neither me, nor my cat hit the alt+sysreq+u. Nor did I initiate the shutdown and aborting it. Like I said, I pretty much did nothing on the server for the past month (just occasionally stopping and starting a web service thru Apache Tomcat manager). Also, invoking dmesg failed. Can you tell me why is remounting the filesystem to r-w not recommended at this stage? –  Jas Jul 6 '11 at 12:26
    
Invoking dmesg failed? That's really weird. What's the error message? –  alex Jul 6 '11 at 12:39
    
@alex - the error says : -bash: /bin/dmesg: Input/output error –  Jas Jul 6 '11 at 12:48
1  
Then you have a failing disk drive most probably. –  alex Jul 6 '11 at 15:58
    
@Jas: I don't think there is much other sluth work you can do if you can't READ log files either which is what what seems to be happening. I would reboot this machine into a recovery disk of some kind and run a file system check, and also check for SMART messages or other indications of drive failure. Also turn OFF any backup systems that are not incremental because you may have seriously corrupt data already and you don't want to backup bad stuff over good. –  Caleb Jul 6 '11 at 18:22

I have seen this before. It has happened to us a few times.

We run VMware ESXi systems with back-ended to SAN storage. When we experienced high I/O on the SAN not related to that machine this increased the latency of the storage and essentially made the disc unavailable for a period of time. The system responded by making various volumes read-only. We could run a remount and it would complete successfully but, it would not actually remount the disk. The only way to get it back was to reboot and either go to single user mode and fsck the disk or sometimes the automatic fsck on reboot would catch it.

Our fix was to modify the disk timeouts for the block devices in /sys

# cat /sys/block/sd*/device/timeout

60

We increased the value of this number from 60 to 180 this modifies the time before a "timeout" occurs, essentially allowing you to suffer more latency. But, a side effect is that you can incur corruption in your filesystem especially those with high writes. This is also why if you separate your primary mounts (like we do) onto separate mounts or drives that volumes like /var will be read-only because there is more activity.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.