3

We have a task to resize the filesystem from 8T to 7T which is in LVM. We have given resize2fs command, while the command is still being run we have got the error "Write failed: Connection reset by peer", However when we logged in again, the resize2fs command is still running the process list. After the process got completed, we have reduced the LVM using "lvreduce" and mounted it back.

Now in df -h it is showing still 8TB, however in LVM it is showing 7TB. How to rectify the issue?

0

1 Answer 1

6

resize2fs probably didn't finish the job, but you can't tell because you missed the end of its output.

You should not have gone ahead and executed lvreduce at that point. There is a very good chance that this corrupted part of your filesystem. Note that you cannot undo this operation by running lvextend and hoping that the lost bytes come back and that the filesystem can recover, because lvextend may give you different bytes back.

If I had to guess, what probably happened is that your connection was reset during a point when resize2fs was working for some time without emitting any output. After the connection was reset, it kept on running for a while (it ignored SIGHUP or never received it?). But later on, probably when it was almost finished, it emitted some status output and was promptly killed because it got an error or signal trying to write to a non-existent terminal. And it never completed.

What state resize2fs left the filesystem in is an open question, but one can hope that it left it in a fairly usable state, with the filesystem size reduction uncommitted.

Moral of this part of the story: execute critical operations like resize2fs under screen (or as at jobs, or whatever) if there is any chance that your connection might be interrupted.

No matter what you do, make sure you have a good backup, because anything you might attempt at this point is dangerous. IF you do not have a good backup, you can mount the filesystem read-only (do not mount it read+write as this could exacerbate the damage) and take one now.

The safest is to just delete the filesystem and recover from backup. However, should you want to attempt to recover the filesystem in place, you will probably want to execute some combination of these two tasks:

  • fsck the filesystem as-is. This will notify you of any attempts to read beyond the end of the filesystem and maybe give you an idea of how much of the data lies out beyond those bounds. Hopefully not much, as resize2fs may have succeded in moving that data before it crashed.
  • re-extend the block device using lvextend to the size it used to be, just to make the filesystem happy, then fsck, then try the whole operation again from square one. This could potentially introduce some silent corruption as the filesystem accesses byte ranges that lie beyond the range where valid data lie.

I am not sure what exactly would work best for you. It depends on the layout that things happened to have on the filesystem and on how much it has been touched since the event (you mention that you have mounted it, probably read+write, at least once...)

2
  • Probably i should go with below POA. Please comment ------------------------------------- umount <mnt> lvremove <lv> lvs vgs lvcreate -L <<8T>> -n <</dev/vg/lv>> mkfs.ext3 <</dev/vg/lv>> mount <<mnt> Finally restore the data -------------------------------------
    – xrkr
    Commented Apr 23, 2016 at 13:53
  • Yes, that is safest, except make it 7T, not 8T, since that is your goal
    – Celada
    Commented Apr 23, 2016 at 19:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .