Hot answers tagged

18

There are no guarantees. A Journaling File System is more resilient and is less prone to corruption, but not immune. All a journal is is a list of operations which have recently been done to the file system. The crucial part is that the journal entry is made before the operations take place. Most operations have multiple steps. Deleting a file, for example ...


18

a partial solution dd if=/dev/zero count=100 bs=1k of=fs.fat mkfs -t vfat fs.fat mount fs.fat /mnt ## as root # cp some file umount /mnt ## as root cp fs.fat fs.ref vi fs.ref ## change some bytes cp fs.ref fs.sampleX now you have a good fs (fs.fat) and a corrupted one (fs.ref) sudo mount -t vfat fs.ref /mnt mount: wrong fs type, bad option, bad ...


17

I haven't done much fuzz testing either, but here's two ideas: Write some zeroes into the middle of the file. Use dd with conv=notrunc. dd if=/dev/zero of=file_to_fuzz.zip bs=1 count=1 seek=N conv=notrunc Using /dev/urandom as a source is also an option. Alternatively, punch multiple-of-4k holes with fallocate --punch-hole. You could even fallocate ...


13

No. The most common type of journaling, called metadata journaling, only protects the integrity of the file system, not of data. This includes xfs, and ext3/ext4 in the default data=ordered mode. If a non-journaling file system suffers a crash, it will be checked using fsck on the next boot. fsck scans every inode on the file system, looking for blocks ...


12

If you know at some point in time the file is good, you can make a checksum of it and use it to compare later to make sure it's still whole. This is useful before transferring files between mediums or across networks. If you don't know about the good state of a file, no there is no universal way or checking for corruption. Only the specific file format in ...


9

What you should try is the following: Use file command on the archive to see if it's recognized as gzip-ped data. Run strace gunzip on the file. This will print the last bytes read from the file which might help you identify the point in file where corruption occurs. Run a debug build of gunzip under gdb. Try to correct the corrupted section (you have to ...


8

The other answers seems mostly concerned with hardware errors. Let me list some software-caused corruptions: LF replaced with CRLF. CR removed. (Even if not followed by LF) Extra Null bytes inserted. Extra Unicode "Byte Order Mark" inserted. Character set converted from UTF-8 to Latin-1 or vice versa. DOS EOF-character(#1A) deleted, even when not at End ...


7

The most common way to verify the integrity of downloaded files is to use MD5 checksums. This assumes that the site you are downloading from actually published MD5 checksums of their files. You can verify a MD5 checksum by creating your own checksum of the downloaded file and comparing it to the published checksum. If they are identical the file you have ...


6

If you're looking for advanced filesystems for general-purpose computers in the Linux world, there are two candidates: ZFS and BTRFS. ZFS is older and more mature, but it's originally from Solaris and the port to Linux isn't seamless. BTRFS is still under heavy development, and not all features are ready for prime time yet. Both filesystems offer per-file ...


6

A filesystem cannot guarantee the consistency of its filesystem if a power failure occurs, because it does not know what the hardware will do. If a hard drive buffers data for write but tells the OS that it has written the data and does not support the appropriate write barriers, then out-of-order writes can occur where an earlier write has not hit the ...


6

Suspending implies that the data you were working with is in the memory. This state will be lost if you run out of power. If you were writing a forum post, for example, it will be lost, but nothing more serious would happen. Think that it's less serious than unplugging and removing the battery, since the computer can sync before and even delay the suspension ...


5

(I know this is an old question, I came across this problem myself and got my FS back to life without ddrescue, so I'll share the expericence for anyone else encountering this) Ext filesystems store backups of the superblock -- for an occasion just like this one. First, determine the locations of the backups: mke2fs -n /dev/sdxx This is a test run (i.e. ...


5

rsync can be used to copy directories, and is capable of restarting the copy from the point at which it terminated if any error causes the rsync to die. Using rsync's --dry-run option you can see what would be copied without actually copying anything. The --stats and --progress options would also be useful. and --human-readable or -h is easier to read. ...


5

No, there aren't any general solutions. The only way to check if a file is corrupt is to try and read it; only software which knows how to read that particular format can do that. What you could do is use file to identify the type of the file, and then use the type to choose an appropriate program to check the file. You could write a script like this: # ...


5

Your device has exactly 4294967296 blocks, which is 232, so this smells like a variable-size problem... If you’re running a 32-bit e2fsck, that could explain the error message; the error you’re seeing comes from e2fsck/super.c: check_super_value(ctx, "blocks_count", ext2fs_blocks_count(sb), MIN_CHECK, 1, 0); where check_super_value() is ...


4

If you ran sudo init 6 in the system inside the VirtualBox and that system had corrupted files, then no, init 6 was not the cause of it. There are plenty of other possible causes though, that you may/should investigate. If you ran sudo init 6 in the host OS, and the system inside VirtualBox had file system corruption, then yes, init 6 was the cause of that. ...


4

It's a sparse file. You should consider using -S, so that the file is handled as properly as possible.


4

It looks like the source filesystem is damaged, typically either due to a kernel bug or to bad RAM (a damaged disk is more likely to result in unreadable files than corrupted data). At this point, all bets are off. However, if the corruption was very localized, it's only that one file's inode that's corrupted, and other files are undamaged, so you can safely ...


4

If one could that easily detect when sectors are about to go bad or do go bad, it would likely have been worked into the filesystem by now. Due to the nature of the error, it will often be silent. You need a filesystem that does checksumming. On GNU/Linux BTRFS may be a good bet since I looked online and apparently support was introduced in Debian 6. ...


4

Use dd to truncate the file, or try a binary editor like hexer to edit and introduce some corruptions. Example of truncating file using dd Create 5MB file # dd if=/dev/zero of=foo bs=1M count=5 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0243189 s, 216 MB/s # ls -l foo -rw-r--r-- 1 root root 5242880 Aug 12 20:13 foo # Truncate 10 ...


3

I suggest inspecting the file itself and seeing if it is actually valid and consistent. cksum computes a CRC. There is a purpose-specific instruction for this on modern x86 CPUs which may or may not be in use here; in that case it is possible that the CPU might be faulty, and also possible that this fault might not show up anywhere else. Consider ensuring ...


3

There might still be hope, but your drive seems to have hardware problems (my interpretation of the read error in dmesg output). You should try to make a copy of what is recoverable from that partition onto another drive (to minimize disc access). Use ddrescue for that, it might take a while but gets most if not all of the recoverable data of the partition. ...


3

ZFS has by default multiple copies of every meta data block. You can enable this feature for data blocks and then have some protection against (localized and non massive) disk errors. http://blogs.oracle.com/bill/entry/ditto_blocks_the_amazing_tape Automatic ZFS Snapshots are also a popular way to protect files against accidental deletion or corruption.


3

You seem to forget to unmount the pen drive before unplugging. The data is being written at hte time when you're physically detaching the drive; this results in a corruption. In Nautilus, locate the drive in the left panel, click the eject button near its name. Alternatively, find it on the desktop, right-click, choose 'Eject'. This is the same thing as ...


3

After investigation (see the comments in the question), it appeared that the "corrupted" files were in fact empty. This can happen when a downloading program create the entries in the filesystem but fails before having downloaded their content. To look for them in the current directory and its subdirectories and move them to a directory called trash in your ...


3

The return code of the command used to download the file will tell you if the command executed successfully or not. Typically, a return code of 0 denotes success and any non-zero number denotes an error. You can access the return code through the $? variable. A basic example using wget would go: #!/bin/bash wget foo.tgz &> /dev/null if [[ "$?" ...


3

It is possible in theory: the data+parity gives you three opinions on what the data should be; if two of them are consistent, you can assume the third is the incorrect one and re-write it based on the first two. Linux RAID6 does not do this. Instead, any time there is a mismatch, the two parity values are assumed to be incorrect and recalculated from the ...


3

You can install and configure the SMART monitoring tools. On Debian the package is called smartmontools. These won't prevent disk failure but they will help identify precursors to possible disk failure. There is no configuration in the package installation, so you need first to enable SMART monitoring in the file /etc/default/smartmontools: # uncomment to ...


2

The answer is in most cases no: As already mikel said, most journaling file systems can only protect file metadata (information like the name of a file, its size, its permissions, etc.), not file data (the file's contents). This is happening because protecting file data results in a very slow (in practice useless) file system. Since the journal is also a ...


2

ZFS, which is close but not exactly a journaling filesystem, is guaranteeing by design against corruption after a power failure. It doesn't matter if an ongoing write is interrupted in the middle as in such case, its checksum will be certainly incorrect so the block will be ignored. As the file system is copy on write, the previous correct data (or ...



Only top voted, non community-wiki answers of a minimum length are eligible