Strange behavior in mv command - maybe open sys call issue?

Question

I'm noticing very strange behavior from the "mv" command that may (or may not) be related to the "open" system call. We're running RedHat v5. There are two separate storage devices, one mounted to "/diskTo" and the other "/diskFrom" (for this example).

In normal operations, we are moving (mv'ing) hundreds, if not low thousands, of files from /diskFrom to /diskTo. The majority of the files move fine. However, out of, say 1000 files, we have 1-5 that fail. The failure is a permission denied error. When we check the file destination, a file exists, but the inode contents are garbage. For example, the timestamp is junk ("1969", but varies), and the permissions are "0."

So, I figured we should run strace on the mv commands and capture the output of the failures. Here's what I found:

munmap(0x2b0328770000, 4096)            = 0
geteuid()                               = 31169
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff90de9500) = -1 ENOTTY (Inappropriate ioctl for device)
stat("/diskTo/foo.dat", 0x7fff90de95d0) = -1 ENOENT (No such file or directory)
lstat("/diskFrom/bar.dat", {st_mode=S_IFREG|0444, st_size=234632119, ...}) = 0
lstat("/diskTo/foo.dat", 0x7fff90de9370) = -1 ENOENT (No such file or directory)
rename("/diskFrom/bar.dat", "/diskTo/foo.dat") = -1 EXDEV (Invalid cross-device link)
unlink("/diskTo/foo.dat") = -1 ENOENT (No such file or directory)
open("/diskFrom/bar.dat", O_RDONLY|O_NOFOLLOW) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=234632119, ...}) = 0
open("/diskTo/foo.dat", O_WRONLY|O_CREAT|O_EXCL, 0400) = -1 EEXIST (File exists)
write(2, "mv: ", 4)                     = 4
write(2, "cannot create regular file `/diskTo"..., 76) = 76
write(2, ": File exists", 13)           = 13
write(2, "\n", 1)                       = 1

As you can see, The unlink is called, which returns -1, which shows the file doesn't exist. Then, "mv" tries to open the file and receives an EEXIST error. But the file can't possibly exist! I'm not showing this here, but the script that is creating this test case is using unique numbers to build directories - so it's very unlikely (if not impossible) that the file truly existed. Not to mention, the unlink proves the file didn't exist.

Could this be an issue with how open is creating the inode contents? I'm not sure where to look at this point. Maybe looking into "mv" more, or the "open" system call?

Answer 1

Run a full fsck(8) on the disk, it looks like some messed up inodes are floating around (or are created on the fly). Run badblocks(8) to see if there is something wrong with the disk. See if the logs (or other disk diagnostics) say something about it.

All software up to date?

What filesystem(s) are these? Any other activity on diskTo which might be interfering? Is the disk almost full (in space, in inodes) by any chance? Any other reports in the logs?

Also do a thorough check on the machine, memory errors might cause something like this, as could overheating. This is a very unlikely cause, but it won't hurts to take it apart and check fans and so are working properly, and there isn't enough earth for a middle-sized farm inside. If the machine isn't connected to a UPS or similar, voltage fluctuations could cause random errors too.