Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm noticing very strange behavior from the "mv" command that may (or may not) be related to the "open" system call. We're running RedHat v5. There are two separate storage devices, one mounted to "/diskTo" and the other "/diskFrom" (for this example).

In normal operations, we are moving (mv'ing) hundreds, if not low thousands, of files from /diskFrom to /diskTo. The majority of the files move fine. However, out of, say 1000 files, we have 1-5 that fail. The failure is a permission denied error. When we check the file destination, a file exists, but the inode contents are garbage. For example, the timestamp is junk ("1969", but varies), and the permissions are "0."

So, I figured we should run strace on the mv commands and capture the output of the failures. Here's what I found:

munmap(0x2b0328770000, 4096)            = 0
geteuid()                               = 31169
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff90de9500) = -1 ENOTTY (Inappropriate ioctl for device)
stat("/diskTo/foo.dat", 0x7fff90de95d0) = -1 ENOENT (No such file or directory)
lstat("/diskFrom/bar.dat", {st_mode=S_IFREG|0444, st_size=234632119, ...}) = 0
lstat("/diskTo/foo.dat", 0x7fff90de9370) = -1 ENOENT (No such file or directory)
rename("/diskFrom/bar.dat", "/diskTo/foo.dat") = -1 EXDEV (Invalid cross-device link)
unlink("/diskTo/foo.dat") = -1 ENOENT (No such file or directory)
open("/diskFrom/bar.dat", O_RDONLY|O_NOFOLLOW) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=234632119, ...}) = 0
open("/diskTo/foo.dat", O_WRONLY|O_CREAT|O_EXCL, 0400) = -1 EEXIST (File exists)
write(2, "mv: ", 4)                     = 4
write(2, "cannot create regular file `/diskTo"..., 76) = 76
write(2, ": File exists", 13)           = 13
write(2, "\n", 1)                       = 1

As you can see, The unlink is called, which returns -1, which shows the file doesn't exist. Then, "mv" tries to open the file and receives an EEXIST error. But the file can't possibly exist! I'm not showing this here, but the script that is creating this test case is using unique numbers to build directories - so it's very unlikely (if not impossible) that the file truly existed. Not to mention, the unlink proves the file didn't exist.

Could this be an issue with how open is creating the inode contents? I'm not sure where to look at this point. Maybe looking into "mv" more, or the "open" system call?

share|improve this question
1  
I would just contact rhel support as this is a strange issue –  Ulrich Dangel Apr 9 '13 at 12:15
    
Do you have any non standard option for the target file system? I am referring to journaling options specially. –  eppesuig Apr 9 '13 at 13:35
    
@eppesuig: I will check that when the file system is back online. Currently running badblocks. I also posted some information in a comment to vonbrand. –  Jmoney38 Apr 9 '13 at 14:23
    
@Jmoney38 you may probably simply check /etc/fstab. –  eppesuig Apr 9 '13 at 14:29
    
Looks to be defaults: ext4 defaults 0 0 –  Jmoney38 Apr 9 '13 at 14:33

1 Answer 1

Run a full fsck(8) on the disk, it looks like some messed up inodes are floating around (or are created on the fly). Run badblocks(8) to see if there is something wrong with the disk. See if the logs (or other disk diagnostics) say something about it.

All software up to date?

What filesystem(s) are these? Any other activity on diskTo which might be interfering? Is the disk almost full (in space, in inodes) by any chance? Any other reports in the logs?

Also do a thorough check on the machine, memory errors might cause something like this, as could overheating. This is a very unlikely cause, but it won't hurts to take it apart and check fans and so are working properly, and there isn't enough earth for a middle-sized farm inside. If the machine isn't connected to a UPS or similar, voltage fluctuations could cause random errors too.

share|improve this answer
1  
fschk came back with no issues in 37k files and X blocks (large number, can't remember off hand). Badblocks is currently running. The kernel and software is all up to date for the most part - this is a new system being built, so all is less than 1 month old. The fs is ext4. One point worth mentioning - we use multipath'ing between the server and the storage. I'm not entirely familiar with mp. The disk is nowhere near full, and we're using <2% of available inodes. There are reports in the messages log related to multipathd which I'm currently investigating. Will report back. –  Jmoney38 Apr 9 '13 at 14:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.