2

I am manually running a program in a cluster by copying the executable file to each node, running it and them copying back the output to the server node. Suddenly, I noticed that a file that takes part in all this process is being deleted consistently and I cannot find out where I am doing that or at least which script is deleting it.

This is related to the question in SO: https://stackoverflow.com/questions/23608486/how-to-find-which-process-removing-a-directory-in-linux

but notice that here I can replicate the problem. I don't have admin rights so I cannot use auditd but I can repeat the process much as I want. Interestingly, I just tried changing the permissions of the file and it keeps vanishing.

I am not expecting anything magical like

$ please tell me who did

it but I am happy to listen to clever ideas on how to try to tackle this.

btw, I asked the admin whether any process might be deleting the file and he said that that is not possible.

NOTE: I originally posted this in SO but was tagged as offtopic

EDIT: Ok, from what I can see using Mark Plotnick suggestion strace, I see that first I there is a call to gcc where the .c file is compiled. Then there is a lstat on the executable. And then it appears on an execve call to /usr/bin/ld which has the PID of the unlink:

5980 execve("/usr/bin/ld", ["/usr/bin/ld", "-plugin", "/cm/local/apps/gcc/6.3.0/libexec"..., "-plugin-opt=/cm/local/apps/gcc/6"..., "-plug       in-opt=-fresolution=/tmp/cc"..., "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_"..., "-plugin-opt=-pass-through=-lc",        "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_"..., "--eh-frame-hdr", "-m", "elf_x86_64", "-dynamic-linker", "/lib64       /ld-linux-x86-64.so.2", "-o", "MY/EXECUTABLE/BEING/DELETED", "/lib/../lib64/crt1.o", "/lib/../lib64/crti.o", "/cm/local/apps/gcc/6.3.0/lib/g       cc"..., "-L/cm/shared/apps/slurm/17.02.2/"..., "-L/cm/local/apps/gcc/6.3.0/lib/g"..., "-L/cm/local/apps/gcc/6.3.0/lib/g"..., "-L/lib/../lib       64", "-L/usr/lib/../lib64", "-L/cm/shared/apps/slurm/17.02.2/"..., "-L/cm/shared/apps/slurm/17.02.2/"..., "-L/cm/local/apps/gcc/6.3.0/lib/g       "..., "/tmp/ccdL3Sax.o", "-lm", "-lgcc", "--as-needed", ...], [/* 39 vars */]) = 0

and then:

5980  stat("MY/EXECUTABLE/BEING/DELETED", {st_mode=S_IFREG|0755, st_size=17392, ...}) = 0
5980  lstat("MY/EXECUTABLE/BEING/DELETED", {st_mode=S_IFREG|0755, st_size=17392, ...}) = 0
5980  unlink("MY/EXECUTABLE/BEING/DELETED") = 0
5980  open("MY/EXECUTABLE/BEING/DELETED", O_RDWR|O_CREAT|O_TRUNC, 0666) = 3
  • Try using auditd, it can watch directory content. – atype May 7 '18 at 11:21
  • well, I said that in the question, I can't without permissions – myradio May 7 '18 at 13:11
  • 3
    If you think the deletion is the result of a command that you ran or a child process of that command, you can run strace -f -o /path/to/outputfile -e trace=file yourcommand ... and then look in the output file for anything that accesses the file you're interested in. (That is literally the word file in the argument to strace -e ) – Mark Plotnick May 7 '18 at 14:08
  • Try chmod -w on the parent directory of the file (or try asking the admin to set the immutable attribute on the file) and see if any script errors out? (Note that this is different from changing the file permissions, as it is the directory permissions that control ability to delete) – Grisha Levit May 7 '18 at 23:36
  • 2
    The ld option -o MY/EXCECUTABLE/BEING/DELETED means "write the new executable to that file". One way a program could do that would be to open the file for writing, but that might fail if the file is currently being executed, which isn't uncommon for an executable. So ld removes the file and then creates a new file with the same name. – Mark Plotnick May 9 '18 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.