I have a problem with removing empty dir, strace shows error:

rmdir("empty_dir") = -1 ENOTEMPTY (Directory not empty)

And ls -la empty_dir shows nothing. So i connected to the fs (ext4) with debugfs and see the hidden file inside this dir:

# ls -lia empty_dir/
total 8
44574010 drwxr-xr-x 2 2686 2681 4096 Jan 13 17:59 .
44573990 drwxr-xr-x 3 2686 2681 4096 Jan 13 18:36 ..

debugfs:  ls empty_dir
 44574010  (12) .    44573990  (316) ..  
 26808797  (3768) _-----------------------------------------------------------.jpg  

Why could this happen? And any chance to solve this problem without unmounting and full checking fs?

Additional information:

The "hidden" file is just a normal jpg file and can be opened by the image viewer:

debugfs:  dump empty_dir/_-----------------------------------------------------------.jpg /root/hidden_file

# file /root/hidden_file 
/root/hidden_file: JPEG image data, JFIF standard 1.02

rm -rf empty_dir is not working with the same error:

unlinkat(AT_FDCWD, "empty_dir", AT_REMOVEDIR) = -1 ENOTEMPTY (Directory not empty)

find empty_dir/ -inum 26808797 shows nothing.

  • 3
    Do you have any sort of kernel-level rootkit installed? That would do all kinds of mess like this. – Mukesh Sai Kumar Jan 13 '18 at 16:47
  • it can be due to special characters into a filename if so the good methods are renaming the file if it can be done from then wildcards ones ... or in case you are lucky & just want to remove the directory trying rm -rf thatdirectory most of time does the job. else the last method is to erase file from it's inode number . find . -inum [inode-number] -exec rm -i {} \; for example – francois P Jan 13 '18 at 17:19
  • debufs show a file still registered as confirmed by @Hauke Laging – francois P Jan 13 '18 at 17:41
  • 1
    @MukeshSaiKumar This is a backup server with a minimal numbers of processes and without public network access. There are not any unknown processes, kernel modules or malicious activities. I edited question with adding additional information. – Jo Kin Jan 13 '18 at 17:54
  • 1
    @MukeshSaiKumar You was right about rootkit. I added answer. – Jo Kin Jan 15 '18 at 0:16

I straced ls and got more information to dig (stripped non-important syscalls):

getdents(3, /* 3 entries */, 32768)     = 80
write(1, ".\n", 2.)                     = 2
write(1, "..\n", 3..)                   = 3

Hmm, we see that syscall getdents works correctly and returned all 3 entries ('.','..' and '_---*'), but ls wrote only '.' and '..'. It means that we have some problem with wrapper around getdents which is used by coreutils. And coreutils use readdir glibc wrapper for getdents. Also to prove that there are no problems with getdents i tested little prog from example section of getdents' man page. This prog showed all files.

Maybe we just found a bug at the glibc? So i updated glibc package to the last version in my distro but didn't get any good result. Also i didn't find any correlating information in bugzilla.

So let's go deeper:

# gdb ls
(gdb) break readdir
(gdb) run
Breakpoint 1, 0x00007ffff7dfa820 in readdir () from /lib64/libncom.so.4.0.1
(gdb) info symbol readdir
readdir in section .text of /lib64/libncom.so.4.0.1

Wait, what? libncom.so.4.0.1? Not a libc? Yes, we just see a malicious shared library with libc functions for hiding malicious activity:

# LD_PRELOAD=/lib64/libc.so.6 find / > good_find
# find / > injected_find
# diff good_find injected_find
< /lib64/libncom.so.4.0.1
< /usr/bin/_-config
< /usr/bin/_-pud
< /usr/bin/_-minerd
< /etc/ld.so.preload

Removing rootkit files, checking all packages' files (rpm -Va in my case), auto-start scripts, preload/prelink configs, system files (find / + rpm -qf in my case), changing affected passwords, finding and killing rootkit's processes:

# for i in /proc/[1-9]*; do name=$(</proc/${i##*/}/comm); ps -p ${i##*/} > /dev/null || echo $name; done

In the end full system update, reboot and problem solved. Reason of the successful hacking: ipmi interface with very old firmware which suddenly was available from the public network.

  • Wow, you were nice enough to give a full disclosure and diagnosis here! +1 :) – Mukesh Sai Kumar Jan 15 '18 at 13:17
  • just curious, did "minerd" show up in 'ps' or any other tool? and how did you root cause it to ipmi? – rajaganesh87 Jan 18 '18 at 8:31
  • @rajaganesh87 _-minerd process was invisible for libproc based utils (ps, top, etc ...), but was visible via procfs (that's why i used for-loop with /proc/[1-9]* and ps -p for every pid for detection invisible processes). Hackers compiled malicious libproc-3.2.8.so and located it to the /lib/ dir (distro's libproc-3.2.8.so was in /lib64, but /lib dir in this distro has more priority when shared libs is loading). With hacked ipmi it's possible to reboot server. Reboot -> grub -> init=/bin/bash -> make backdoor -> reboot again to the normal system -> use backdoor for malicious activity. – Jo Kin Jan 20 '18 at 0:16

Within debugfs you can delete the file. You do not even need the file name (which may be relevant if there are problems with special chars as francois P guessed in the comments):

kill_file <26808797>
  • 1
    Is it safe to do on mounted fs? On lab environment (fresh ext4 on loop device) after kill_file for normal file and creating new file i got: touch: cannot touch '/test/3': Input/output error and EXT4-fs error (device loop2): __ext4_new_inode:1111: comm touch: failed to insert inode 13: doubly allocated? – Jo Kin Jan 13 '18 at 18:11
  • The same result with rm via debugfs on mounted fs. So its unsafe to do. – Jo Kin Jan 13 '18 at 18:18
  • @JoKin I am not a filesystem expert and thus cannot tell you if there is a way to get it really safe. I guess it is rather safe as long as the page cache does not contain the affected area. I have forgotten how to do that but IIRC you can somehow throw away the whole page cache. – Hauke Laging Jan 13 '18 at 18:18
  • @JoKin Because you asked "without unmounting and full checking fs" I assumed that umount alone (without fsck) would be OK for you. – Hauke Laging Jan 13 '18 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.