1

Running MacOS in a terminal which is essentially BSD, hence posting my question here instead of askdifferent. I want to trim my image dataset down in subsequent steps, by deleting files randomly. Some directories have over a 1 million jpgs. My data is in master directory with subdirectories that is only a maxdepth of 1:

-master
     -data1
       image.jpgs
     -data2
       image.jpgs
     -data3
       image.jpgs
     -data4
       image.jpgs
... and so forth

I found this link:

https://superuser.com/questions/1186350/delete-all-but-1000-random-files-in-a-directory

... and came up with:

for f in *.jpg; do find "$f" -type f -print0 | sort -R | tail -n +50001 | xargs -0 rm; done

While it does work, I'd like for it to do this recursively for subdirectories, so I don't have to manually do it for every directory. So my questions/requests are:

  1. Can I optimize this somehow to speed it up?
  2. Will sort/tail return an error when it encounters a directory with fewer than 50,000 files?
2
  • 2
    Does that really work? If I read the loop correctly, you're taking each jpg file, running find on that file, which should just output that file (and nothing else), running sort -R on it (which has no effect since it's just one line), followed by tail, which has no output for the same reason, and running xargs rm on no output at all, so nothing gets deleted.
    – muru
    Jun 1 '19 at 3:44
  • Crap, you're right. I assumed it was working because "something" was happening.
    – SciGuy
    Jun 1 '19 at 4:22
1

Having checked the linked source post, it looks like your loop should actually be:

for d in */; do find "$d" -iname '*.jpg' -type f -print0 | sort -zR | tail -zn +50001 | xargs -0r rm; done

To be run from the master directory.

The -z options for sort and tail are necessary, since the input is null-delimited. Neither will complain if there are fewer than 50000 lines - sort doesn't care and tail won't output anything, since there is nothing after the 50000th line. rm might complain about being run with no arguments, but the -r option to GNU xargs will prevent it from running rm if it gets no input (BSD xargs doesn't need it, but probably won't complain).

Lastly, but most importantly, the -z option for null-delimited input probably won't be supported by BSD tail. You'd need the GNU tail, which can be installed using homebrew.

You probably could do without null-delimited lines, if your filenames are guaranteed not to have spaces, newlines, quotes, backslashes, etc. in them. In that case:

for d in */; do find "$d" -type f | sort -R | tail -n +50001 | xargs rm; done
3
  • I'm guessing the same is true for head?
    – SciGuy
    Jun 1 '19 at 4:23
  • 1
    Going by the FreeBSD manpage for head, yes
    – muru
    Jun 1 '19 at 4:27
  • 3
    The xargs -r behaviour is the default on macOS and FreeBSD, but not on e.g. OpenBSD. macOS xargs additionally does not have -r and will complain if you use it. OpenBSD xargs has -r. We really need to stop lumping the BSDs together... :-/
    – Kusalananda
    Jun 1 '19 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.