0

I have directory which contains millions of files of small size (output from a big data program). I need to delete those directories but when I type the standard rm * I get:

zsh: sure you want to delete more than 100 files in data/output [yn]? y
zsh: argument list too long: rm

The files all have the same prefix with a unique number appended after like this

data-12343
data-12344
... etc

So I can't even use regular expressions to trash the files in a piecemeal way. Looking for advice and tips on how to do this efficiently and in an automated way.

Thanks.

4

Use xargs to run your rm command. It will re-run rm, using up the maximum number of arguments each time, until it has completed your original intention of *.

In the words of the man page:

The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits. Some commands can usefully be executed in parallel too; see the -P option.

  • 1
    Great. I just did ls | xargs rm and it did the trick! – M.K. Apr 28 at 12:55
  • 3
    @M.K. although that worked in this case, you shouldn't rely on the output of ls for things like this: a better option would be printf './%s\0' * | xargs -0 rm. Please refer to Why you shouldn't parse the output of ls(1) – steeldriver Apr 28 at 14:59
  • 2
    I'm glad it worked for you. Similar to @steeldriver's comment, another option would be to use find and pipe the results to xargs like so: find . -type f -print0 | xargs -0 rm – Jeff H. Apr 28 at 15:34
  • @steeldriver if rm * failed why would printf ... * work? – roaima Apr 28 at 16:06
  • @roaima I believe it's because (at least in zsh, which the OP is using, and bash) printf is a shell builtin and so not subject to the ARG_MAX limit – steeldriver Apr 28 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.