11

What's a fast and not too complicated way to delete all files in a directory that are under x lines long, in bash?

12

Here is a POSIX solution that should be pretty straightforward to understand:

find . -type f -exec awk -v x=10 'NR==x{exit 1}' {} \; -exec echo rm -f {} \;

As in Stephane's answer, remove the echo when happy with what will be removed.


Explanations, written for those totally new to Unix/Linux:

The dot . represents the current directory. find finds files and directories recursively within ., and can do things with them.

-type is one of find's primaries; it is a test that will be performed for each file and directory that is recursively found (within .), and the rest of the primaries on the line are only evaluated if this results in "true."

In this particular case, we only continue if we are dealing with a regular file, not a directory or something else (e.g. a block device.)


The -exec primary (of find) calls an external command, and only proceeds to the next primary if the external command exits successfully (exit status of "0"). The {} is replaced with the file name being "considered" by the find command. So the first -exec call is equivalent to the following shell command, executed for each file in turn:

awk -v x=10 'NR==x{exit 1}' ./somefilename

Awk is an entire language in itself, designed for handling delimited text files such as CSVs. The Awk conditionals and commands (which are contained between single quotes and start with the letters NR) are executed for every line of a text file. (Implicit looping.)

To learn Awk fully, I highly recommend the Grymoire Tutorial, but I will explain the Awk features used in the above command.


The -v flag to Awk allows us to set an Awk variable (once) before the Awk commands are executed (for each line of the file.) In this case we set x to 10.


NR is a special Awk variable referring to the "Number of the current Record." In other words, it is the line number we are looking at in any particular pass through the loop.

(Note that it is possible, though unusual, to use a different "Record Separator" than the default of a newline character, by setting RS. Here is an example of playing with record separators.)


Awk scripts in general consist of conditions (outside curly braces) combined with actions (inside curly braces.) There can be compound conditions and compound actions, and there is a default condition (true) and a default action (print), but we needn't bother with those.

The condition here is, "Is this the 10th line?" If this is the case, we exit with a non-zero exit status, which in shell scripting means "unsuccessful command termination."

Thus the only way this Awk command will exit successfully is if the end of the file is reached before the 10th line is reached.

So if the Awk script exits successfully, it means you have a file of less than ten lines.


The next -exec call (if you remove the echo) will remove each file (that gets that far in evaluation of find's primaries) by running:

rm -f ./somefilename
5

Assuming a find implementation that supports the -readable predicate (if your find doesn't support it, just remove it, you'll just get error messages for non-readable files, or replace with -exec test -r {} \;):

x=10 find . -type f -readable -exec sh -c '
  for file do
    lines=$(wc -l < "$file") && [ "$((lines))" -lt "$x" ] && echo rm -f "$file"
  done' sh {} +

Remove the echo if happy.

That's not particularly efficient in that it counts all the lines in every file while it only needs to stop at the xth one and it runs one wc (and potentially one rm) command for each file.

With GNU awk, you can make it a lot more efficient with:

x=10
find . -type f -readable -exec awk -v x="$x" -v ORS='\0' '
  FNR == x {nextfile}
  ENDFILE {if (FNR < x) print FILENAME}' {} +|
  xargs -r0 echo rm -f

(again, remove echo when happy).

The same with perl:

x=10 find . -type f -readable -exec perl -Tlne '
  if ($. == $ENV{x}) {close ARGV}
  elsif (eof) {print $ARGV; close ARGV}' {} +

Replace print with unlink if happy.

6
  • 1. What's the last sh for? 2. Is wc -l < "$file" faster than wc -l "$file"? 3. How does sh know the value of $x, which is defined in the calling Bash shell?
    – user147505
    Dec 1 '16 at 17:52
  • 3
    @tomas, the last sh is what goes in that inline script's $0, to be used for error messages for instance. wc -l "$file" would print the file name which we don't want here and would run wc even if the file can't be opened. $x is exported to find (x=10 find...) which itself passes it to sh. Dec 1 '16 at 17:59
  • Thanks! But I guess this error that I get on OSX means that my Bash version does not support the -readable flag? find: -readable: unknown primary or operator.
    – durrrutti
    Dec 1 '16 at 18:10
  • 1
    @durrrutti, that's not down to bash. bash is just a command-line interpreter, but of the find implementation. -readable is a GNU extension, not available in OS/X find. It's only used to limit to the files that are readable (you wouldn't be able to get the line count for non-readable files). You can omit it for the first one, you'd then just get error messages when opening the files for wc for the files that are not readable. Dec 1 '16 at 19:26
  • @StéphaneChazelas, this answer is so tricky I'm left wondering: Did I miss any edge cases with my answer? :)
    – Wildcard
    Dec 1 '16 at 19:49
2

For the sake of completeness, aside AWK you can also use GNU sed to achieve the same result:

find . -type f -exec sed 11q1 '{}' ';' -exec echo rm -f '{}' ';'

Which results in a bit more concise command line.

Explanation

11 - is the address, i.e. "the eleventh line"
q - is for _q_uit (abort the execution)
1 - is the exit code parameter for q (GNU sed extension) 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.