Fast way to delete files with less than x lines

Question

What's a fast and not too complicated way to delete all files in a directory that are under x lines long, in bash?

Answer 1

Here is a POSIX solution that should be pretty straightforward to understand:

find . -type f -exec awk -v x=10 'NR==x{exit 1}' {} \; -exec echo rm -f {} \;

As in Stephane's answer, remove the echo when happy with what will be removed.

---

Explanations, written for those totally new to Unix/Linux:

The dot . represents the current directory. find finds files and directories recursively within ., and can do things with them.

-type is one of find's primaries; it is a test that will be performed for each file and directory that is recursively found (within .), and the rest of the primaries on the line are only evaluated if this results in "true."

In this particular case, we only continue if we are dealing with a regular file, not a directory or something else (e.g. a block device.)

---

The -exec primary (of find) calls an external command, and only proceeds to the next primary if the external command exits successfully (exit status of "0"). The {} is replaced with the file name being "considered" by the find command. So the first -exec call is equivalent to the following shell command, executed for each file in turn:

awk -v x=10 'NR==x{exit 1}' ./somefilename

---

Awk is an entire language in itself, designed for handling delimited text files such as CSVs. The Awk conditionals and commands (which are contained between single quotes and start with the letters NR) are executed for every line of a text file. (Implicit looping.)

To learn Awk fully, I highly recommend the Grymoire Tutorial, but I will explain the Awk features used in the above command.

---

The -v flag to Awk allows us to set an Awk variable (once) before the Awk commands are executed (for each line of the file.) In this case we set x to 10.

---

NR is a special Awk variable referring to the "Number of the current Record." In other words, it is the line number we are looking at in any particular pass through the loop.

(Note that it is possible, though unusual, to use a different "Record Separator" than the default of a newline character, by setting RS. Here is an example of playing with record separators.)

---

Awk scripts in general consist of conditions (outside curly braces) combined with actions (inside curly braces.) There can be compound conditions and compound actions, and there is a default condition (true) and a default action (print), but we needn't bother with those.

The condition here is, "Is this the 10th line?" If this is the case, we exit with a non-zero exit status, which in shell scripting means "unsuccessful command termination."

Thus the only way this Awk command will exit successfully is if the end of the file is reached before the 10th line is reached.

So if the Awk script exits successfully, it means you have a file of less than ten lines.

---

The next -exec call (if you remove the echo) will remove each file (that gets that far in evaluation of find's primaries) by running:

rm -f ./somefilename

Answer 2

Assuming a find implementation that supports the -readable predicate (if your find doesn't support it, just remove it, you'll just get error messages for non-readable files, or replace with -exec test -r {} \;):

x=10 find . -type f -readable -exec sh -c '
  for file do
    lines=$(wc -l < "$file") && [ "$((lines))" -lt "$x" ] && echo rm -f "$file"
  done' sh {} +

Remove the echo if happy.

That's not particularly efficient in that it counts all the lines in every file while it only needs to stop at the x^th one and it runs one wc (and potentially one rm) command for each file.

With GNU awk, you can make it a lot more efficient with:

x=10
find . -type f -readable -exec awk -v x="$x" -v ORS='\0' '
  FNR == x {nextfile}
  ENDFILE {if (FNR < x) print FILENAME}' {} +|
  xargs -r0 echo rm -f

(again, remove echo when happy).

The same with perl:

x=10 find . -type f -readable -exec perl -Tlne '
  if ($. == $ENV{x}) {close ARGV}
  elsif (eof) {print $ARGV; close ARGV}' {} +

Replace print with unlink if happy.

Answer 3

For the sake of completeness, aside AWK you can also use GNU sed to achieve the same result:

find . -type f -exec sed 11q1 '{}' ';' -exec echo rm -f '{}' ';'

Which results in a bit more concise command line.

Explanation

11 - is the address, i.e. "the eleventh line"
q - is for _q_uit (abort the execution)
1 - is the exit code parameter for q (GNU sed extension)