9

I have files in subdirectories of the current directory that may or may not have new lines at the end; how can I find files that don't have a newline at the end?

I've tried this:

find . -name '*.styl' | while read file; do
    awk 'END{print}' $file | grep -E '^$' > /dev/null || echo $file;
done

but it doesn't work. awk 'END{print}' $file prints the line before an empty new line, the same as tail -n 1 $file.

  • @don_crissti I need files that don't have trailing empty line. – jcubic Oct 12 '16 at 13:48
  • 2
    May I ask the reason you need to find those files? I guess it has to do with the fact that Text files in unix are supposed to be terminated with a newline (vi will "almost silently" add one when you save, for example), and several (text-oriented) commands will ignore the last line if it is not terminated by a newline (wc, iirc .... but there are others). And this may help – Olivier Dulac Oct 12 '16 at 15:24
  • awk 'END{print}' $file : this ignores totally the content of $file, and after finishing parsing all the files contained in "$file" it adds a newline. As it is the only thing that awk command prints, it could be replaced with : printf '\n' (without any mentino of $file at all) and do the same thing. I think this is NOT what you were aiming at (ie: print the last line of the file?) – Olivier Dulac Oct 12 '16 at 15:43
  • @don_crissti: if the last character of a file is not a newline, then that file is not stricly posixly a unix TEXT file. see: unix.stackexchange.com/a/263919/27616 . note that many text commands (wc, for example) simply ignore that last "line" if it is not terminated by a newline – Olivier Dulac Oct 12 '16 at 15:44
  • 1
    @OlivierDulac: gawk prints c and so does FreeBSD, but I had not noticed it is documented as implementation-dependent: gnu.org/software/gawk/manual/… . So it does happen but not always. – dave_thompson_085 Oct 15 '16 at 17:36
13

To clarify, the LF (aka \n or newline) character is the line delimiter, it's not the line separator. A line is not finished unless it's terminated by a newline character. A file that only contains a\nb is not a valid text file because it contains characters after the last line. Same for a file that contains only a. A file that contains a\n contains one non-empty line.

So a file that ends with at least one empty line ends with two newline characters or contains a single newline character.

If:

 tail -c 2 file | od -An -vtc

Outputs \n or \n \n, then the file contains at least one trailing empty line. If it outputs nothing, then that's an empty file, if it outputs <anything-but-\0> \n, then it ends in a non-empty line. Anything else, it's not a text file.

Now, to use that to find files that end in an empty line, OK that's efficient (especially for large files) in that it only reads the last two bytes of the files, but first the output is not easily parsable programmatically especially considering that it's not consistent from one implementation of od to the next, and we'd need to run one tail and one od per file.

find . -type f -size +0 -exec gawk '
  ENDFILE{if ($0 == "") print FILENAME}' {} +

(to find files ending in an empty line) would run as few commands as possible but would mean reading the full content of all files.

Ideally, you'd need a shell that can read the end of a file by itself.

With zsh:

zmodload zsh/system
for f (**/*(D.L+0)) {
  {
    sysseek -w end -2
    sysread
    [[ $REPLY = $'\n' || $REPLY = $'\n\n' ]] && print -r -- $f
  } < $f
}
  • a way to use this answer's method to know if some file(s) are text files: are_textfiles () { nontext=0; rem="return 0 if all args are files with terminating newline, or n [=number of non-textfiles]" ; for f in "$@" ; do [ -f "$f" ] && { tail -c 1 "$f" | od -An -vtc | grep "\\n" ;} >/dev/null 2>&1 || ((nontext++)) ; done ; return $nontext ; }. Use as: if ( are_textfiles this that otherthing ) ; then echo all are text files ; else echo "are_textfiles returned : $?" ; fi – Olivier Dulac Oct 14 '16 at 16:19
6

With gnu sed and a shell like zsh (or bash with shopt -s globstar):

sed -ns '${/./F}' ./**/*.styl

this checks if the last line of each file is not empty, if so it prints the filename.
If you want the opposite (print file names if the last line is empty) just replace /./ with /^$/

  • 1
    Never seen -s in action before. Thank you GNU! – glenn jackman Oct 12 '16 at 14:36
  • Note: The F option exists from sed version 4.2.2 (December 22, 2012) – sorontar Oct 12 '16 at 15:19
3

A correctly terminated text file with an empty last line ends in two \n.

Then, we expect that tail -c2 must be equal to $'\n\n'.

Sadly command expansions remove trailing new lines. We will need a bit of tweaking.

f=filename
nl='
'
t=$(tail -c2 $f; printf x)  # capture the last two characters.
r="${nl}${nl}$"                 # regex for: "ends in two newlines".
[[ ${t%x} =~ $r ]] &&  echo "file $f ends in an empty line"

We could even expand a bit to check which files fail to have a trailing new line:

nl='
'
nl=$'\n'
find . -type f -name '*.styl' | while read f; do
    t=$(tail -c2 $f; printf x); r1="${nl}$"; r2="${nl}${r1}"
    [[ ${t%x} =~ $r1 ]] || echo "file $f is missing a trailing newline"
    [[ ${t%x} =~ $r2 ]] && echo "$f"
done

Note that the newline could be changed to something like $'\r\n if needed.
In that case, also change tail -c2 to tail -c4.

0
for file in *; do
    # Check if the file is readable to avoid clutter
    if cat "./$file" 2&>1 /dev/null; then
        # Compare the last character with a single newline character.
        if [ -n "$(tail -c 1 -- "./$file")" ]; then
            echo "$file"
        fi
        # Also report empty files.
        if [ $(wc -c  < "./$file") -eq 0 ]; then
            echo "$file"
        fi
    fi
done
  • 1
    this don't work with empty files but I can live with that. – jcubic Oct 12 '16 at 14:16
  • There might be some more errors because string comparison doesn't seem to work the way I expected. I added a check for empty files. – Oskar Skog Oct 12 '16 at 14:32
  • Ah, it ignores the newline characters. – Oskar Skog Oct 12 '16 at 14:44
  • Consider the more readable cat $file 2>&1 /dev/null, or if this is Bash-only, cat $file &> /dev/null. – cat Oct 12 '16 at 17:52
  • 1
    Also, consider quoting $file everywhere it's used -- and please, use $(commands ...) instead of `backticks`... – cat Oct 12 '16 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.