I currently use the following simplified command to remove trailing whitespace and add a newline at end of file where needed:

find . -type f -exec sed -i -e 's/[ \t]\+\(\r\?\)$/\1/;$a\' {} \+

As you'll quickly see, this has two problems: It will change binary files and it will add a newline to the end of files with ␍␊ line separators. These modifications are easy to undo or skip when committing in git gui or the like, but I'd like to minimize* the amount of reverting. To that end:

Is there a way to skip the whole file if any line matches a regex in sed?

* I'm aware that there might be binary files without ␀ characters, and there could be files with deliberately mixed newlines or ␀s. But I'm looking for the solution which requires the minimal human intervention. I could conceivably list all the file extensions that I'd like to operate on, but it would be a very long list which would have to be constantly reviewed, and because of name clashes it would still be possible that binary files slip through.

Complicated workaround:

while IFS= read -r -d '' -u 9
    if [[ "$(file -bs --mime-type -- "$REPLY")" = text/* ]]
        sed -i -e 's/[ \t]\+\(\r\?\)$/\1/;$a\' -- "$REPLY"
        echo "Skipping $REPLY" >&2
done 9< <(find . -type f -print0)
  • Would you be ok with a git-only solution that trusts git's view of what is a binary file and what is not?
    – Mat
    Apr 11, 2012 at 13:36
  • If you can use Perl, it has a -T test that determines if the file is "an ASCII text file". Apr 11, 2012 at 13:41

3 Answers 3


If you trust git's point of view on what is a binary file or not, you can use git grep to get a list of non-binary files. Assuming t.cpp is a text file, and ls is a binary, both checked in:

$ ls
t.cpp ls
$ git grep -I --name-only -e ''

The -I option means:

Don't match the pattern in binary files.

To combine that with your sed expression:

$ git grep -I --name-only -z -e '' | \
       xargs -0 sed -i.bk -e 's/[ \t]\+\(\r\?\)$/\1/;$a\'

(-z / xargs -0 to help with strange filenames.)

Check out the git grep man page for other useful options - --no-index or --cached could help depending on exactly what set of files you want to operate on.

  • Neat! Note that it skips files of type "Little-endian UTF-16 Unicode text, with very long lines, with CRLF line terminators".
    – l0b0
    Apr 11, 2012 at 14:18
  • Well, yeah, that's why there's the "trust git's point of view" disclaimer :) Note that I'm not sure your sed expression would work on UTF-16 (doesn't on my machine here, but I guess there could be ways/settings to make it work).
    – Mat
    Apr 11, 2012 at 14:22
  • Sure, that wasn't meant as a complaint, just FYI. After all, UTF-16 is really rare these days.
    – l0b0
    Apr 11, 2012 at 14:25

Is there a way to skip the whole file if any line matches a regex in sed?

Yes, there is.

# test case for skipping file if a sed regex match succeeds

echo 'Hello, world!' > hello_world.txt
cat hello_world.txt
ls -li hello_world.txt

sed -i -e '/.*Hello.*/{q;}; s/world/WORLD/g' hello_world.txt # skips file
sed -i -e '/.*HeLLo.*/{q;}; s/world/WORLD/g' hello_world.txt

Here's a Perl script that iterates over its arguments (which must be file names) and appends a newline to every file that doesn't end in a newline. Files containing a null byte are skipped. Files that already end in a newline are unmodified. Files that contain a CR get CRLF appended, others get just LF. Untested.

#!/usr/bin/env perl
foreach my $f (@ARGV) {
    open F, "<", $f or die;
    my $last = undef;
    my $cr = 0;
    while (<>) {if (/\0/) {undef $last; break} $last = $_; ++$cr if /\r$/}
    close F;
    if (defined $last && $last !~ /\n\Z/) {
        open F, ">>", $f or die;
        print($cr ? "\r\n" : "\n");
        close F or die;

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .