I currently use the following simplified command to remove trailing whitespace and add a newline at end of file where needed:

find . -type f -exec sed -i -e 's/[ \t]\+\(\r\?\)$/\1/;$a\' {} \+

As you'll quickly see, this has two problems: It will change binary files and it will add a newline to the end of files with ␍␊ line separators. These modifications are easy to undo or skip when committing in git gui or the like, but I'd like to minimize* the amount of reverting. To that end:

Is there a way to skip the whole file if any line matches a regex in sed?

* I'm aware that there might be binary files without ␀ characters, and there could be files with deliberately mixed newlines or ␀s. But I'm looking for the solution which requires the minimal human intervention. I could conceivably list all the file extensions that I'd like to operate on, but it would be a very long list which would have to be constantly reviewed, and because of name clashes it would still be possible that binary files slip through.

Complicated workaround:

while IFS= read -r -d '' -u 9
    if [[ "$(file -bs --mime-type -- "$REPLY")" = text/* ]]
        sed -i -e 's/[ \t]\+\(\r\?\)$/\1/;$a\' -- "$REPLY"
        echo "Skipping $REPLY" >&2
done 9< <(find . -type f -print0)
  • Would you be ok with a git-only solution that trusts git's view of what is a binary file and what is not?
    – Mat
    Apr 11, 2012 at 13:36
  • If you can use Perl, it has a -T test that determines if the file is "an ASCII text file". Apr 11, 2012 at 13:41

3 Answers 3


If you trust git's point of view on what is a binary file or not, you can use git grep to get a list of non-binary files. Assuming t.cpp is a text file, and ls is a binary, both checked in:

$ ls
t.cpp ls
$ git grep -I --name-only -e ''

The -I option means:

Don't match the pattern in binary files.

To combine that with your sed expression:

$ git grep -I --name-only -z -e '' | \
       xargs -0 sed -i.bk -e 's/[ \t]\+\(\r\?\)$/\1/;$a\'

(-z / xargs -0 to help with strange filenames.)

Check out the git grep man page for other useful options - --no-index or --cached could help depending on exactly what set of files you want to operate on.

  • Neat! Note that it skips files of type "Little-endian UTF-16 Unicode text, with very long lines, with CRLF line terminators".
    – l0b0
    Apr 11, 2012 at 14:18
  • Well, yeah, that's why there's the "trust git's point of view" disclaimer :) Note that I'm not sure your sed expression would work on UTF-16 (doesn't on my machine here, but I guess there could be ways/settings to make it work).
    – Mat
    Apr 11, 2012 at 14:22
  • Sure, that wasn't meant as a complaint, just FYI. After all, UTF-16 is really rare these days.
    – l0b0
    Apr 11, 2012 at 14:25

Is there a way to skip the whole file if any line matches a regex in sed?

Yes, there is.

# test case for skipping file if a sed regex match succeeds

echo 'Hello, world!' > hello_world.txt
cat hello_world.txt
ls -li hello_world.txt

sed -i -e '/.*Hello.*/{q;}; s/world/WORLD/g' hello_world.txt # skips file
sed -i -e '/.*HeLLo.*/{q;}; s/world/WORLD/g' hello_world.txt

Here's a Perl script that iterates over its arguments (which must be file names) and appends a newline to every file that doesn't end in a newline. Files containing a null byte are skipped. Files that already end in a newline are unmodified. Files that contain a CR get CRLF appended, others get just LF. Untested.

#!/usr/bin/env perl
foreach my $f (@ARGV) {
    open F, "<", $f or die;
    my $last = undef;
    my $cr = 0;
    while (<>) {if (/\0/) {undef $last; break} $last = $_; ++$cr if /\r$/}
    close F;
    if (defined $last && $last !~ /\n\Z/) {
        open F, ">>", $f or die;
        print($cr ? "\r\n" : "\n");
        close F or die;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.