Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I currently use the following simplified command to remove trailing whitespace and add a newline at end of file where needed:

find . -type f -exec sed -i -e 's/[ \t]\+\(\r\?\)$/\1/;$a\' {} \+

As you'll quickly see, this has two problems: It will change binary files and it will add a newline to the end of files with ␍␊ line separators. These modifications are easy to undo or skip when committing in git gui or the like, but I'd like to minimize* the amount of reverting. To that end:

Is there a way to skip the whole file if any line matches a regex in sed?

* I'm aware that there might be binary files without ␀ characters, and there could be files with deliberately mixed newlines or ␀s. But I'm looking for the solution which requires the minimal human intervention. I could conceivably list all the file extensions that I'd like to operate on, but it would be a very long list which would have to be constantly reviewed, and because of name clashes it would still be possible that binary files slip through.

Complicated workaround:

while IFS= read -r -d '' -u 9
    if [[ "$(file -bs --mime-type -- "$REPLY")" = text/* ]]
        sed -i -e 's/[ \t]\+\(\r\?\)$/\1/;$a\' -- "$REPLY"
        echo "Skipping $REPLY" >&2
done 9< <(find . -type f -print0)
share|improve this question
Would you be ok with a git-only solution that trusts git's view of what is a binary file and what is not? – Mat Apr 11 '12 at 13:36
If you can use Perl, it has a -T test that determines if the file is "an ASCII text file". – glenn jackman Apr 11 '12 at 13:41
up vote 1 down vote accepted

If you trust git's point of view on what is a binary file or not, you can use git grep to get a list of non-binary files. Assuming t.cpp is a text file, and ls is a binary, both checked in:

$ ls
t.cpp ls
$ git grep -I --name-only -e ''

The -I option means:

Don't match the pattern in binary files.

To combine that with your sed expression:

$ git grep -I --name-only -z -e '' | \
       xargs -0 sed -i.bk -e 's/[ \t]\+\(\r\?\)$/\1/;$a\'

(-z / xargs -0 to help with strange filenames.)

Check out the git grep man page for other useful options - --no-index or --cached could help depending on exactly what set of files you want to operate on.

share|improve this answer
Neat! Note that it skips files of type "Little-endian UTF-16 Unicode text, with very long lines, with CRLF line terminators". – l0b0 Apr 11 '12 at 14:18
Well, yeah, that's why there's the "trust git's point of view" disclaimer :) Note that I'm not sure your sed expression would work on UTF-16 (doesn't on my machine here, but I guess there could be ways/settings to make it work). – Mat Apr 11 '12 at 14:22
Sure, that wasn't meant as a complaint, just FYI. After all, UTF-16 is really rare these days. – l0b0 Apr 11 '12 at 14:25

Here's a Perl script that iterates over its arguments (which must be file names) and appends a newline to every file that doesn't end in a newline. Files containing a null byte are skipped. Files that already end in a newline are unmodified. Files that contain a CR get CRLF appended, others get just LF. Untested.

#!/usr/bin/env perl
foreach my $f (@ARGV) {
    open F, "<", $f or die;
    my $last = undef;
    my $cr = 0;
    while (<>) {if (/\0/) {undef $last; break} $last = $_; ++$cr if /\r$/}
    close F;
    if (defined $last && $last !~ /\n\Z/) {
        open F, ">>", $f or die;
        print($cr ? "\r\n" : "\n");
        close F or die;
share|improve this answer

Is there a way to skip the whole file if any line matches a regex in sed?

Yes, there is.

# test case for skipping file if a sed regex match succeeds

echo 'Hello, world!' > hello_world.txt
cat hello_world.txt
ls -li hello_world.txt

sed -i -e '/.*Hello.*/{q;}; s/world/WORLD/g' hello_world.txt # skips file
sed -i -e '/.*HeLLo.*/{q;}; s/world/WORLD/g' hello_world.txt
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.