10

I have a script mycommand.sh that I can't run twice. I want to split output to two different files one file containing the lines that match a regex and one file containing the lines that don't match a regex. What I wish to have is basically something like this:

./mycommand.sh | grep -E 'some|very*|cool[regex].here;)' --match file1.txt --not-match file2.txt

I know I can just redirect the output to a file and then to two different greps with and without -v option and redirect their output to two different files. But I was jsut wondering if it was possible to do it with one grep.

So, Is it possible to achieve what I want in a single line?

16

There are many ways to accomplish this.

Using awk

The following sends any lines matching coolregex to file1. All other lines go to file2:

./mycommand.sh | awk '/[coolregex]/{print>"file1";next} 1' >file2

How it works:

  1. /[coolregex]/{print>"file1";next}

    Any lines matching the regular expression coolregex are printed to file1. Then, we skip all remaining commands and jump to start over on the next line.

  2. 1

    All other lines are sent to stdout. 1 is awk's cryptic shorthand for print-the-line.

Splitting into multiple streams is also possible:

./mycommand.sh | awk '/regex1/{print>"file1"} /regex2/{print>"file2"} /regex3/{print>"file3"}'

Using process substitution

This is not as elegant as the awk solution but, for completeness, we can also use multiple greps combined with process substitution:

./mycommand.sh | tee >(grep 'coolregex' >File1) | grep -v 'coolregex' >File2

We can also split up into multiple streams:

./mycommand.sh | tee >(grep 'coolregex' >File1) >(grep 'otherregex' >File3) >(grep 'anotherregex' >File4) | grep -v 'coolregex' >File2
  • Oh cool! Is it also possible to split it to several files without just doing another awk instead of file2? I mean in a way that regexes can overlap for example. – yukashima huksay Dec 31 '17 at 21:35
  • 1
    @aran Yes, awk is very flexible. Precisely how one does it would depend on how the regexes overlap. – John1024 Dec 31 '17 at 21:37
  • I would love to see a solution even if it doesn't supports overlapping regexes. by overlapping I mean like having the subset's intersection not nervelessly empty. – yukashima huksay Dec 31 '17 at 21:39
  • 1
    @aran I've added to the answer examples with multiple streams for both methods. – John1024 Dec 31 '17 at 21:42
7
sed -n -e '/pattern_1/w file_1' -e '/pattern_2/w file_2' input.txt

w filename - write the current pattern space to filename.

If you want all matching lines to go to file_1 and all non-matching lines to file_2, you can do:

sed -n -e '/pattern/w file_1' -e '/pattern/!w file_2' input.txt

or

sed -n '/pattern/!{p;d}; w file_1' input.txt > file_2

Explanation

  1. /pattern/!{p;d};
    • /pattern/! - negation - if a line doesn't contain pattern.
    • p - print the current pattern space.
    • d - delete pattern space. Start next cycle.
    • so, if a line doesn't contain pattern, it prints this line to the standard output and picks the next line. Standard output is redirected to the file_2 in our case. The next part of the sed script (w file_1) doesn't reached while the line doesn't match to the pattern.
  2. w file_1 - if a line contains pattern, the /pattern/!{p;d}; part is skipped (because it is executed only when pattern doesn't match) and, thus, this line goes to the file_1.
  • Can you please add some more explanation to the last solution? – yukashima huksay Dec 31 '17 at 23:24
  • @aran Explanation added. Also the command is corrected - file_1 and file_2 were swapped to the right order. – MiniMax Dec 31 '17 at 23:49
0

I liked the sed solution since it does not rely on bashisms and treats the output files on the same footing. AFAIK, there is no standalone Unix tool that does what you want so you'd need to program it yourself. If we would abandon the Swiss army knife approach, we could use any of the scripting languages (Perl, Python, NodeJS).

This is how it would be done in NodeJS

  #!/usr/bin/env node

  const fs = require('fs');
  const {stderr, stdout, argv} = process;

  const pattern = new RegExp(argv[2] || '');
  const yes = argv[3] ? fs.createWriteStream(argv[3]) : stdout;
  const no = argv[4] ? fs.createWriteStream(argv[4]) : stderr;

  const out = [no, yes];

  const partition = predicate => e => {
    const didMatch = Number(!!predicate(e));
    out[didMatch].write(e + '\n');
  };

  fs.readFileSync(process.stdin.fd)
    .toString()
    .split('\n')
    .forEach(partition(line => line.match(pattern)));

Example usage

# Using designated files
./mycommand.sh | partition.js pattern file1.txt file2.txt

# Using standard output streams
./partition.js pattern > file1.txt 2> file2.txt
-1

If you don't mind the use of Python and a different regular expression syntax:

#!/usr/bin/env python3
import sys, re

regex, os1, os2 = sys.argv[1:]
regex = re.compile(regex)
with open(os1, 'w') as os1, open(os2, 'w') as os2:
    os = (os1, os2)
    for line in sys.stdin:
        end = len(line) - line.endswith('\n')
        os[regex.search(line, 0, end) is not None].write(line)

Usage

./match-split.py PATTERN FILE-MATCH FILE-NOMATCH

Example

printf '%s\n' foo bar baz | python3 match-split.py '^b' b.txt not-b.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.