5

I am attempting to write a filter using something like sed or awk to do the following:

  • If a given pattern does not exist in the input, copy the entire input to the output
  • If the pattern exists in the input, copy only the lines after the first occurrence to the output

This happens to be for a "git clean" filter, but that's probably not important. The important aspect is this needs to be implemented as a filter, because the input is provided on stdin.

I know how to use sed to delete lines up to a pattern, eg. 1,/pattern/d but that deletes the entire input if /pattern/ is not matched anywhere.

I can imagine writing a whole shell script that creates a temporary file, does a grep -q or something, and then decides how to process the input. I'd prefer to do this without messing around creating a temporary file, if possible. This needs to be efficient because git might call it frequently.

  • 1
    Single pass this would have to buffer lines until pattern - if any (no pattern = buffering the entire input). Is this known input and how big is it: like a few lines, millions of lines etc ? – don_crissti Jul 23 '15 at 22:08
  • @don_crissti: They're source files managed by git. Thousands of lines at most. Buffering in memory is probably a workable solution. – Greg Hewgill Jul 23 '15 at 22:10
5

If your files are not too large to fit in memory, you could use perl to slurp the file:

perl -0777pe 's/.*?PAT[^\n]*\n?//s' file

Just change PAT to whatever pattern you're after. For example, given these two input files and the pattern 5:

$ cat file
1
2
3
4
5
11
12
13
14
15
$ cat file1 
foo
bar
$ perl -0777pe 's/.*?5[^\n]*\n?//s' file
11
12
13
14
15
$ perl -0777pe 's/.*?10[^\n]*\n?//s' file1
foo
bar

Explanation

  • -pe : read the input file line by line, apply the script given by -e to each line and print.
  • -0777 : slurp the entire file into memory.
  • s/.*?PAT[^\n]*\n?//s : remove everything until the 1st occurrence of PAT and until the end of the line.

For larger files, I don't see any way to avoid reading the file twice. Something like:

awk -vpat=5 '{
              if(NR==FNR){
                if($0~pat && !a){a++; next} 
                if(a){print}
              }
              else{ 
                if(!a){print}
                else{exit} 
              }
             }' file1 file1

Explanation

  • awk -vpat=5 : run awk and set the variable pat to 5.
  • if(NR==FNR){} : if this is the 1st file.
  • if($0~pat && !a){a++; next} : if this line matches the value of pat and a is not defined, increment a by one and skip to the next line.
  • if(a){print} : if a is defined (if this file matches the pattern), print the line.
  • else{ } : if this is not the 1st file (so it's the second pass).
  • if(!a){print} if a is not defined, we want the whole file, so print every line.
  • else{exit} : if a is defined, we've already printed in the 1st pass so there's no need to reprocess the file.
| improve this answer | |
  • These are source files managed by git (not too large), so using perl is probably a good solution. – Greg Hewgill Jul 23 '15 at 22:09
  • @GregHewgill cool. Note that the first version used -00 (paragraph mode) instead of -0777 (slurp mode) so make sure you use the updated one. – terdon Jul 23 '15 at 22:10
  • I had to look up precisely what -0 actually does, but this works great. – Greg Hewgill Jul 23 '15 at 22:27
  • @GregHewgill it turns out that so did I, despite having used it for years. I keep getting it confused. The 0777 is just convention, any value >= 0400 will slurp the whole thing. Then, you also get weirdness like this. – terdon Jul 23 '15 at 22:30
5

GNU grep; cat:

{   grep -m1 'pattern' && 
    cat || ! cat ./infile
}   <./infile

POSIX sed; cat:

{ sed -ne'/PATTERN/q;H;1h;$!d;x;p'; cat; } <infile

GNU sed; cat:

{ sed -une'/PATTERN/q;H;1h;$!d;x;p'; cat; } <infile

(just add -u)


sharing is nice

All of the above commands work because the file-descriptor from which they read() is their parent process's - the shell's. It does the open() and its children inherit its descriptors. Here they all address it on stdin. One thing about file-descriptors which tends to differ from almost every other kind of inherited environment is that the child processes can affect the parent environment's file-descriptors.

These all require a regular, lseek()-able ./infile - (not counting GNU sed's -unbuffered mode, that is). And this is because each of these processes will still do some buffering, but when they have completed whatever their task is they will lseek() the descriptor right back to the last point they affected it. Otherwise it would be pretty difficult to get things to line up properly (though dd can be used to effect this).

And because the same descriptor is also passed to the next child process the shell invokes, and the last child has altered its offset, the next command will begin its input immediately where the last left off. And so when we...

seq 10 >nums
{   grep -m1 5; cat; } <nums

grep quits input on the first match for 5 after printing only the one match it found, and cat begins copying stdin to stdout just after the newline character which follows 5.

5
6
7
8
9
10

Another thing about grep is its return value can easily reveal to us if a match is found in input...

{   grep -m1 pattern && 
    cat || ! cat ./infile
}   <./infile

...where grep's returns 0 if any match is found && cat takes over from there, || else a different cat copies the whole of ./infile to output.


some grep examples


seq 100 >nums
only_after()(
    [ -f "$1" ] && {
    >/dev/null \
    grep -m1 "$2" &&
    cat  ||! cat "$1"
} <"$1")
only_after nums '[89]\{2\}'

89
90
91
92
93
94
95
96
97
98
99
100

grep's return reveals to us whether or not it has consumed all of stdin or not. If it returns true the chances are very good that cat will have some remaining to work with (the only case where it wouldn't be is if grep found its first match on the last line of input, in which case, by your rules, it shouldn't print anything anyway). But if it consumes the whole stream looking for a match and fails, it will return false, and so the second ||cat will just print the whole file.

Like this:

seq 5 >nums
only_after nums 8; echo return: "$?"

1
2
3
4
5
return: 1

some sed examples

seq 200 >nums
{ sed -une'/190/q;H;1h;$!d;x;p'; cat; } <nums

191
192
193
194
195
196
197
198
199
200

...which stacks every input line in sed's Hold space until either PATTERN is found and sed quits input entirely to leave the rest to cat, or the $last line is instead found, at which point sed prints all that it has saved. Like this:

seq 10 >nums
{ sed -une'/190/q;H;1h;$!d;x;p'; cat; } <nums

1
2
3
4
5
6
7
8
9
10

Unfortunately, it's prone to implode depending on memory availability and the sed implementation. Furthermore, a GNU sed doesn't typically play very nice with others unless you switch it to -unbuffered mode, which can have pretty deleterious effects on performance. A POSIX sed on the other hand is spec'd to play nicely in this way, and so it is definitely an avenue worth exploring.

For a non-lseek()-able input (such as a pipe) the following could work similarly:

seq 200 | sed -ne'/195/!{H;1h;$!d;x;:p' -ep -e'};n;bp'

196
197
198
199
200

...or...

seq 3 | sed -ne'/195/!{H;1h;$!d;x;:p' -ep -e'};n;bp'

1
2
3

edit-in-place


If you want to replace ./infile - in other words edit in-place - then you can actually write it over by buffering it to a temp file first:

{   g=$(grep -m1  pattern) &&
    cut -c2- <<IN >./infile
$(  printf " %s\n" "$g"    &&
    paste -d\  /dev/null -  )
IN
} <./infile

...which will not take any action at all if the pattern cannot be found - and so never reads ./infile more than the one time - but for a successful match always fully buffers the processed tail of ./infile out to a temp file before writing that over ./infile. More specifically - the only part of the infile which gets written out to the shell's here-document is the part after grep's match. All of the input that grep consumes in the matching stays consumed and so it is only the buffered tail-end which winds up in the temporary buffer at all.

What's more, most shells will back their here-documents in /tmp - which, because /tmp is very often a tmpfs on a Linux system, means that on said systems the buffered part never winds up on disk at all. In fairness, though, because of the way the kernel will handle file caching and etc, there probably isn't much difference between writing it to a tmpfs and writing it anywhere else provided you have the memory to cache it anyway. /tmp is maybe just a little more explicit. The shell also unlink()'s the buffer file before ever writing a byte to it - and so it only exists at all for as long as its read and/or write descriptors remain open. There is nothing to clean up.


all wrapped up


I wrote a little program to do this...

allor()(
        set -f; unset o z i m;  OPTIND=1 IFS="
"
        op()    while   getopts :i:o:m: O               &&
                        case    $O$OPTARG               in
                        ([$z]*|m*[!0-9]*|[!imo]*) ! :   ;;
                        (o+)    o= O=;; (o-)     O=     ;;
                        esac||! o=${o+${o:-$i}}  m=${m:-1}
                do      eval "z=$z${O:-o #} $O=\$OPTARG"||exit
                done

        op "$@";[ -f "${i:?No input specified!}" ]      ||i=
        exec < "${i:?Input is not a regular file!}"     &&
        shift   $((OPTIND-(${#O}+1)))                   &&
        z=$( !  { {     grep -m$m "$@" 2>&3 |
                   >&4  sed  -ne'$=;$s/^/ /p'
                } 3>&1| grep . >&2;}   4>&1 )           &&
        set     ${z:?No match found!}   ${o:+'>"$o"'}   &&
        case    $((m==$1))$o    in      (0"$i") ! :     ;;
        (0*)    <"$i"   eval "  cat $3     &&   ! :"    ;;
        (1*)    <<-i    eval "  cut -c2-   $3"
                $(      printf %s\\n $2;paste /dev/null -)
                i
        esac
)

...which adds some option parsing and whatnot. Basically you can pass any argument to grep that you might like to do - all arguments are passed through verbatim except the first of either -i or -o or -m.

You use -i and its argument to specify the input file. You can use -o- to write to stdout - which is the default behavior anyway - or -o+ to edit the -i file in-place, or -o and any writable pathname. You use -m to specify a match count - which is to say that you can get all of a file after -m count matches, or just all of the file if that many matches cannot be found in input. All arguments from the first which isn't a valid -[iom] switch, or from the second occurring -[io] are handed directly to grep.

It tests whether the requested matches were successful and where output should go before attempting to write it. For example, if the matches were not successful and output is directed back over ./infile it will not do anything at all and leave ./infile alone. If the matches were successful and the outfile and infile are the same, it will shorten infile. But if the matches are not successful and output is directed anywhere else it will just cat input to output.

A little demo:

seq 20 >nums
allor -inums -m2 5

15
16
17
18
19
20

...and...

seq 10 >nums
allor -inums -m2 5; echo return: "$?"

1
2
3
4
5
6
7
8
9
10
return: 1

...and...

seq 20000 >nums
allor -m1999 -inums -o+ 5$; cat nums

19985
19986
19987
19988
19989
19990
19991
19992
19993
19994
19995
19996
19997
19998
19999
20000
| improve this answer | |
2

Using GNU sed, you can do this:

:x;/PATTERN/{s/.*//;:z;N;bz};N;bx

For example is we use 7 as the pattern we want to match and input data generated by seq, this will print the numbers 8 to 20 (including 17):

seq 20 | sed ':x;/7/{s/.*//;:z;N;bz};N;bx'

And this will print 1 to 6:

seq 6 | sed ':x;/7/{s/.*//;:z;N;bz};N;bx'

As noted in the comments, this effectively reads the whole file into memory - make sure that is an OK thing to do in your case.

Note also this currently has the caveat that an extra leading newline is output for the 8 to 20 case - I'm trying to figure out how to remove that robustly - not sure if that matters for your application or not.

| improve this answer | |
  • maybe like this: ':x;/PATTERN/{$d;N;s/.*\n//;:z;N;bz};N;bx' to avoid the leading newline – don_crissti Jul 24 '15 at 0:28
  • Can be rewritten as this sed ':;N;$!b;${s/[^PATTERN]*PATTERN[^\n]*\n//}' – 123 Jul 24 '15 at 13:46
  • @User112638726 - no it can't unless you've got a one char pattern or are 100% certain that none of the chars in your pattern can be matched anywhere else in any other order than the one you look for. But it can be written like: sed -ne'/PATTERN/!d;:n' -e'n;p;bn' - Digital Trauma - the previous little loop won't have any extra/missing char issues, and it doesn't stack like Next either - it repeatedly overwrites the current line w/ the next. – mikeserv Jul 24 '15 at 14:43
  • @mikeserv Yeah, i only checked with single chars. Yours doesn't work either though, if the patterns not found it deletes everything. – 123 Jul 24 '15 at 14:45
  • @User112638726 - damn. Forgot about that. That is an excellent point. So you gotta kinda burn at bOth ends. sed -ne'/PATTERN/!{H;1h;$!d;x;:p' -ep -e'};n;bn' – mikeserv Jul 24 '15 at 14:57
1

Pipe the input (cat file in this case) into a basic single-pass of awk first. Then, if pattern is not found (ie. nothing was printed), the process continues on with cat file

cat file | { awk -v pat='^a.c$' '
             { if( m ) print; else{ if( $0 ~ pat ) m=1 } 
             } END{ exit !m }' || cat file
           } 

input:

1
2
abc
4
aXc
6

output

4
aXc
6
| improve this answer | |
  • Nice! The OP doesn't want to print the line with the 1st occurrence of pat though, see the 2nd bullet point in the question. – terdon Jul 24 '15 at 0:14
  • The first occurence of the pattern is not on line 1, it is on line 5 - the pattern is limited to just 3 characters on a 3 character long line: ^p.t$ – Peter.O Jul 24 '15 at 0:42
  • I've now abandoned sed in favour of awk (plus cat when no match) ... – Peter.O Jul 24 '15 at 10:14
0

TXR solution, on the command line:

$ txr -c '@(maybe)
@(skip)
@(trailer)
@/pattern/
@(end)
@(repeat)
@line
@(do (put-line line))
@(end)' -

Example: lines in which pattern doesn't occur. Code abbreviated:

$ txr -c '@(maybe)
[...]' -
a
b
c
d
Ctrl-DEnter
a
b
c
d

When pattern occurs:

$ txr -c '@(maybe)
[...]
@(end)' -
a
b
c
pattern
pattern
x
x
y
y
z
z
Ctrl-DEnter

As you can see, once pattern occurs, the lines start being echoed from then on.

The logic is very simple: material enclosed in @(maybe)...@(end) is optionally matched. There we have a @(skip) which skips any number of lines, followed by a @(trailer) which means "match the following as a trailing context (do not consume it)". (This feature and its name is inspired by the slash trailing contexts in Lex.) If we take out @(trailer) then the line which matches the pattern is excluded from the output.

Of course @/pattern/ is a regex. It is implicitly anchored: it must match the whole line. So to match a line which contains abc we would use @/.*abc.*/ or @(skip)abc@(skip).

In the case when the pattern doesn't occur, the skip inside maybe scans the entire input, and eventually fails. maybe catches this failure and sweeps it under the rug, producing a success. The material which follows the maybe is then matched, against the original input on which the interior of maybe failed (i.e. start of the stream).

At the end we we have an repetitive matching structure, @(repeat), which contains an output side effect.

A different TXR solution which uses the @(data ...) directive to capture the current data cursor (a lazy list pointer) in the variable start, then captures it again at EOF, and uses the ancient Lisp function ldiff to calculate the output:

$ txr -c '@(maybe)
@(skip)
@/pattern/
@(end)
@(data start)
@(skip)
@(eof)
@(data end)
@(do (tprint (ldiff start end)))' -

The start is captured after optionally matching some lines followed by the pattern. If the pattern doesn't occur, it's as if this @(maybe)...@(end) block isn't there, and the start of the data is captured.

In short, "maybe skip some lines ending in pattern; in either case, note the position as start; then skip to EOF and note the position as the end; print all between start and end".

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.