Filter or pipe certain sections of a file

Question

I have an input file with some sections the are demarcated with start and end tags, for example:

line A
line B
@@inline-code-start
line X
line Y
line Z
@@inline-code-end
line C
line D

I want to apply a transformation to this file such that lines X,Y,Z are filtered through some command (nl, for example), but the rest of the lines pass through unchanged. Notice that nl (number lines) accumulates state across lines, so it is not a static transformation that is being applied to each of lines X,Y,Z. (Edit: it was pointed out that nl can work in a mode that doesn't require accumulated state, but I am just using nl as an example to simplify the question. In reality the command is a more complex custom script. What I am really looking for is a generic solution to the problem of applying a standard filter to a subsection of an input file)

The output should look like:

line A
line B
     1 line X
     2 line Y
     3 line Z
line C
line D

There can be several such sections in the file that require the transformation.

Update 2 I didn't originally specify what should happen if there is more one section, for example:

line A
line B
@@inline-code-start
line X
line Y
line Z
@@inline-code-end
line C
line D
 @@inline-code-start
line L
line M
line N
@@inline-code-end

My expectation would be that state would only need to be maintained within a given section, giving:

line A
line B
     1 line X
     2 line Y
     3 line Z
line C
line D
     1 line L
     2 line M
     3 line N

but, I think interpreting the problem as requiring the state to be kept across sections is valid, and useful in many contexts.

End Update 2

My first thought is to build a simple state machine that tracks what section we are in:

#!/usr/bin/bash
while read line
do
  if [[ $line == @@inline-code-start* ]]
  then
    active=true
  elif [[ $line == @@inline-code-end* ]]
  then
    active=false
  elif [[ $active = true ]]
  then
    # pipe
  echo $line | nl
  else
    # output
    echo $line
  fi
done

Which I run with:

cat test-inline-codify | ./inline-codify

This doesn't work since each call to nl is independent, so the line numbers do not increase:

line A
line B
     1  line X
     1  line Y
     1  line Z
line C
line D

My next attempt was to use a fifo:

#!/usr/bin/bash
mkfifo myfifo
nl < myfifo &
while read line
do
  if [[ $line == @@inline-code-start* ]]
  then
    active=true
  elif [[ $line == @@inline-code-end* ]]
  then
    active=false
  elif [[ $active = true ]]
  then
    # pipe
    echo $line > myfifo
  else
    # output
    echo $line
  fi
done
rm myfifo

This gives the correct output, but in the wrong order:

line A
line B
line C
line D
     1  line 1
     2  line 2
     3  line 3

There's probably some caching going on.

Am I going about this all wrong? This seems like a pretty generic problem. I feel like there ought to be a simple pipeline that would solve this.

Answer 1

I would agree with you - it probably is a generic problem. Some common utilities have some facilities for handling it, though.

---

nl

nl, for example, separates input into logical pages as -delimited by a two character section delimiter. Three occurrences on a line all alone indicate the start of a heading, two the body and one the footer. It replaces any of these found in input with a blank line in output - which are the only blank lines it ever prints

I altered your example to include another section and put it in ./infile. So it looks like this:

line A
line B
@@inline-code-start
line X
line Y
line Z
@@inline-code-end
line C
line D
@@start
line M
line N
line O
@@end

Then I ran the following:

sed 's/^@@.*start$/@@@@@@/
     s/^@@.*end$/@@/'  <infile |
nl -d@@ -ha -bn -w1

nl can be told to accumulate state across logical pages, but it does not by default. Instead it will number the lines of its input according to styles, and by section. So -ha means number all header lines and -bn means no body lines - as it starts out in a body state.

Until I learned this I used to use nl for any input, but after realizing that nl might distort output according to its default -delimiter \: I learned to be more careful with it and started using grep -nF '' for untested input instead. But another lesson learned that day was that nl can be very usefully applied in other respects - such as this one - if you just modify its input only a little - as I do with sed above.

OUTPUT

  line A
  line B

1       line X
2       line Y
3       line Z

  line C
  line D

1       line M
2       line N
3       line O

Here's some more about nl - do you notice above how all lines but the numbered ones start with spaces? When nl numbers lines it inserts a certain number of characters into the head of each. For those lines it doesn't number - even blanks - it always matches the indent by inserting ( -width count + -separator len ) * spaces at the head of unnumbered lines. This allows you to reproduce the not-numbered content exactly by comparing it to the numbered content - and with little effort. When you consider that nl will divide its input into logical sections for you, and that you can insert arbitrary -strings at the head of each line it numbers, then it gets pretty easy to handle its output:

sed 's/^@@.*start$/@@@@@@/
     s/^@@.*end/@@/; t
     s/^\(@@\)\{1,3\}$/& /' <infile |
nl -d@@ -ha -bn -s' do something with the next line!
'

The above prints...

                                        line A
                                        line B

 1 do something with the next line!
line X
 2 do something with the next line!
line Y
 3 do something with the next line!
line Z

                                        line C
                                        line D

 1 do something with the next line!
line M
 2 do something with the next line!
line N
 3 do something with the next line!
line O

---

GNU sed

If nl is not your target application, then a GNU sed can execute an arbitrary shell command for you depending on a match.

sed '/^@@.*start$/!b
     s//nl <<\\@@/;:l;N
     s/\(\n@@\)[^\n]*end$/\1/
Tl;e'  <infile

Above sed collects input in pattern space until it has enough to successfully pass the substitution Test and stop branching back to the the :label. When it does, it executes nl with input represented as a <<here-document for all of the rest of its pattern-space.

The workflow is like this:

1. /^@@.*start$/!b
   • if an ^entire line$ does !not /match/ the above pattern, then it is branched out of the script and autoprinted - so from this point on we are only working with a series of lines which began with the pattern.
2. s//nl <<\\@@/
   • the empty s//field/ stands in for the last address sed attempted to match - so this command substitutes the entire @@.*start line for nl <<\\@@ instead.
3. :l;N
   • The : command defines a branch label - here I set one named :label. The Next command appends the next line of input to pattern space followed by a \newline character. This is one of only a few ways to get a \newline in a sed pattern space - the \newline character is a sure delimiter to a sedder who has been doing it awhile.
4. s/\(\n@@\)[^\n]*end$/\1/
   • this s///ubstitution can only be successful after a start is encountered and only on the first following occurrence of an end line. It will only act on a pattern space in which the final \newline is immediately followed by @@.*end marking the very end$ of pattern space. When it does act, it replaces the whole matched string with the \1first \(group\), or \n@@.
5. Tl
   • the Test command branches to a label (if provided) if a successful substitution has not occurred since the last time an input line was pulled into pattern space (as I do w/ N). This means that each time a \newline is appended to pattern space which does not match your end delimiter, the Test command fails and branches back to the :label, which results in sed pulling in the Next line and looping until successful.

6. e

   • When the substitution for the end match is successful and the script does not branch back for a failed Test, sed will execute a command that looks like this:

     nl <<\\@@\nline X\nline Y\nline Z\n@@$
     

You can see this for yourself by editing the last line there to look like Tl;l;e.

It prints:

line A
line B
     1  line X
     2  line Y
     3  line Z
line C
line D
     1  line M
     2  line N
     3  line O

---

while ... read

One last way to do this, and maybe the most simple way, is to use a while read loop, but for good reason. The shell - (most especially a bash shell) - is typically pretty abysmal at handling input in large amounts or in steady streams. This makes sense, too - the shell's job is to handle input character by character and to call up other commands which can handle the bigger stuff.

But importantly about its role there is that the shell must not read overmuch of the input - it is specified to not buffer input or output to the point that it consumes so much or doesn't relay enough in time that the commands it calls are left lacking - to the byte. So read makes for an excellent input test - to return information about whether there is input remaining and you should call up the next command to read it - but it is otherwise generally not the best way to go.

Here's an example, however, of how one might use read and other commands to process input in sync:

while   IFS= read -r line        &&
case    $line in (@@*start) :;;  (*)
        printf %s\\n "$line"
        sed -un "/^@@.*start$/q;p";;
esac;do sed -un "/^@@.*end$/q;=;p" |
        paste -d: - -
done    <infile

The first thing that happens for each iteration is read pulls in a line. If it is successful it means the loop has not yet hit EOF and so in the case it matches a start delimiter the do block is immediately executed. Else, printf prints the $line it read and sed is called.

sed will print every line until it encounters the start marker - when it quits input entirely. The -unbuffered switch is necessary for GNU sed because it can buffer rather greedily otherwise, but - according to spec - other POSIX seds should work without any special consideration - so long as <infile is a regular file.

When the first sed quits, the shell executes the do block of the loop - which calls another sed that prints every line until it encounters the end marker. It pipes its output to paste, because it prints line numbers each on their own line. Like this:

1
line M
2
line N
3
line O

paste then pastes those together on : characters, and the whole output looks like:

line A
line B
1:line X
2:line Y
3:line Z
line C
line D
1:line M
2:line N
3:line O

These are just examples - anything could be done in either the test or do blocks here, but the first utility must not consume too much input.

All of the utilities involved read the same input - and print their results - each in their own turn. This kind of thing can be difficult to get the hang of - because different utilities will buffer more than others - but you can generally rely on dd, head, and sed to do the right thing (though, for GNU sed, you need the cli-switch) and you should always be able to rely on read - because it is, by nature, very slow. And that's why the above loop calls it only the one time per input block.

Answer 2

One possibility is to do this with the vim text editor. It can pipe arbitrary sections through shell commands.

One way to do this is by line numbers, using :4,6!nl. This ex command will run nl on the lines 4-6 inclusive, achieving what you want on your example input.

Another, more interactive way is to select the appropriate lines using line-selection mode (shift-V) and arrow keys or search, and then using :!nl. A full command sequence for your example input could be

/@@inline-code-start
jV/@@inline-code-end
k:!nl

This is not very suited to automation (answers using e.g. sed are better for that), but for one-off edits it's very useful not having to resort to 20-line shellscripts.

If you're not familiar with vi(m), you should at the very least know that after these changes you can save the file using :wq.

Answer 3

The simplest fix I can think of is to not use nl but count the lines yourself:

#!/usr/bin/env bash
while read line
do
    if [[ $line == @@inline-code-start* ]]
    then
        active=true
    elif [[ $line == @@inline-code-end* ]]
    then
        active=false
    elif [[ $active = true ]]
    then
        ## Count the line number
        let num++;
        printf "\t%s %s\n" "$num" "$line"
    else
        # output
        printf "%s\n" "$line"
    fi
done

You then run it on the file:

$ foo.sh < file
line A
line B
    1 line X
    2 line Y
    3 line Z
line C
line D

Answer 4

If your goal is to send the entire code block to a single process instance then you could accumulate the lines and delay piping until you reach the end of the code block:

#!/bin/bash

acc=""

while read line
do
  if [[ $line == @@inline-code-start* ]]
  then
    active=true
    acc=""
  elif [[ $line == @@inline-code-end* ]]
  then
    active=false
    # Act on entire block of code
    echo "${acc:1}" | nl  # Chops off first leading new-line character using ${VAR:1}
  elif [[ $active = true ]]
  then
    acc=$( printf "%s\n%s" "$acc" "$line" )
  else
    # output
    echo $line
  fi
done

This produces the following for an input file which repeats the test case three times:

line A
line B
     1  line X
     2  line Y
     3  line Z
line C
line D
line A
line B
     1  line X
     2  line Y
     3  line Z
line C
line D
line A
line B
     1  line X
     2  line Y
     3  line Z
line C
line D

To do something else with the code block, e.g reverse and then number, just pipe it through something else: echo -E "${acc:1}" | tac | nl. Result:

line A
line B
     1  line Z
     2  line Y
     3  line X
line C
line D

Or wordcount echo -E "${acc:1}" | wc:

line A
line B
      3       6      21
line C
line D

Answer 5

Edit added an option to define a user-provided filter

#!/usr/bin/perl -s
use IPC::Open2;
our $p;
$p = "nl" unless $p;    ## default filter

$/ = "\@\@inline-code-end\n";
while(<>) { 
   chomp;
   s/\@\@inline-code-start\n(.*)/pipeit($1,$p)/se;
   print;
}

sub pipeit{my($text,$pipe)=@_;
  open2(my $R, my $W,$pipe) || die("can open2");
  local $/ = undef;
  print $W $text;
  close $W;
  return <$R>;
}

By default filter is "nl". To change the filter use option "-p" with some user provided command:

codify -p="wc" file

or

codify -p="sed -e 's@^@ ║ @; 1s@^@ ╓─\n@; \$s@\$@\n ╙─@'" file

This last filter will output:

line A
line B
 ╓─
 ║ line X
 ║ line Y
 ║ line Z
 ╙─
line C
line D

Update 1 The use of IPC::Open2 has scaling problems: if the buffersize is exceeded it may block. (in my machine the pipe buffersize if 64K correspond to 10_000 x "line Y").

If we need bigger things (is we need more the 10000 "line Y"):

(1) install and use use Forks::Super 'open2';

(2) or substitute the function pipeit by:

sub pipeit{my($text,$pipe)=@_;
  open(F,">","/tmp/_$$");
  print F $text;
  close F;
  my $out = `$pipe < /tmp/_$$ `;
  unlink "/tmp/_$$";
  return $out;
}

Answer 6

That's a job for awk.

#!/usr/bin/awk -f
$0 == "@@inline-code-start" {pipe = 1; next}
$0 == "@@inline-code-end" {pipe = 0; close("nl"); next}
pipe {print | "nl"}
!pipe {print}

When the script sees the start marker, it notes that it should start piping into nl. When the pipe variable is true (nonzero), the output is piped into the nl command; when the variable is false (unset or zero), the output is printed directly. The piped command is forked the first time the pipe construct is encountered for each command string. Subsequent evaluations of the pipe operator with the same string reuse the existing pipe; a different string value would create a different pipe. The close function closes the pipe for the given command string.

---

This is essentially the same logic as your shell script using a named pipe, but a lot easier to spell out, and the close logic done right. You need to close the pipe at the right time, to make the nl command exit, flushing its buffers. Your script actually closes the pipe too early: the pipe is closed as soon as the first echo $line >myfifo finishes executing. However the nl command only sees the end of the file if it gets a time slice before the next time the script executes echo $line >myfifo. If you had a large volume of data, or if you add sleep 1 after writing to myfifo, you'll see that nl only processes the first line or first quick bunch of lines, then it exits because it's seen the end of its input.

Using your structure, you'd need to keep the pipe open until you no longer need it. You need to have a single output redirection into the pipe.

nl <myfifo &
exec 3>&1
while IFS= read -r line
do
  if [[ $line == @@inline-code-start* ]]
  then
    exec >myfifo
  elif [[ $line == @@inline-code-end* ]]
  then
    exec >&3
  else
    printf '%s\n' "$line"
  fi
done

(I also took the opportunity to add correct quoting and such — see Why does my shell script choke on whitespace or other special characters?)

If you're doing that, you might as well use a pipeline rather than a named pipe.

while IFS= read -r line
do
  if [[ $line == @@inline-code-start* ]]
  then
    while IFS= read -r line && [[ $line != @@inline-code-end* ]] do
      printf '%s\n' "$line"
    done | nl
  else
    printf '%s\n' "$line"
  fi
done

Answer 7

OK, first off; I understand that you’re not looking for a way to number the lines in sections of your file.  Since you haven’t given an actual example of what your filter might be (other than nl), let’s suppose that it is

tr "[[:lower:]]" "[[:upper:]]"

i.e., convert text to all upper case; so, for an input of

line A
line B
@@inline-code-start
line X
line Y
line Z
@@inline-code-end
line C
line D

you want an output of

line A
line B
LINE X
LINE Y
LINE Z
line C
line D

Here’s my first approximation of a solution:

#!/bin/sh
> file0
> file1
active=0
nl -ba "$@" | while IFS= read -r line
do
        case "$line" in
            ([\ 0-9][\ 0-9][\ 0-9][\ 0-9][\ 0-9][\ 0-9]"        @@inline-code-start")
                active=1
                ;;
            ([\ 0-9][\ 0-9][\ 0-9][\ 0-9][\ 0-9][\ 0-9]"        @@inline-code-end")
                active=0
                ;;
            (*)
                printf "%s\n" "$line" >> file$active
        esac
done
(cat file0; tr "[[:lower:]]" "[[:upper:]]" < file1) | sort | sed 's/^[ 0-9]\{6\}        //'

where the spaces before the @@ strings, and near the end of the last line, are tabs.  Please note that I am using nl for my own purposes.  (Of course I’m doing it to solve your problem, but not to give you line-numbered output.)

This numbers the lines of the input so we can break it apart at the section markers and know how to put it together again later.  The main body of the loop is based on your first attempt, taking into consideration the fact that the section markers have line numbers on them.  It breaks the input apart into two files: file0 (inactive; not in a section) and file1 (active; in a section).  This is what they look like for the above input:

file0:
     1  line A
     2  line B
     8  line C
     9  line D

file1:
     4  line X
     5  line Y
     6  line Z

Then we run file1 (which is the concatenation of all the in-section lines) through the capitalization filter; combine that with the unfiltered out-of-section lines; sort, to put them back into their original order; and then strip off the line numbers.  This produces the output shown near the top of my answer.

This assumes that your filter leaves the line numbers alone.  If it doesn’t (e.g., if it inserts or deletes characters at the beginning of the line), then, I believe, this general approach can still be used, but will require some slightly trickier coding.

Answer 8

A shell script that uses sed to output chunks of non-demarcated lines and feed demarcated chunks of lines into a filter program:

#!/bin/bash

usage(){
    echo "  usage: $0 <input file>"
}

# Check input file
if [ ! -f "$1" ]; then
    usage
    exit 1
fi

# Program to use for filtering
# e.g. FILTER='tr X -'
FILTER='./filter.sh'

# Generate arrays with starting/ending line numbers of demarcators
startposs=($(grep -n '^@@inline-code-start$' "$1" | cut -d: -f1))
endposs=($(grep -n '^@@inline-code-end$' "$1" | cut -d: -f1))

nums=${#startposs[*]}
nume=${#endposs[*]}

# Verify both line number arrays have the same number of elements
if (($nums != $nume)); then
    echo "Tag mismatch"
    exit 2
fi

lastline=1
i=0
while ((i < nums)); do
    # Exclude lines with code demarcators
    sprev=$((${startposs[$i]} - 1))
    snext=$((${startposs[$i]} + 1))
    eprev=$((${endposs[$i]} - 1))

    # Don't run this bit if the first demarcator is on the first line
    if ((sprev > 1)); then
        # Output lines leading up to start demarcator
        sed -n "${lastline},${sprev} p" "$1"
    fi

    # Filter lines between demarcators
    sed -n "${snext},${eprev} p" "$1" | $FILTER

    lastline=$((${endposs[$i]} + 1))
    let i++
done

# Output lines (if any) following last demarcator
sed -n "${lastline},$ p" "$1"

I wrote this script into a file named detagger.sh and used it as so: ./detagger.sh infile.txt. I created a separate filter.sh file to mimic the filtering functionality in the question:

#!/bin/bash
awk '{ print "\t" NR " " $0}'

But the filtering operation can be changed in the code.

I attempted to follow the idea of a generic solution with this so that operations like numbering lines do not require additional/internal counting . The script does some rudimentary checking to see that demarcator tags are in pairs and does not handle nested tags gracefully at all.

Answer 9

Thanks for all the great ideas. I've come up with my own solution by keeping track of the subsection in a temp file and piping it all at once to my external command. This is very similar to what Supr suggested (but with a shell variable instead of temp file). Also, I really like the idea of using sed, but the syntax for this case seems a bit over the top for me.

My solution:

(I use nl just as an example filter)

#!/usr/bin/bash

while read line
do
  if [[ $line == @@inline-code-start* ]]
  then
    active=true
    tmpfile=$(mktemp)
    trap "rm -f $tmpfile" EXIT
  elif [[ $line == @@inline-code-end* ]]
  then
    active=false
    <$tmpfile nl
    rm $tmpfile
  elif [[ $active = true ]]
  then
    echo $line >> $tmpfile
  else
    echo $line
  fi
done

I would prefer not to have to deal with managing the temp files, but I understand that shell variables can have rather low size limits, and I don't know of any bash construct that would work like a temp file, but vanish automatically when the process ends.