# Reorder Multiple Line Blocks with Sed

I am trying to reorder generated output as follows; given a tex fragment as follows, I want to move the /^Constant/ down below /Dummies/, with the line before and after /^Constant/. Further, I would like a stable result, one that does not make a change if Constant is already below the last Dummies row (it runs as part of a script to fix up my research results, and so far that script is stable).

This is the tex fragment input:

[1em]
Application Grade&                  &                  &  -0.0857\sym{***}& -0.00412\sym{***}\\
&                  &                  & (0.0149)         &(0.00107)         \\
[1em]
Constant        &   -3.701\sym{***}&   -0.311\sym{***}&        0         &        0         \\
&  (1.130)         & (0.0853)         &      (.)         &      (.)         \\
[1em]
Major Dummies   &       No         &       No         &      Yes         &      Yes         \\
[1em]
Semester Dummies &       No         &       No         &      Yes         &      Yes         \\
\hline


This is the desired output, which remains stable (returns the same) if given as input:

[1em]
Application Grade&                  &                  &  -0.0857\sym{***}& -0.00412\sym{***}\\
&                  &                  & (0.0149)         &(0.00107)         \\
[1em]
Major Dummies   &       No         &       No         &      Yes         &      Yes         \\
[1em]
Semester Dummies &       No         &       No         &      Yes         &      Yes         \\
[1em]
Constant        &   -3.701\sym{***}&   -0.311\sym{***}&        0         &        0         \\
&  (1.130)         & (0.0853)         &      (.)         &      (.)         \\
\hline


The following code will do so, but it is not stable. Basically it matches on Constant, drops and re-inserts [1em] (which it has to do on a separate call since i\ affects the output, not the pattern space), and pastes below the target. It's not stable because it will add bogus [1em]s after Semester Dummies. Frankly, its a bit ugly (requiring separate calls to sed).

sed -E -i'' -e'/^Constant/ {N;h;N;d;}; /^Semester Dummies/ {G;};' fragment.tex
sed -E -i'' -e '/^Semester Dummies/ {a\
$1em$
};' fragment.tex


I believe a one-liner will do it, matching the multiline pattern /^$1em$\nConstant.*/ (after a call to N;N;), putting this in the hold buffer, and then pasting it (G;) after /Semester Dummies/. But after many hours, man pages, web searches, and upgrading to gnused for good measure, I cannot get such a script to work. This expresses the basic idea but is not valid (s cannot conditionally hold as I do here):

sed -E -i'' -e'/$1em$/ {N;N;s/^$1em$\nConstant.*//h;}; /Semester Dummies/ {G;}' fragment.tex


I tried many variations relying on N;P;D;h;G; with various addressing but I still don't understand the execution order for multiply addressed commands when you are messing with the pattern space (calling N;). N and P work nicely, but as far as I can tell, D is completely useless.

Yes, I know it is easier with awk and perl; in fact, I would be interested in seeing all three solutions for comparison--but here I am specifically asking how to do it in sed.

-

Here's an ugly piece of awk that plays tricks with the record separator

awk -v RS='\$1em\$\n|\\\\hline' '
!/[^[:space:]]/ {next}
/^Constant/ {c=$0; next} {printf "[1em]\n%s",$0}
END {printf "[1em]\n%s\\hline\n", c}
'


Since the text starts and ends with "record separator", there are some spurious empty-ish records, hence the first rule

-
This is helpful, though it produces somewhat corrupted output (it seems to replace any backslashes present in the source tex file with '[1em]\n' (a real newline)). I think awk is probably the right tool for this anyway, so thanks for the hints. – rjturn Feb 25 '14 at 21:56

Here is one way to do it with sed using the hold buffer:

sed '/$1em$/{N;/Constant/{N;x;/^$/d;x};/Semester/{x;/^$/!{H;x};//g}}' file


On each line matching [1em] it reads in the Next line and then

1. If the pattern space matches Constant it reads in aNother line, exchanges the buffers and if the pattern space is just an empty line (that means the hold buffer was empty before exchanging) it is deleted. If the pattern space is not an empty line (which means there was something in the hold space because of 2 ) it exchanges the buffers again, returning the lines in the pattern space.
2. If the pattern space matches Semester it exchanges the buffers then, if the patters space isn't empty it means the lines [1em]\nConstant.* were in the hold buffer so it appends them to the [1em]\nSemester.* lines (which are now in the Hold buffer) and then exchanges buffers again. If the pattern space is empty it means the lines [1em]\nConstant.* are after the Semester line so it just copies the hold space over the pattern space. That way the [1em]\nSemester.* lines are restored but now there is something in the hold buffer when it reaches 1.

This way, the [1em]\nConstant.* lines are moved only if they are before [1em]\nSemester.*, otherwise nothing happens.

sed '/$1em$/{             # if line matches [1em]
N                           # read in the next line
/Constant/{                 # if pattern space matches Constant
N                           # read in another line
x                           # exchange buffers
/^$/d # if pattern space is now empty, delete it x # otherwise, exchange again } /Semester/{ # if pattern space matches Semester x # exchange buffers /^$/!{                      # if pattern space is not empty
H                           # append it to hold space
x                           # then exchange buffers
}
//g                         # if pattern space is currently empty
}                           # copy the hold space over the pattern space
}' infile                   # so now the hold space is no longer empty


This is easier with ed. Just select the range of lines and move it after the line matching Semester:

ed -s infile <<<\$'/Constant/-1,/Constant/+1m/Semester/\n,p\nq'


replace ,p with w to write changes to file:

ed -s infile <<IN
/Constant/-1,/Constant/+1m/Semester/
w
q
IN

-