How to join lines that meet until pattern until the same pattern occurs

Question

I have a file :

 "p1"data
 "p2"data
 "p3"data
 "p1"data
 "p2"data
 "p3"data
 "p2"data
 "p3"data
 "p1"data
 

I want concatenate all lines with "p1" until the next "p1" is encountered.

 "p1"data"p2"data"p3"data
 "p1"data"p2"data"p3"data"p2"data"p3"data
 "p1"data
 

The quotes are literal quotes. There may be anywhere from 1 to 10 groups of "p2" and "p3" The input file has thousands of lines. The output file should have about 600 lines.

I tried using sed -e'/^"p1/N;s/\n//' but I needed to run it multiple times and it eventually concatenates "p1" with another "p1"

Any assistance would be appreciated. FYI this file is from an XML file. I am aware that there are xml tools, but I want to do this using sed.

Answer 1

based on this, with GNU sed (assuming there's no $POSIXLY_CORRECT in the environment):

sed ':a;N;/\n"p1"/!s/\n//;ta;P;D' file

In standard sed syntax, you can't have anything even comments after branching labels, and running N on the last line discards the pattern space, so it would have to be:

sed -e :a -e '$!N;/\n"p1"/!s/\n//;ta' -e 'P;D' file

With comments:

sed '
  # Label to jump to:
  :a
  # Append next line to pattern space unless we are
  # on the last line:
  $!N
  # If the newline is NOT followed by "p1", append
  # the line by replacing the newline with a space:
  /\n"p1"/!s/\n//
  # If we changed something, jump to label:
  ta
  # Print part until newline
  P
  # Delete part until newline
  D' file

Answer 2

sed isn't a great choice of tool for this. Using any awk in any shell on every Unix box:

$ awk -v ORS= 'NR>1 && /^"p1"/{print RS} {print} END{print RS}' file
"p1"data"p2"data"p3"data
"p1"data"p2"data"p3"data"p2"data"p3"data
"p1"data

Answer 3

I'd use perl for this, not sed - because perl's support for multi-line strings makes it easy:

$ perl -0777 -n -e 's/\n//g; s/"p1"/\n$&/g; s/^\n//; print "$_\n"' input.txt 
"p1"data"p2"data"p3"data
"p1"data"p2"data"p3"data"p2"data"p3"data
"p1"data

• -0777 tells perl to read the entire file into memory at once (i.e. "slurps" it into variable $_)
• -n makes perl run similarly to sed -n (i.e. read the input without auto-printing anything)

The script first deletes all newline characters, then adds a new line before every "p1", then deletes the newline that probably adds to the beginning of the string (assuming the first line begins with "p1").

Finally, the modified input is printed with a trailing newline (so that the output is valid for a text file - unix text files must end in a newline. Many tools work just fine with almost-text-files that don't end with a newline but a) some don't - they can't process the last "line" because it doesn't end with a newline, according to POSIX that technically isn't a "line", b) while it's good to be flexible in what you accept as input, it's even better to produce correct output, c) not printing the final newline looks ugly and causes the next shell prompt to appear on the same line as the output, and d) similarly it causes problems when cat-ing multiple files or appending text to a file. See What's the point in adding a new line to the end of a file?)

Alternatively:

$ perl -0777 -n -e 's/\n//g; s/(.)("p1")/$1\n$2/g; print "$_\n"' input.txt 
"p1"data"p2"data"p3"data
"p1"data"p2"data"p3"data"p2"data"p3"data
"p1"data

Like the first version, this deletes all newline characters, but then it adds a newline before every instance of "p1" that follows another character (.) - i.e. not the first line. Then it prints the modified input with a trailing newline.

Yet another variant:

$ perl -0777 -p -e 's/\n//g; s/(.)("p1")/$1\n$2/g; s/$/\n/' input.txt 
"p1"data"p2"data"p3"data
"p1"data"p2"data"p3"data"p2"data"p3"data
"p1"data

This uses perl's -p option instead of -n. -p makes perl run like sed (i.e. read the input and automatically print it after any modifications). Otherwise, it's very similar to the second version above but uses s/$/\n/ to add a newline to the end of the input before it is auto-printed.

Answer 4

Using the ed editor:

v/^"p1"/ -,. j

This would join each line that doesn't start with the substring "p1" with the previous line.

This assumes that the first line starts with the "p1" substring. If that is not guaranteed to be the case, then avoid running the v command on the first line:

2,$ v/^"p1"/ -,. j

Testing on the data given in the question:

$ printf '%s\n' 'v/^"p1"/ -,. j' ,p Q | ed -s file
"p1"data"p2"data"p3"data
"p1"data"p2"data"p3"data"p2"data"p3"data
"p1"data