Extract lines from a text file based on a parameter to a single line

Question

I'm quite new to scripting so I appreciate any help. I have a text file that in some cases can be quite long, each section of text lines can be around 6/7 lines long. It is a logfile and each section starts with the word timestamp. There is a blank line between each section of lines. Each section line ends with a semi-colon.

timestamp=201706291035.....;
  line 2;
  line 3;
  line 4;
  line 5;
  line 6;
  line 7;

timestamp=201706291038.....;
  line 2;
  line 3;
  line 4;
  line 5;
  line 6;

I need to be able to extract each section out to another text file in a single line. Preferably without the last semi-colon.

timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7
timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6

Is that enough information for a solution?

Here's a direct example:

timestamp=2017-06-28-01.01.35.080576;
event status=0;
userid=user1;
authid=user1;
application id=10.10.10.10.11111.12345678901;
application name=GUI;

timestamp=2017-06-28-01.01.36.096486;
event status=0;
userid=user1;
authid=user1;
application id=10.10.10.10.11111.12345678901;
application name=GUI;
statement text=SELECT table.field, table.field, table.field from database where table.field = value

After I run the scripts @steeldriver, the source and destination file looks the same.

Answer 1

This could be done with idiomatic awk like so:

awk '$1=$1' RS= OFS= infile

Output:

timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7;
timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6;

Explanation

There is a lot packed in here. Basically there are three steps:

1. First the input is split into records based on the record separator (RS).
2. Each record is split into fields based on the field separator (FS).
3. When printing, the output field separator (OFS) is used as the field delimiter.

When awk parses its input, there are several implicit rules at work. The data is read, one record at a time, records being separated by RS (default is \n). When RS is empty, as in the above example, an empty line delimits records. Thus each section is read in as a record.

In order to force awk to replace FS with OFS we set the first field $1 to itself.

Edit

As noted by steeldriver, the OP wants to remove trailing semi-colons. Shamelessly copied:

awk '{ sub(/;$/,"",$NF); $1=$1 } 1' RS= OFS= infile

Answer 2

This can be done using the following way:

perl -lF';\n?' -00ne '$,=";"; print @F' yourfile

---

Output

timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7
timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6

---

Working

1. Perl options

   a) -l => ORS="\n" + RS = "\n"

   b) -F';\n?' => will make the FS to be a semicolon followed by an optional newline.

   c) -00 => will make RS= thereby enabling paragraph mode.

   d) -n => will enable the implicit file read-in + explicit printing.

2. Main: $,=; will make OFS a semicolon, @F is the fields which have been carved out of the current record $_ based on the FS.

Answer 3

If there's an empty line before the timestamp, you can use a simple

perl -pe 'chomp unless /^$/'

If the newlines are not there, you need to remember the previous line.

perl -pe 'chomp; print "\n" if $. > 1 && /^timestamp=/; print }{ print "\n"'

Answer 4

The shortest answer for the use case would be:

awk '$1=$1' > "single.txt" RS= test.txt

As RS was explained by Thor, we only need RS here to get our result.

Answer 5

Just because, here's a way to do it in sed

Take as our starting point this one-liner from Peter Krumins' Sed One-Liners Explained, Part I: File Spacing, Numbering and Text Conversion and Substitution

39. Append a line to the next if it ends with a backslash "\".

    sed -e :a -e '/\\$/N; s/\\\n//; ta'
    

The first expression ':a' creates a named label "a". The second expression looks to see if the current line ends with a backslash "\". If it does, it joins it with the line following it using the "N" command. Then the slash and the newline between joined lines get erased with "s/\\n//" command. If the substitution was successful we branch to the beginning of expression and do the same again, in hope that we might have another backslash. If the substitution was not successful, the line did not end with a backslash and we print it out.

replacing \\ by ; and adjusting the replacement to leave the ; but remove the leading spaces, we get

$ sed -e :a -e '/;$/N; s/\n *//; ta' infile
timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7;

timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6;

Close! now we want to squeeze out the blank line - we can do that by testing if the pattern ends in a newline (i.e. the appended line is empty) and if so printing up to the newline and then discarding the pattern:

$ sed -e :a -e '/;$/N; /\n$/{P;d;}; s/\n *//; ta' infile
timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7;
timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6;

Now we just need to trim the trailing ;. One way to do that is to remove each ; as we append the line to the pattern space, and then re-insert it when we discard the newline:

$ sed -e :a -e '/;$/{s///;N;}; /\n$/{P;d;}; s/\n */;/; ta' infile
timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7
timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6

The final ; doesn't get re-inserted because we've already eaten the newline with the {P;d;} so the s//\n /;/ substitution doesn't get applied.