1

I'm quite new to scripting so I appreciate any help. I have a text file that in some cases can be quite long, each section of text lines can be around 6/7 lines long. It is a logfile and each section starts with the word timestamp. There is a blank line between each section of lines. Each section line ends with a semi-colon.

timestamp=201706291035.....;
  line 2;
  line 3;
  line 4;
  line 5;
  line 6;
  line 7;

timestamp=201706291038.....;
  line 2;
  line 3;
  line 4;
  line 5;
  line 6;

I need to be able to extract each section out to another text file in a single line. Preferably without the last semi-colon.

timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7
timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6

Is that enough information for a solution?

Here's a direct example:

timestamp=2017-06-28-01.01.35.080576;
event status=0;
userid=user1;
authid=user1;
application id=10.10.10.10.11111.12345678901;
application name=GUI;

timestamp=2017-06-28-01.01.36.096486;
event status=0;
userid=user1;
authid=user1;
application id=10.10.10.10.11111.12345678901;
application name=GUI;
statement text=SELECT table.field, table.field, table.field from database where table.field = value

After I run the scripts @steeldriver, the source and destination file looks the same.

  • I boldly edited based on what I think was meant, do check if I got it right. – ilkkachu Jun 29 '17 at 19:27
  • Yes, the example is correct now. When I entered the example, it ended up all on the same line. Sorry. – david Jun 29 '17 at 19:28
  • Is there an empty line before the timestamp? – choroba Jun 29 '17 at 19:42
  • Typically the logfile doesn't have an empty line at the beginning prior to the timestamp. If it does, I can remove it. – david Jun 29 '17 at 19:50
  • I screwed up with the example, each line after the timestamp is indented 2 spaces. I'm not sure if that would make a difference though. – david Jun 30 '17 at 14:55
4

This could be done with idiomatic awk like so:

awk '$1=$1' RS= OFS= infile

Output:

timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7;
timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6;

Explanation

There is a lot packed in here. Basically there are three steps:

  1. First the input is split into records based on the record separator (RS).
  2. Each record is split into fields based on the field separator (FS).
  3. When printing, the output field separator (OFS) is used as the field delimiter.

When awk parses its input, there are several implicit rules at work. The data is read, one record at a time, records being separated by RS (default is \n). When RS is empty, as in the above example, an empty line delimits records. Thus each section is read in as a record.

In order to force awk to replace FS with OFS we set the first field $1 to itself.

Edit

As noted by steeldriver, the OP wants to remove trailing semi-colons. Shamelessly copied:

awk '{ sub(/;$/,"",$NF); $1=$1 } 1' RS= OFS= infile
  • Why not just awk '$1=$1' RS= filename We'd only need the RS here. – Valentin Bajrami Jun 29 '17 at 20:25
  • Again, brand new here, so would the command be like this? awk '$1=$1' RS= FS='\n' OFS= test.txt > single.txt where test.txt is the original file. – david Jun 29 '17 at 20:52
  • @david you only need awk '$1=$1' > "single.txt" RS= test.txt – Valentin Bajrami Jun 29 '17 at 21:02
  • 1
    awk 'sub(/;$/,"",$NF)' RS= OFS= file would strip the trailing semicolon from the last field, as the OP also requested. (Note that in this form, it only prints lines for which the replacement is successful; if that's not the desired result then awk '{sub(/;$/,"",$NF)} 1' RS= OFS= file). – steeldriver Jun 29 '17 at 22:01
  • 1
    @val0x00ff: Right, but according to the OP we also need OFS= – Thor Jun 29 '17 at 22:11
1

This can be done using the following way:

perl -lF';\n?' -00ne '$,=";"; print @F' yourfile

Output

timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7
timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6

Working

  1. Perl options

    a) -l => ORS="\n" + RS = "\n"

    b) -F';\n?' => will make the FS to be a semicolon followed by an optional newline.

    c) -00 => will make RS= thereby enabling paragraph mode.

    d) -n => will enable the implicit file read-in + explicit printing.

  2. Main: $,=; will make OFS a semicolon, @F is the fields which have been carved out of the current record $_ based on the FS.

0

If there's an empty line before the timestamp, you can use a simple

perl -pe 'chomp unless /^$/'

If the newlines are not there, you need to remember the previous line.

perl -pe 'chomp; print "\n" if $. > 1 && /^timestamp=/; print }{ print "\n"'
  • Don't know anything about perl. If the source file is test.txt and the destination file is single.txt, what would the command line look like? Thanks for your help. – david Jun 30 '17 at 14:48
  • As with anything else, perl -pe '...' < test.txt > single.txt – choroba Jun 30 '17 at 14:50
  • Thanks. Does it make a difference if the lines following the timestamp line, are indented by 2 spaces? The output file looks the same as the input file. – david Jun 30 '17 at 15:35
  • @david: The code only checks the empty lines by ^$ or the substring timestamp= at the beginning of a line. – choroba Jun 30 '17 at 16:26
0

The shortest answer for the use case would be:

awk '$1=$1' > "single.txt" RS= test.txt

As RS was explained by Thor, we only need RS here to get our result.

  • When I run the above command the output I get is - statement text=SELECT table.field, table.field, table.field from database where table.field = value. This is the last line in the original file. – david Jul 4 '17 at 17:35
0

Just because, here's a way to do it in sed

Take as our starting point this one-liner from Peter Krumins' Sed One-Liners Explained, Part I: File Spacing, Numbering and Text Conversion and Substitution

  1. Append a line to the next if it ends with a backslash "\".

    sed -e :a -e '/\\$/N; s/\\\n//; ta'
    

The first expression ':a' creates a named label "a". The second expression looks to see if the current line ends with a backslash "\". If it does, it joins it with the line following it using the "N" command. Then the slash and the newline between joined lines get erased with "s/\\n//" command. If the substitution was successful we branch to the beginning of expression and do the same again, in hope that we might have another backslash. If the substitution was not successful, the line did not end with a backslash and we print it out.

replacing \\ by ; and adjusting the replacement to leave the ; but remove the leading spaces, we get

$ sed -e :a -e '/;$/N; s/\n *//; ta' infile
timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7;

timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6;

Close! now we want to squeeze out the blank line - we can do that by testing if the pattern ends in a newline (i.e. the appended line is empty) and if so printing up to the newline and then discarding the pattern:

$ sed -e :a -e '/;$/N; /\n$/{P;d;}; s/\n *//; ta' infile
timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7;
timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6;

Now we just need to trim the trailing ;. One way to do that is to remove each ; as we append the line to the pattern space, and then re-insert it when we discard the newline:

$ sed -e :a -e '/;$/{s///;N;}; /\n$/{P;d;}; s/\n */;/; ta' infile
timestamp=201706291035.....;line 2;line 3;line 4;line 5;line 6;line 7
timestamp=201706291038.....;line 2;line 3;line 4;line 5;line 6

The final ; doesn't get re-inserted because we've already eaten the newline with the {P;d;} so the s//\n /;/ substitution doesn't get applied.

  • Still doesn't seem to work for me. The script runs without error but the source and destination file look the same. – david Jun 30 '17 at 18:59
  • timestamp=2017-06-28-01.01.35.080576;<br/> event status=0;<br/> userid=user1;<br/> authid=user1;<br/> application id=10.10.10.10.11111.12345678901;<br/> application name=GUI;<br/> <br/> timestamp=2017-06-28-01.01.36.096486;<br/> event status=0;<br/> userid=user1;<br/> authid=user1;<br/> application id=10.10.10.10.11111.12345678901;<br/> application name=GUI;<br/> statement text=SELECT table.field, table.field, table.field from database where table.field = value<br/> – david Jun 30 '17 at 19:01
  • @david it's hopeless posting as a comment - please edit your original question – steeldriver Jun 30 '17 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.