9

I have a Log file which need to be parsed and analysed. File contains something similar like below:

File:

20141101 server contain dump
20141101 server contain nothing
    {uekdmsam ikdas 

jwdjamc ksadkek} ssfjddkc * kdlsdl
sddsfd jfkdfk 
20141101 server contain dump

Based on the above scenario, I have to check if the starting line doesn't contain date or Number I have to append to previous line.

Output file:

20141101 server contain dump
20141101 server contain nothing {uekdmsam ikdas jwdjamc ksadkek} ssfjddkc * kdlsdl sddsfd jfkdfk 
20141101 server contain dump
11

A version in perl, using negative lookaheads:

$ perl -0pe 's/\n(?!([0-9]{8}|$))//g' test.txt
20141101 server contain dump
20141101 server contain nothing    {uekdmsam ikdas jwdjamc ksadkek} ssfjddkc * kdlsdlsddsfd jfkdfk
20141101 server contain dump

-0 allows the regex to be matched across the entire file, and \n(?!([0-9]{8}|$)) is a negative lookahead, meaning a newline not followed by 8 digits, or end of the line (which, with -0, will be the end of the file).

  • @terdon, updated to save last newline. – muru Nov 13 '14 at 18:15
  • Nice one! I'd upvote you but I'm afraid I already had :) – terdon Nov 13 '14 at 18:17
  • No, -0 if for NUL-delimited records. Use -0777 to slurp the entire file in memory (which you don't need to here). – Stéphane Chazelas Nov 14 '14 at 12:16
  • @StéphaneChazelas So whats the best way to make Perl match the newline, other than reading the whole file in? – muru Nov 14 '14 at 12:18
  • See the other answers that process the file line by line. – Stéphane Chazelas Nov 14 '14 at 14:23
5

May be a little bit easy with sed

sed -e ':1 ; N ; $!b1' -e 's/\n\+\( *[^0-9]\)/\1/g'
  • first part :1;N;$!b1 collect all lines in file divided by \n in 1 long line

  • second part strip newline symbol if it followed non-digit symbol with possible spaces between its.

To avoid memory limitation (espesially for big files) you can use:

sed -e '1{h;d}' -e '1!{/^[0-9]/!{H;d};/^[0-9]/x;$G}' -e 's/\n\+\( *[^0-9]\)/\1/g'

Or forget a difficult sedscripts and to remember that year starts from 2

tr '\n2' ' \n' | sed -e '1!s/^/2/' -e 1{/^$/d} -e $a
  • Nice, +1. Could you add an explanation of how it works please? – terdon Nov 13 '14 at 18:24
  • 1
    Aw. Nice. I always do tr '\n' $'\a' | sed $'s/\a\a*\( *[^0-9]\)/\1/g' | tr $'\a' '\n' myself. – mirabilos Nov 14 '14 at 9:23
  • Sorry, have to downvote though for using things that are not POSIX BASIC REGULAR EXPRESSIONS in sed(1), which is a GNUism. – mirabilos Nov 14 '14 at 9:23
  • 1
    @Costas, that's GNU grep's man page. POSIX BRE spec are there. BRE equivalent of ERE + is \{1,\}. [\n] is not portable either. \n\{1,\} would be POSIX. – Stéphane Chazelas Nov 14 '14 at 12:18
  • 1
    Also, you can't have another command after a label. : 1;x is to define the 1;x label in POSIX seds. So you need: sed -e :1 -e 'N;$!b1' -e 's/\n\{1,\}\( *[^0-9]\)/\1/g'. Also note that many sed implementations have a small limit on the size of their pattern space (POSIX only guarantees 10 x LINE_MAX IIRC). – Stéphane Chazelas Nov 14 '14 at 12:28
5

One way would be:

 $ perl -lne 's/^/\n/ if $.>1 && /^\d+/; printf "%s",$_' file
 20141101 server contain dump
 20141101 server contain nothing    {uekdmsam ikdas jwdjamc ksadkek} ssfjddkc * kdlsdlsddsfd jfkdfk 
 20141101 server contain dump

However, .that also removes the final newline. To add it again, use:

$ { perl -lne 's/^/\n/ if $.>1 && /^\d+/; printf "%s",$_' file; echo; } > new

Explanation

The -l will remove trailing newlines (and also add one to each print call which is why I use printf instead. Then, if the current line starts with numbers (/^\d+/) and the current line number is greater than one ($.>1, this is needed to avoid adding an extra empty line at the beginning), add a \n to the beginning of the line. The printf prints each line.


Alternatively, you can change all \n characters to \0, then change those \0 that are right before a string of numbers to \n again:

$ tr '\n' '\0' < file | perl -pe 's/\0\d+ |$/\n$&/g' | tr -d '\0'
20141101 server contain dump
20141101 server contain nothing    {uekdmsam ikdas jwdjamc ksadkek} ssfjddkc * kdlsdlsddsfd jfkdfk 
20141101 server contain dump

To make it match only strings of 8 numbers, use this instead:

$ tr '\n' '\0' < file | perl -pe 's/\0\d{8} |$/\n$&/g' | tr -d '\0'
  • The first argument to printf is the format. Use printf "%s", $_ – Stéphane Chazelas Nov 14 '14 at 12:31
  • @StéphaneChazelas why? I mean, I know it's cleaner and perhaps easier to understand but is there any danger that that would protect from? – terdon Nov 14 '14 at 13:04
  • Yes, it's wrong and potentially dangerous if the input may contain % characters. Try with an input with %10000000000s for instance. – Stéphane Chazelas Nov 14 '14 at 14:09
  • In C, that's a very well known very bad practice and vulnerability source. With perl, echo %.10000000000f | perl -ne printf brings my machine to its knees. – Stéphane Chazelas Nov 14 '14 at 14:19
  • @StéphaneChazelas wow, yes. Mine too. Fair enough then, answer edited and thanks. – terdon Nov 14 '14 at 14:39
3

Try doing this using :

#!/usr/bin/awk -f

{
    # if the current line begins with 8 digits followed by
    # 'nothing' OR the current line doesn't start with 8 digits
    if (/^[0-9]{8}.*nothing/ || !/^[0-9]{8}/) {
        # print current line without newline
        printf "%s", $0
        # feeding a 'state' variable
        weird=1
    }
    else {
        # if last line was treated in the 'if' statement
        if (weird==1) {
            printf "\n%s", $0
            weird=0
        }
        else {
            print # print the current line
        }
    }
}
END{
    print # add a newline when there's no more line to treat
}

To use it:

chmod +x script.awk
./script.awk file.txt
2

Another simplest way (than my other answer) using and terdon's algorithm :

awk 'NR>1 && /^[0-9]{8}/{printf "%s","\n"$0;next}{printf "%s",$0}END{print}' file
  • ITYM END{print ""}. Alternative: awk -v ORS= 'NR>1 && /^[0-9]{8}/{print "\n"};1;END{print "\n"}' – Stéphane Chazelas Nov 14 '14 at 15:17
1
sed -e:t -e '$!N;/\n *[0-9]{6}/!s/\n */ /;tt' -eP\;D
0

Le programme en bash:

while read LINE
do
    if [[ $LINE =~ ^[0-9]{8} ]]
    then
        echo -ne "\n${LINE} "
    else
        echo -n "${LINE} "
    fi
done < file.txt

in one-line form:

while read L; do if [[ $L =~ ^[0-9]{8} ]]; then echo -ne "\n${L} "; else echo -n "${L} "; fi done < file.txt

Solution with backslashes preserving (read -r) and leading spaces (just IFS= after while):

while IFS= read -r LINE
do
    if [[ $LINE =~ ^[0-9]{8} ]]
    then
        echo
        echo -nE "\n${LINE} "
    else
        echo -nE "${LINE} "
    fi
done < file.txt

one-line form:

while IFS= read -r L; do if [[ $L =~ ^[0-9]{8} ]]; then echo; echo -nE "${L} "; else echo -nE "${L} "; fi done < file.text
  • This will break if the line contains, say, a backslash and an n. It also strips whitespace. But you can use mksh to do this: while IFS= read -r L; do [[ $L = [0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]* ]] && print; print -nr -- "$L"; done; print – mirabilos Nov 15 '14 at 14:32
  • Of course it is not for everything algorithm, but solution for the requirements provided by the task. Of course the final solution will be more complex and less readable at a glance as it usually happens in Real Life :) – rook Nov 17 '14 at 10:12
  • I agree, but I’ve learned the hard way to not assume too much about the OP ☺ especially if they replace the actual text by dummy text. – mirabilos Nov 17 '14 at 12:55
0
[shyam@localhost ~]$ perl -lne 's/^/\n/ if $.>1 && /^\d+/; printf "%s",$_' appendDateText.txt

that will work

i/p:
##06/12/2016 20:30 Test Test Test
##TestTest
##06/12/2019 20:30 abbs  abcbcb abcbc
##06/11/2016 20:30 test test
##i123312331233123312331233123312331233123312331233Test
## 06/12/2016 20:30 abc

o/p:
##06/12/2016 20:30 Test Test TestTestTest
##06/12/2019 20:30 abbs  abcbcb abcbc
##06/11/2016 20:30 test ##testi123312331233123312331233123312331233123312331233Test
06/12/2016 20:30 abc vi appendDateText.txt 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.