1

I have been tasked with parsing a very large single line text file to put into a database. The file contains huge amounts of text data (48 gigs) and was provided me in this format:

col1*#(%&col2*#(%&col3*#(%&col4*#(%&col5*#(%&col1*#(%&col2*#(%&col3*#(%&col4...

So the delimiter in this file is '*#(%&' the columns basically loop from col1-col5 in a single line.

My goal is to try to get these into a record format ie:

col1*#(%&col2*#(%&col3*#(%&col4*#(%&col5*#(%&
col1*#(%&col2*#(%&col3*#(%&col4*#(%&col5*#(%&
col1*#(%&col2*#(%&col3*#(%&col4*#(%&col5*#(%&

So I want to add a '\n' after every 5 '*#(%&'. I looked around and found a handy sed command that is able to do such thing:

sed -r 's/([^\*#\(%\&|]*\*#\(%\&){5}/&\n/g'

For the most part, this works, however, due to col4 being an enormous text field, I noticed that for records where a col contains any single character of '#(%&' ie '#', the count seems to reset and it's not behaving the way that I want. Is there a way to tweak or circumvent this from happening? I just want a new line only when the exact pattern of '#(%&' occurs.

  • Does sed -e 's/#(%&/&\n/5' -e 'P;D' have the same issue? – steeldriver Oct 8 '18 at 0:26
  • Oh wow this does fix it. What exactly is the 'P;D' doing here? – user3348557 Oct 8 '18 at 0:38
  • 1
    P is printing everything up to the first \n, then D is deleting that portion of the pattern space before implicitly repeating the "replace the 5th delimiter" command – steeldriver Oct 8 '18 at 0:45
  • Gotcha. Is there a reason why the '*' was removed as well? – user3348557 Oct 8 '18 at 0:46
  • Oops, no - I just missed that that was part of the delimiter - you should probably use s/\*#(%&/&\n/5 – steeldriver Oct 8 '18 at 0:48
1

With GNU awk, you could do something like:

gawk -v RS='\\*#\\(%&' -v ORS= '{print $0 RT};NR%5 == 0{printf "\n"}'

With sed (but beware that some sed implementations have a low limit on the size of lines):

sed 's/*#(%&/&\
/5;P;D'
  • Interesting.I was not aware of the low limit restriction that could exist with SED. I will see if this happens later today and try your awk implementation. – user3348557 Oct 8 '18 at 16:34
  • Question, if i want to write to a file with your awk command, how would i go about doing this? – user3348557 Oct 12 '18 at 21:08
0
perl -F'\*#\(%&' -lane 'print join "*#(%&", splice @F, 0, 5 while @F'

Working:

 ° Split on the string `*#(%&`  it is stored in the array @F.
 ° Then, while the array still has elements in it, pluck out the leading 5, or whatever are left in the last gasp, and join these with the same string they were split on. And this is printed to stdout followed by a newline which is ensured by the `-l` option. 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.