1

I'm trying to parse a large text file (600MB) of raw data which I have no control on its format.

The problem is that the end of line is not placed well:

head -1 filename | wc -w

Produces a count of ~1.2K words per line. My target lines are are delimited by the \x06 character.

How can I re-organize the lines so that each one would end/start with ^F (\x06)? I've read many sed tutorials, but none mentioned this kind of a process...

^F1990451178631 E04:19:03.86 USER_INPUT 0 ID=34586238477^F1990451178631 E04:19:03.86 USER_INPUT 0 ID=34586238477^F1990451178631 E04:19:03.86 USER_INPUT 0 ID=34586238477^F1990451178631 E04:19:03.86 USER_INPUT 0 ID=34586238477^F1990451178631 E04:19:03.86 USER_INPUT 0 ID=34586238477^F1990451178631 E04:19:03.86 USER_INPUT 0 ID=34586238477^F1990451178631 E04:19:03.86 USER_INPUT 0 ID=34586238477^F1990451178631 E04:19:03.86 USER_INPUT 0 ID=34586238477^F1990451178631 E04:19:03.86 USER_INPUT 0 ID=34586238477^F1990451178631 E04:19:03.86 USER_INPUT 0 ID=34586238477^F1990451178631 E04:19:03.86 USER_INPUT 0

What's the best way to do this by sed or similar tools?

2

awk allows you to use a record separator other than newline. You can use it as follows:

awk 'BEGIN { ORS=RS="\x06" } ...' input-file

Where ... is whatever you want to do with the data.

  • +1. don't forget to set ORS too. – cas May 6 '16 at 13:39
  • @cas Right. In most cases that would be necessary, too. – Michael Vehrs May 7 '16 at 6:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.