0

We have the following file where a bunch of records (million records) of interest are between the start and end of patterns. The start pattern always start with an A and the end pattern always starts with a Z.

Apattern1   somethinghere    #start of pattern always starts with A
Line-of-data-here-aaa
Line-of-data-here-xxxxx
Zpattern1  #end of pattern always starts with Z
ApatternX   somethinghere    #Repeat: start of pattern always starts with A
Line-of-data-here-bbbb
Line-of-data-here-yyyy
Line-of-data-here-nnnnn
ZpatternX

We Want to transform the data by appending the start pattern (entire line) to the lines within the pattern. We want to remove the end pattern. We want to use sed and/or awk to append the lines that start the pattern (starts with A) to the lines within the pattern (starts with L) and get rid of the end of the pattern (always starts with Z)So the data file would look like this:

Apattern1   somethinghere  Line-of-data-here-aaa
Apattern1   somethinghere   Line-of-data-here-xxxxx
ApatternX   somethinghere   Line-of-data-here-bbbb
ApatternX   somethinghere  Line-of-data-here-yyyy
ApatternX   somethinghere  Line-of-data-here-nnnnn
1

As I understand your question you are asking for this:

sed -E '/^A/h;/^[AZ]/d;G;s/(.*)\n(.*)/\2 \1/' yourfile

So lines starting with A are copied to the hold space (/^A/h), then lines starting with either A or Z are deleted as we don't want to print them (/^[AZ]/d). For all other lines the hold space with the A line gets appended (G), both parts get exchanged separated by a whitespace instead of a newline (s/(.*)\n(.*)/\2 \1/)

(A little easier with an sed expansion I did for myself: \h in the replacement gets expanded to the contents of the hold buffer:

sed -E '/^A/h;/^[AZ]/d;G;s/^/\h /' yourfile

If I meet more questions like this, I'll do a merge request for it.)

  • I think you got it. Thank you ...Thank you....Thank you.....I really appreciate it. Can I ask you what would you recommend as a study material for sed and or awk. – N.E Aug 1 '17 at 14:29
  • 1
    I've seen some, but they are huge and I'm not sure whether this is a good idea to spend hours on theoretically learning such a language. Long ago, I started using sed with some basics like addressing, s and d command and not much more. Each time I met a more complex task I looked in the man page what could help me do it, so I learned something new each time. You might give this tutorial a try, but I recommend not reading all. Better learn by doing. And consider setting a mark that the question has been answered. – Philippos Aug 1 '17 at 14:41
  • The test data worked but the actual data did not work. I wonder what am I missing Actual file: P36215ARK L12929010235171953162F 09570 752NN03800005002 1DE N 201705 T36215ARK0 P36215BYU L13042310235630823162V2 08723 721NN 1 1AR N 201705 T36215BYU I ran this: sed -E '/^P/h;/^[PT]/d;G;s/(.*)\n(.*)/\2 \1/' filename and got this (not appending the lines starting with P) L12929010235171953162F 09570 752NN03800005002 1DE N 201705 L13042310235630823162V2 08723 721NN 1 1AR N 201705 – N.E Aug 1 '17 at 15:15
  • So when the pattern is changed from A to P and T to Z it did not work. – N.E Aug 1 '17 at 15:35
  • Please place that actual file in the question (edit question and paste with 4 spaces in front of each line), so it becomes readable – Philippos Aug 1 '17 at 15:36
0
sed -e '
   /^A/,/^Z/!d
   //{h;d;}
   G;s/\(.*\)\n\(.*\)/\2\t\1/
' yourfile

Results:

Apattern1   somethinghere       Line-of-data-here-aaa
Apattern1   somethinghere       Line-of-data-here-xxxxx
ApatternX   somethinghere       Line-of-data-here-bbbb
ApatternX   somethinghere       Line-of-data-here-yyyy
ApatternX   somethinghere       Line-of-data-here-nnnnn

Explanation

  • Select the right range by rejecting the incorrect range: '/^A/,/^Z/!d'
  • Store the range boundaries in te hold space.
  • Range internals, append hold into current line and flip + change \n to tab.
  • Hi, Actual data is here and the first column is being chopped off. P36215ARK L12929010235171953162F 09570 752NN03800005002 1DE N 201705 T36215ARK0 P36215BYU L13042310235630823162V2 08723 721NN 1 1AR N 201705 T36215BYU – N.E Aug 1 '17 at 16:58
  • I am sorry but how do I paste the data so the lines are not merged. – N.E Aug 1 '17 at 17:00
  • @N.E Why don't you upload your data file on the pastebin.com website ? – user218374 Aug 1 '17 at 17:13
  • @N.E So what you are saying is that insead of the A and Z as the markers of the beginning/end of data ranges you have P and T. Is that correct? Why didn't you provide the original data to begin with. It would been so much better for everybody concerned. – user218374 Aug 1 '17 at 17:19
  • I was trying to simplify it. I did not expect that there would be a big difference for switching from A to P ....(is it?) – N.E Aug 1 '17 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.