0

We have the following file where a bunch of records (million records) of interest are between the start and end of patterns. The start pattern always start with an A and the end pattern always starts with a Z.

Apattern1   somethinghere    #start of pattern always starts with A
Line-of-data-here-aaa
Line-of-data-here-xxxxx
Zpattern1  #end of pattern always starts with Z
ApatternX   somethinghere    #Repeat: start of pattern always starts with A
Line-of-data-here-bbbb
Line-of-data-here-yyyy
Line-of-data-here-nnnnn
ZpatternX

We Want to transform the data by appending the start pattern (entire line) to the lines within the pattern. We want to remove the end pattern. We want to use sed and/or awk to append the lines that start the pattern (starts with A) to the lines within the pattern (starts with L) and get rid of the end of the pattern (always starts with Z)So the data file would look like this:

Apattern1   somethinghere  Line-of-data-here-aaa
Apattern1   somethinghere   Line-of-data-here-xxxxx
ApatternX   somethinghere   Line-of-data-here-bbbb
ApatternX   somethinghere  Line-of-data-here-yyyy
ApatternX   somethinghere  Line-of-data-here-nnnnn

2 Answers 2

1

As I understand your question you are asking for this:

sed -E '/^A/h;/^[AZ]/d;G;s/(.*)\n(.*)/\2 \1/' yourfile

So lines starting with A are copied to the hold space (/^A/h), then lines starting with either A or Z are deleted as we don't want to print them (/^[AZ]/d). For all other lines the hold space with the A line gets appended (G), both parts get exchanged separated by a whitespace instead of a newline (s/(.*)\n(.*)/\2 \1/)

(A little easier with an sed expansion I did for myself: \h in the replacement gets expanded to the contents of the hold buffer:

sed -E '/^A/h;/^[AZ]/d;G;s/^/\h /' yourfile

If I meet more questions like this, I'll do a merge request for it.)

9
  • I think you got it. Thank you ...Thank you....Thank you.....I really appreciate it. Can I ask you what would you recommend as a study material for sed and or awk.
    – N.E
    Aug 1, 2017 at 14:29
  • 1
    I've seen some, but they are huge and I'm not sure whether this is a good idea to spend hours on theoretically learning such a language. Long ago, I started using sed with some basics like addressing, s and d command and not much more. Each time I met a more complex task I looked in the man page what could help me do it, so I learned something new each time. You might give this tutorial a try, but I recommend not reading all. Better learn by doing. And consider setting a mark that the question has been answered.
    – Philippos
    Aug 1, 2017 at 14:41
  • The test data worked but the actual data did not work. I wonder what am I missing Actual file: P36215ARK L12929010235171953162F 09570 752NN03800005002 1DE N 201705 T36215ARK0 P36215BYU L13042310235630823162V2 08723 721NN 1 1AR N 201705 T36215BYU I ran this: sed -E '/^P/h;/^[PT]/d;G;s/(.*)\n(.*)/\2 \1/' filename and got this (not appending the lines starting with P) L12929010235171953162F 09570 752NN03800005002 1DE N 201705 L13042310235630823162V2 08723 721NN 1 1AR N 201705
    – N.E
    Aug 1, 2017 at 15:15
  • So when the pattern is changed from A to P and T to Z it did not work.
    – N.E
    Aug 1, 2017 at 15:35
  • Please place that actual file in the question (edit question and paste with 4 spaces in front of each line), so it becomes readable
    – Philippos
    Aug 1, 2017 at 15:36
0
sed -e '
   /^A/,/^Z/!d
   //{h;d;}
   G;s/\(.*\)\n\(.*\)/\2\t\1/
' yourfile

Results:

Apattern1   somethinghere       Line-of-data-here-aaa
Apattern1   somethinghere       Line-of-data-here-xxxxx
ApatternX   somethinghere       Line-of-data-here-bbbb
ApatternX   somethinghere       Line-of-data-here-yyyy
ApatternX   somethinghere       Line-of-data-here-nnnnn

Explanation

  • Select the right range by rejecting the incorrect range: '/^A/,/^Z/!d'
  • Store the range boundaries in te hold space.
  • Range internals, append hold into current line and flip + change \n to tab.
8
  • Hi, Actual data is here and the first column is being chopped off. P36215ARK L12929010235171953162F 09570 752NN03800005002 1DE N 201705 T36215ARK0 P36215BYU L13042310235630823162V2 08723 721NN 1 1AR N 201705 T36215BYU
    – N.E
    Aug 1, 2017 at 16:58
  • I am sorry but how do I paste the data so the lines are not merged.
    – N.E
    Aug 1, 2017 at 17:00
  • @N.E Why don't you upload your data file on the pastebin.com website ?
    – user218374
    Aug 1, 2017 at 17:13
  • @N.E So what you are saying is that insead of the A and Z as the markers of the beginning/end of data ranges you have P and T. Is that correct? Why didn't you provide the original data to begin with. It would been so much better for everybody concerned.
    – user218374
    Aug 1, 2017 at 17:19
  • I was trying to simplify it. I did not expect that there would be a big difference for switching from A to P ....(is it?)
    – N.E
    Aug 1, 2017 at 17:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .