I want to manipulate a text-file. It contains many blocks of the form

   ...some stuff ...

I want to duplicate each of these blocks and add and remove a line of it's content. Can I script that?

4 Answers 4


Well, you can script anything.

However, if I were doing this, I would use vim macros. (If you are not familiar with vim already, you should learn it, but otherwise ignore this)

A few basic pointers:

q -- start recording a macro

qx -- stop recording a macro and name it 'x'

@x -- run the macro named 'x'

A macro will immediately re-run whatever commands you typed during the recording session.

So if you wanted a macro to duplicate a block like that type:

q          start macro
/CLASS     jump to next "CLASS"
V          start selecting full lines
/END       jump to next "END
Y          yank selected lines (yank = copy)
j          scroll down one line
p          put (paste)
qd         stop recording and name the macro d

There are plenty of good resources and examples out there for vim usages and vim macros.

For what it's worth, sed is a sort of a command line version of vim, so you might be able to adapt what I said to sed, but I'm less familiar with it, so I can't vouch for it.

~~ edit ~~

Another thing that makes this more useful is vim's ability to open up multiple files. Multiple files opened are referred to as buffers. You activate this by having multiple filenames after vim (IE vim *.txt to open all txt files)

To run a macro on multiple buffers, first open up all the files at the same time. Then, use :bufdo normal @x to run the macro x on each buffer.

Note, you must have :w as the last step of the macro to avoid getting stuck when it tries to switch to the next buffer if the current one isn't saved.

:bn will take manually to the next buffer.

Also, see https://stackoverflow.com/questions/1291962/replay-a-vim-macro-until-end-of-buffer for how to run a macro for each line. Mix and match :)


The natural tools for this are awk and Perl (assuming you want to script: for a once-off, the natural tool is an interactive editor). Here's an awk script that duplicates all CLASSEND blocks (no balancing supported: each CLASS matches the next END), except that foo lines are omitted from the second copy.

awk '
  /^CLASS$/ { store = 1; }  # start storing
  store && ! /^foo$/ { hold = hold ORS $0; }  # if storing, maybe save line
  /^END$/ {
      $0 = $0 hold;  # append hold copy to current line
      store = 0; hold = "";  # end of block
  1 { print; }  # print original line, with hold prepended if at end of block

Here's a sed solution; don't take it too seriously. Don't expect it to behave if the CLASS/END lines are not in strict alternation.

sed -e '/^CLASS$/,/^END$/!b' \
    -e '/^CLASS$/{' -e 'h' -e 'b' -e '}' \
    -e '/^foo$/!H' \
    -e '/^END$/G'

The exact tool to use will depend on the specifics of the job. If the job is only going to happen once, you might consider using a macro in vim or emacs. If you truly want to script it, I would look into sed:

Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).

Again, the exact way to do this is will depend on your situation, but to add a line after a line starting with PATTERN you can do this:

sed -e 's/\(^PATTERN.*\)/\1\nTextToAdd/' file.old > file.new

To add a line before a line starting with PATTERN you could do something like this:

sed -e 's/\(^PATTERN.*\)/TextToAdd\n\1/' file.old > file.new

To remove a line starting with PATTERN

sed -e 's/\(^PATTERN.*\)//' file.old > file.new

The trick will be to find the correct PATTERN that will catch only the lines you want.

You can combine these by doing soemthing like this:

sed -e 's/\(^PATTERN1.*\)/\1\nTextToAdd/' -e 's/\(^PATTERN2.*\)/TextToAdd\n\1/' file.old > file.new
  • To find only blocks beetwen CLASS and END we could use sed -n -e '/^CLASS$/,/^END$/p' file.txt, but as sed is stream editor we are getting blocks in parts - each word is another part so we should remeber everything after CLASS and before END and then paste it to file. I think that tool better than sed in this case will be awk, but as I don't know it I spent one and half hour trying to write some useful awk script and I failed... ;)
    – pbm
    Nov 16, 2010 at 18:55
  • Completely right, I didn't read this question closely enough before firing off an answer.
    – Steven D
    Nov 16, 2010 at 20:50

And Python solution from me:

file = open('a.txt', 'r')
arr = []
tmp = []
remember = False

for line in file:
    if "CLASS" in line:
        remember = True     # start storing lines
        tmp.append(line)    # we're storing lines in list
        continue            # go to next line
    if "END" in line:
        tmp.append(line)    # store line
        remember = False    # stop storing lines
        arr.append(tmp)     # add line to list
        tmp = []            # clear temp list
        continue            # go to next line
    if remember:
        tmp.append(line)    # add to list line between CLASS and END

file = open('a.txt', 'a')
for tmp in arr:
    if '   ...some stuff ...1\n' in tmp:    # we can skip 
        continue                            # whole block if we find this
    for i in tmp:
        if "333" in i:  # we can skip only one line 

I think that it is easiest to understand and modify than awk version (if you don't know awk)... Btw awk is next language on my list of languages that I want to learn... ;)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .