2

I only have little experience in programming, and I'm currently working on improving my skills. Basically, I need to write a program, that can do some specific processes to some data in a .txt file.

To start from scratch, I have a .txt file with data looking like this:

>tex_1 abcdefghijklmnopqrstu
>tex_2 abcdefghijklmnopqrstuv
>tex_3 abcdefghijklmnopqrstuv
>tex_4 abcdefghijklmnopqrst
//     x
>tex_1 abcdefghijklmnopqrstu
>tex_2 abcdefghijklmnopqrstuv
>tex_3 abcdefghijklmnopqrst
>tex_4 abcdefghijklmnopqrstuv
//         x  x
>tex_1 abcdefghijklmnopqrstuv
>tex_2 abcdefghijklmnopqrstuv
//     x   x

I need to do some weird stuff to this data, in order to end up with a data set, that can be analyzed in the software I use. Each "//..."-line refers to the group of data above, until the next "//..."-line

Here's a line-up of what I want to do:

Shift the "//..."-line, so the group of data it refers to is below this line, and not above it:

//         x
>tex_1 abcdefghijklmnopqrstu
>tex_2 abcdefghijklmnopqrstuv
>tex_3 abcdefghijklmnopqrstuv
>tex_4 abcdefghijklmnopqrst
//         x  x
>tex_1 abcdefghijklmnopqrstu
>tex_2 abcdefghijklmnopqrstuv
>tex_3 abcdefghijklmnopqrst
>tex_4 abcdefghijklmnopqrstuv
//              x   x
>tex_1 abcdefghijklmnopqrstuv
>tex_2 abcdefghijklmnopqrstuv

Add a unique name to each group after //, without shifting remaining text on the line:

//Name 1  x
>tex_1 abcdefghijklmnopqrstu
>tex_2 abcdefghijklmnopqrstuv
>tex_3 abcdefghijklmnopqrstuv
>tex_4 abcdefghijklmnopqrst
//Name 2   x  x
>tex_1 abcdefghijklmnopqrstu
>tex_2 abcdefghijklmnopqrstuv
>tex_3 abcdefghijklmnopqrst
>tex_4 abcdefghijklmnopqrstuv
//Name 3        x   x
>tex_1 abcdefghijklmnopqrstuv
>tex_2 abcdefghijklmnopqrstuv

Output this to a new file, without changing the original. Then grab each name-line + line below, and output this to a File2:

//Name 1  x
>tex_1 abcdefghijklmnopqrstu
//Name 2   x  x
>tex_1 abcdefghijklmnopqrstu
//Name 3        x   x
>tex_1 abcdefghijklmnopqrstuv

Change the structure, so the naming is like the following, and output this to File3:

>Name 1 abcdefghijklmnopqrstu
>Name 2 abcdefghijklmnopqrstu
>Name 3 abcdefghijklmnopqrstuv

The above data is in a structure, I can actually analyse.

Now I know this is guite the task, and I am not asking you "how do I program this. I would just like to know, where you guys would start with such a project, and what language do you think fits the project best?

I managed to do a few things in unix, by getting help on this site. E.g. Giving unique names to each "//..." line, by the following unix code:

awk -F '' '/\/\//{n++ ; t=" Name "n ; sub("// {0,"length(t)-1"}","//"t)}{print}' File1.txt

Could you give me some hints for where to start?
Is the problem suitable as a Python project?
The original .txt data file contains a lot of data, so it is not possible to do the processing by hand. Also this project is meant as a way to get further into programming.

3

This looks more like a python job to me. A general rule of thumb is: if your task requires only a "flat" and content-blind processing, core utilities (preferably gnu) are the way to go. This goes for string replacement, deletion, line-based processing, simple sorting, counting, filtration and so on... these tools allow you to very quickly write a one-liner to do what you want, with practice it doesn't even require much thinking.

On the other hand, if you need a complex job that needs to look around the file, imply a hierarchical (tree-like) structure, custom delimiters, and context-aware strings, then it's much easier to have the same structure available in your programming language. An extreme example are html/xml/json files, any files with nested braces and so on... if you have very powerful data structures available (python does), just write it in, process it in object-like fashion and read it back out. You can still do it in awk (or with a bit more effort in sed), but it's not worth it.

In your case, you will need to memorize position of the previous marker (or hold a long buffer) which gets a bit clumsy with line-processing software. In python, however, it's easy.

Example code:

#!/usr/bin/python
import sys

with open(sys.argv[1],'r') as file, open(sys.argv[2],'w') as file1, open(sys.argv[3],'w') as file2, open(sys.argv[4],'w') as file3:
    counter = 1
    current_buffer = []
    for line in file:
        if line.startswith('//'):
            #we found a delimiter, flush the buffer
            #could use regular expressions, but for the sake of this example
            #this is enough
            prefix = '//Name {}'.format(counter)
            new_header = prefix+line[len(prefix):]
            file1.write(new_header)
            file2.write(new_header)
            for oldline in current_buffer:
                file1.write(oldline)
            if current_buffer: #only first line to file 2
                first_line = current_buffer[0]
                file2.write(first_line)
                #same here, could use regular expressions from "import re" but we don't have to
                rest_of_line = first_line.split(' ',1)[1]
                file3.write('//Name {} {}\n'.format(counter,rest_of_line.strip()))
            current_buffer=[]
            counter+=1
        else:
            current_buffer.append(line)

    #if the file is well-formatted, current_buffer should be empty now - otherwise, do special handling

Call it as ./test.py inputfile file1 file2 file3 and see if that's what you wanted.

  • Wow, thank you for your answer, @orion! I'm having some trouble running the script, though. I get the following error message: $ ./test.py test_file.txt file1.txt file2.txt file3.txt -bash: ./test.py: Permission denied I am on a mac, if that makes a difference. I hope I'm doing it the right way. I have the file saved as test.py in the directory I am in in my terminal. Then trying to call it with the command you gave me. – Hjalte Jan 29 '15 at 18:55
  • Right, I thought you knew... to run a script as an executable command, the file needs to have executable permissions. Either run chmod +x test.py and it'll work, or just call it explicitly with python: python test.py .... Also, this was tested for python 3, python 2 may have some differences. – orion Jan 29 '15 at 21:30
  • Okay, @orion, thank you! I have run the script, and I cannot believe how well it worked in the first run! :) I guess this underlines why it is a huge advantage, to know some programming! The only trouble I'm having is, that the insertion of the name shifts the rest of the line to the right. This is a problem in my data, since the signs on the "//Name…" line refers to the column of letters just above (or beneath, after moving the name to the top) of the them. – Hjalte Jan 30 '15 at 10:20
  • Just replace line[2:] which means line, from 2th character forward, with line[7:]. – orion Jan 30 '15 at 10:43
  • 1
    @mikeserv Less code, but probably not less effort. I use awk and sed regularly, but for these kinds of jobs, legibility suffers, and python is still completely transparent what it does (but more verbose and therefore longer). I realize this is sort of a borderline case (as I said, parsing hierarchical data is way worse), but especially if a less experienced user will be using this code, having something that can be understood at a glance, and changed to adjust for needs, is much better. Of course all solutions are valid and the sed one is quite impressive, I would not have thought of using nl. – orion Jan 30 '15 at 17:17
3

You should go with whatever makes you most comfortable. That said, you shouldn't avoid learning the use of a new tool either. I am pretty comfortable with sed, and it is my belief that nl is uniquely suited to this task, and so using a combination of those two tools I did:

<<\INFILE \
nl -bp'^//' -w1 -s'      ' |\
sed -ne ' s|^ *>|>|;//H;x;//x;t' -e 'G
        x;s|.*|### The following is written to: File1|p
        x;s|\([0-9]*\)\( *\)\( \)*//\2|//Name \1\3|pw File1
        x;s|1|2|p;x;s|\n\(\n.*\)*|&&|2;s|||2p;w File2' -e 'x;s|2|3|p
        x;s|//\([^ ]* [^ ]*\).*\n[^ ]*|>\1|pw File3'
>tex_1 abcdefghijklmnopqrstu
>tex_2 abcdefghijklmnopqrstuv
>tex_3 abcdefghijklmnopqrstuv
>tex_4 abcdefghijklmnopqrst
//     x
>tex_1 abcdefghijklmnopqrstu
>tex_2 abcdefghijklmnopqrstuv
>tex_3 abcdefghijklmnopqrst
>tex_4 abcdefghijklmnopqrstuv
//         x  x
>tex_1 abcdefghijklmnopqrstuv
>tex_2 abcdefghijklmnopqrstuv
//     x   x

...which seems to do the trick.

When I say nl is uniquely suited to this task it is because, besides easily numbering blocks in order according to a pattern and injecting an arbitrary --separator string between the text on the numbered line and the line's number, nl also always indents lines it doesn't number to match the indent it adds to those it does.

This means that any line which does not get numbered as a match to the ^// pattern now begins with a <space> instead. And so even if your actual text is much more complicated than the data in your examples, the sed is actually very easy to do. In fact, much of the above work is intended only to generate meaningful debug output as sed works. A fully functional statement could be as simple as:

nl -bp'^//' -w1 -s'      ' <infile |
sed -ne 's|^ *>|>|;//H;x;//x;t' -e 'G;h
         s|\([0-9]*\)\( *\)\( \)*//\2|//Name \1\3|w File1
         s|\n|&&|2;s|\n\n.*||;w File2
         s|//\([^ ]* [^ ]*\).*\n[^ ]*|>\1|w File3'

I am a little confused about your statement without shifting remaining text on the line because it seems to contradict your examples (and the realm of possibility). I say this because:

//     x

...is obviously shifted a little to the right when it is transformed to:

//Name 1  x

...and in fact the 1 and the x occupy the same column in the two examples. I tried to meet that contradiction in the middle - it does not shift the text unless it must to avoid overwriting a non-<space> character while still inserting the Name [num] pair plus a single space.

Thus does not account for the discrepancy between...

//Name 3        x   x

...and...

//     x   x

...which I have disregarded as (hopefully?) a mistake in the example.

My pipeline prints everything to stdout, but the blocks following the lines beginning with ### are appended each in turn to the indicated files as well. Here's what my pipeline prints:

### The following is written to: File1
//Name 1 x
>tex_1 abcdefghijklmnopqrstu
>tex_2 abcdefghijklmnopqrstuv
>tex_3 abcdefghijklmnopqrstuv
>tex_4 abcdefghijklmnopqrst
### The following is written to: File2
//Name 1 x
>tex_1 abcdefghijklmnopqrstu
### The following is written to: File3
>Name 1 abcdefghijklmnopqrstu
### The following is written to: File1
//Name 2   x  x
>tex_1 abcdefghijklmnopqrstu
>tex_2 abcdefghijklmnopqrstuv
>tex_3 abcdefghijklmnopqrst
>tex_4 abcdefghijklmnopqrstuv
### The following is written to: File2
//Name 2   x  x
>tex_1 abcdefghijklmnopqrstu
### The following is written to: File3
>Name 2 abcdefghijklmnopqrstu
### The following is written to: File1
//Name 3 x   x
>tex_1 abcdefghijklmnopqrstuv
>tex_2 abcdefghijklmnopqrstuv
### The following is written to: File2
//Name 3 x   x
>tex_1 abcdefghijklmnopqrstuv
### The following is written to: File3
>Name 3 abcdefghijklmnopqrstuv
  • 1
    Good work! The idea to use nl to mark blocks and count is gorgeous! Everytime I have to refuse to use sed in case of even simple calculations but now, thanks of you, its obstacle is removed! – Costas Jan 30 '15 at 18:00
  • @Costas - it truly warms the heart to recognize the mark of another devoted sedder. – mikeserv Jan 30 '15 at 18:03
  • @Costas - come to think of it, another standard tool with which sed works very well is dc. A particularly interesting combination can be had when dc macros are programmed to read and compute line by line with the ? read-input-line operator as sed feeds it from a pipe (and vice-versa, for that matter). Add to that dc's ability to execute arbitrary system(...) commands in-stream with the ! operator, its huge, stackable arrays, [string] handling, ASCII num>char (and vice-versa) conversion... and the imaginative sedder could do some pretty powerful stuff between the two. – mikeserv Jan 30 '15 at 18:29
  • To my big regret I have never met dc in my life (just bc but it is not the same, is it?) – Costas Jan 30 '15 at 20:01
  • @Costas - bc is based on dc, and - though this changed a few years ago - bc used to be only a front-end to dc. It used to be that bc was basically just the lexer because (and perhaps rightfully so) many were daunted by dc's terse, stack-oriented syntax style. It was only fairly recently that bc came into its own as a fully separate entity in its own right, when the standard engine was generally adjudged good enough to do so. Still, I'm fairly sure that dc can outperform bc in most cases. Do man dc and have a look if you're curious. – mikeserv Jan 30 '15 at 21:20
2

I can't agree that python is more acceptable in the case. There is no difficult line manipulation - just set variables and print various set of it. So its are usual bash' operations. So we need some calculations I prefer awk instead of sed(usual text manipulation programm).

awk -F '' '
          /\/\//{
                 n++
                 t="Name "n" "
                 sub("// {0,"length(t)"}","//"t)
                 print $0 "\n" l1 l2 > "File2"
                 print $0 "\n" l1 > "File3"
                 sub("[^ ]* ","<"t,l1)
                 print l1 > "data.file" 
                 l1=""
                 l2=""
                 next
                 }
          {
                 if (l1=="")
                     l1=$0
                 else l2=l2 "\n" $0
                 }' Input.txt

But everybody prefer that tool which he knows better.

P.S. Regarding the length of input string. If OP check your examples you can see that «x» is above «e»

//         x
>tex_1 abcdefghijklmnopqrstu

but after input Name 1 «x» is above «d»

//Name 1  x
>tex_1 abcdefghijklmnopqrstu

so in script shoul be length(t) not lenght(t)-1. Additionally if there in not enough space to input name script make necessary shift string's residue to right.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.