1

I have a load of files in which I need to take out specific rows and then put the data I've taken out into a spreadsheet. An example would be my file shows:

Name: w

Age: x

Height: y

Weight: z

I only want the age, height and weight, so I first ran:

grep -E 'Age|Height|Weight' [input file] > output.txt

Because of the number of files, my output now looks like

Age 1

Height 1

Weight 1

Age 2

Height 2

Weight 2

etc...

What I now want is to run an awk script so it goes through my new output.txt file and first finds every row with the word 'Age' and the prints it. Once it has done all the 'Age' ones, it then does the heigh and then weight. I ran the script:

awk -F"\t" '/Age/ {print} /Height/ {print}' output.txt >output2.txt

But if just prints it like the original out put file. How do I change it so after it has done all the Age ones, it then finds the height ones?

EDIT:

My desired output is for the file to be

Age 1

Age 2

Height 1

Height 2

Weight 1

Weight 2

etc..

Just to clarify, the Age 1 is the row with 'age' in it from file 1, etc.

  • Why not run 3 separate scripts? – LatinSuD Jun 24 '14 at 10:08
  • I want to try and do it all in one go because that was just the example. I have 12 words I'm looking for in 23 different folders, so 23 * 12 scripts will be annoying to run and I'm lazy – Kaish Jun 24 '14 at 10:11
  • 1
    Why the title is print newline? I don't see it relates to your question. – cuonglm Jun 24 '14 at 10:11
  • is it fine if we suggest solution in python? – harish.venkat Jun 24 '14 at 10:12
  • @harish.venkat I'm not sure what Python is...at my work we have unix and emacs (if that means anything). I don't even know how it works in general. I'm trying to learn as I go along as I'm new lol – Kaish Jun 24 '14 at 10:14
1

awk only runs through the file once by default, running all the blocks in order, which is why it gives you the output you got. You can get the behaviour you want using an array to save the lines as you go, while still only processing the file once:

BEGIN {
    AgeIndex = 1
    HeightIndex = 1
}
/Age/ {
    ages[AgeIndex] = $0
    AgeIndex+=1
}
/Height/ {
    heights[HeightIndex] = $0
    HeightIndex+=1
}
END {
    for (x = 1; x < AgeIndex; x++)
        print ages[x] "\n"
    for (x = 1; x < HeightIndex; x++)
        print heights[x] "\n"
}

Save that into, say, filter.awk and then run:

awk -f filter.awk output.txt > output2.txt

to get the output you want:

$ awk -f filter.awk < data
Age 1

Age 2

Height 1

Height 2

What we're doing is making two arrays ages and heights and saving each matching line into them as we go. AgeIndex holds how far through the array we're up to. At the end, we're printing out every line we saved (and an extra newline like you want), first all the ages, then all the heights.

The arrays will hold the entire file in memory by the end, so if your file is particularly large you have to trade off that memory usage for time in going through the whole file more than once. At this point it's essentially the same as a program in any other language - if you don't have any particular reason to use awk, you might prefer another language. To be honest, I think I'd recommend that - awk isn't buying you much here.

1

With gawk:

$ awk -F"\t" '
    { a[$1]++ }
    END {
        n = asorti(a,b);
        for (i = 1; i <= n; i++) {
            print b[i];
            if (i%2 == 0) {
                printf "\n";
            }
        }
    }
' output.txt
Age 1
Age 2

Height 1
Height 2

Weight 1
Weight 2
  • That didn't work, it just printed out Age and Height, the exact same way as before, but twice – Kaish Jun 24 '14 at 10:18
  • @Kaish: Can you give us your wanted output? – cuonglm Jun 24 '14 at 10:20
  • Edited my question – Kaish Jun 24 '14 at 10:23
0

I assume that the blank lines are not part of your actual file, or that at least you don't care about them. If so, all you need is sort:

$ cat output.txt
Age 1
Height 1
Weight 1
Age 2
Height 2
Weight 2

$ sort output.txt
Age 1
Age 2
Height 1
Height 2
Weight 1
Weight 2

However, unless your files are too large to hold in memory, it might be simpler to do the whole thing in a single step:

grep -whE 'Age|Height|Weight' *txt | sort > outfile

The above will search for Age or Height or Weight in all files whose name ends in txt in the current directory (*txt). The -w means "match whole words only" (so that Age does not match Ageing for example), the -h is needed because without it, the name of the file is printed along with the matching line when more than one input file is given. The -E enables extended regular expressions which gives us | for OR.

NOTE: If, for some reason, you do actually want the extra blank line between each entry (which is not what your grep command would produce), you can add it with:

grep -whE 'Age|Height|Weight' *txt | sort | sed 's/$/\n/'

Example

$ for i in {1..3}; do echo -e "Name $i\nAge $i\nHeight $i\nWeight $i" > $i.txt; done
$ for f in *txt; do echo " -- $f --"; cat $f; done
 -- 1.txt --
Name 1
Age 1
Height 1
Weight 1
 -- 2.txt --
Name 2
Age 2
Height 2
Weight 2
 -- 3.txt --
Name 3
Age 3
Height 3
Weight 3

$ grep -whE 'Age|Height|Weight' *txt | sort
Age 1
Age 2
Age 3
Height 1
Height 2
Height 3
Weight 1
Weight 2
Weight 3

In any case, even if sort won't cut it for you, I would do this kind of thing in Perl, not awk (this is assuming you want the extra blank lines which,again, you probably don't):

$ perl -ane '$k{$F[0]}.=$_."\n" if /./; 
    END{print $k{$_},"\n" for sort keys (%k)}' output.txt 
Age 1

Age 2


Height 1

Height 2


Weight 1

Weight 2


 

You can pass that through head -n -2 to get rid of the final two blank lines if you don't want them.

  • @teron: The OP want put a newline at the end of each segment. – cuonglm Jun 24 '14 at 12:16
  • @Gnouc I think they probably don't actually. The output of the grep in the OP would not produce those lines so I am assuming they don't exist in the actual file. In any case, they're easy enough to add, see update. – terdon Jun 24 '14 at 12:19
0

You can use sort:

grep -E 'Age|Height|Weight' [input file] | sort > output.txt
0

python solution for this problem:

from collections import defaultdict
f = open("output.txt", "r")
d = defaultdict(list)
for line in f:
   line = line.strip()
   if line != '':
     arr = line.split(" ")
     d[arr[0]].append(arr[1])
print d.items()

I have hashed using first column and put it in a list.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.