Awk - output the second line of a number of .dat files to one file

Question

I have multiple files something like: (in reality i have 80)

file1.dat

2 5

6 9

7 1

file2.dat

3 7

8 4

1 3

I want to end up with a file containing all of the second lines. i.e.

output.dat

6 9

8 4

What i have so far loops though the file names but then over-writes the file before it. e.g. the output of the above files would just be

8 4

my shell script looks like this:

post.sh

TEND = 80

TINDX = 0

while [ $TINDX - lt $TEND]; do

awk '{ print NR==2 "input-$TINDX.dat > output.dat

TINDX = $((TINDX+1))

done

Answer 1

Remove while loop and make use of shell brace expansion and also FNR, a built-in awk variable:

awk 'FNR==2{print $0 > "output.dat"}' file{1..80}.dat

Answer 2

sed would be enough:

sed -sn 2p file{1..80}.dat > output.dat

-s option is needed to print 2nd line from each file, otherwise only 2nd line of first file will be printed.

Answer 3

What about ... head -n 2 input.dat | tail -n 1 | awk ...

Answer 4

aragaer’s sed solution is nicest, yes. But I since I do enjoy a bit of head|tail cutting, have a head|tail solution that supports multiple files, not just a single input.dat. Using a for-loop, instead of passing a list of files to sed, also makes it easier to do other thing with the file before/after extracting the second line with sed.

# empty output.dat first
rm output.dat

# have a one-liner
for file in *.dat; do head -2 $file | tail -1 >> output.dat; done 

Copiously-commented multi-line version:

NB: the code below will run. We are free to put a linebreak after a |, &&, or ||, and continue our command on the next line; we can even put comments in between. I spent years not knowing this (and not really seeing it anywhere). This style is less useful at the interactive prompt, but cleans up script files no end.

# empty output.dat first
rm output.dat

for file in *.dat; do
    # file -> lines 1 and 2 of file
    head -2 $file |
    # lines 1 and 2 of file -> line 2 of file >> appended to output.dat
    tail -1 >> output.dat
done

Answer 5

There are obviously lots of ways to do this - I think I like @aragaer's sed answer best.

Here's one that uses purely bash builtins and doesn't need to fork any external utilities:

for f in file{1..80}.dat; do
    { read && read && printf "%s\n" "$REPLY"; } < "$f"
done > output.dat

Answer 6

For efficiency use of awk and sed in answers here on multiple files, better to use nextfile statement to skip processing unwanted lines in awk.

awk 'FNR==2{ print >"output.dat"; nextfile}' infile{1..80}.dat

and with sed, we can exit when processing on 3^rd line and sed will process the next file.

sed -sn '2p;3q' infile{1..80}.dat > output.dat