Trying to perform two operations on a single huge source file to gain performance

Question

I am performing two operations in a single Linux command. The operations are:

1. Sending data from source file into a new target file.

2. Counting the number of records in the source file.

For example: source file: a.txt, target file: b.txt

cat a.txt > b.txt; cat a.txt | wc -l

In the above example, I extract data from a.txt twice to perform the two operations. However, my source file is very huge. So for better performance, I am trying to find a way to extract data from source file just once and perform both the operations.

How can I accomplish this?

Answer 1

tee is the command you're looking for:

cat a.txt | tee b.txt | wc -l

Also, as suggested in the comments, if you want something shorter you can avoid using cat entirely and redirect a.txt straight into tee's stdin:

tee b.txt < a.txt | wc -l

From the man page:

tee - read from standard input and write to standard output and files

Answer 2

To get both actions in one file read you can do:

awk '++c;END{print c >"/dev/stderr"}' <a.txt >b.txt 

Understand that the count will come from stderr.