2

I'm using wc -l to count the lines in output of a command, as the input is piped to it.

commad | wc -l

This works fine, but if command is doing some heavy computation, this is quite slow. Is there an alternative which displays the count of lines which have been "piped in so far"?

Such a thing would especially be useful when I'm doing a sort of per-item computation, like

cat something | xargs -L1 heavy-per-line-computation | wc -l

One way I could do this manually is to pipe the output to a file (command > file) and periodically do cat file | wc -l on it. But a single command (which doesn't redirect to files, to avoid wasteful I/O), is what I'm after.

6
  • can you know how many lines there are in total beforehand? – gogoud Feb 15 '20 at 12:50
  • 1
    What is the purpose of getting the count of lines which have been piped in so far?. Is it to track the progress of the operation? – Paulo Tomé Feb 15 '20 at 12:56
  • 1
    @gogoud, nope. That'd also defeat the purpose of using wc -l – Peeyush Kushwaha Feb 15 '20 at 13:09
  • @PauloTomé not necessarily. I know a tool (gnu parallel) which is capable of tracking progress. But if the computation itself is producing multiple lines which I want to count, then it's useless and I have to resort to wc -l – Peeyush Kushwaha Feb 15 '20 at 13:10
  • The cat can probably go, just use the xargs and pipe to wc. also you might have a buffering issue? – Jetchisel Feb 15 '20 at 13:23
6
awk '{print NR}'

This command prints a new number for each line encountered. If the final line is complete then the last number will agree with what wc -l would say. If the final line is incomplete then awk may count it (in my Kubuntu GNU awk does) but wc -l would not (because it really counts newlines); so there may be discrepancy.

Another discrepancy is if the input is completely empty: wc -l will print 0, our awk will print nothing. To make it print 0 use this variant:

awk '{print NR} END {if (NR==0) print NR}'

Or maybe you want each new number overwrite the old one in the same line of your console. Then this:

awk '{printf "\r%s",NR} END {print "\r"NR}'

Example: yes | head -n 76543 | awk '{printf "\r%s",NR} END {print "\r"NR}'

Note the command consumes its input (tee may be handy). For monitoring purpose you may be interested in:

awk '{print NR OFS $0}'

which (with the default OFS being space) is almost like cat -n (if your cat supports -n).


pv -l counts lines and it can be used inside a pipeline. Example:

for i in 1 2 3 4 5; do date; sleep 1; done | pv -l | wc -l

Consider pv -lb for quite minimal output.

2
  • 1
    This is exactly what I was looking for, thanks! pv seems to be purpose-built for exactly this. – Peeyush Kushwaha Feb 15 '20 at 18:41
  • Linux Mint seems to lack pv – Paul_Pedant Feb 15 '20 at 20:24
2

Here's a solution using Ruby.

count_lines prints (via stderr) at most every half a second the count of lines received so far, and the total count at the end (via stdout).

read -d '' make_lines <<'EOF'
  STDOUT.sync = true
  [0.2, 0.1, 0.5, 0.1, 0.6, 0.1, 0.3, 0.1, 0.3, 0.01, 0.01].each do |t|
    puts
    sleep t
  end
EOF

read -d '' count_lines <<'EOF'
  lines = 0
  t = 0
  while gets do
    lines += 1
    now = Time.now.to_f
    if now - t > 0.5
      warn lines
      t = now
    end
  end
  puts lines
EOF

ruby -e "$make_lines" | ruby -e "$count_lines"
0
1

I think you are looking for pv (Pipe view):

seq 100000000000 | pv -l | wc -l
0

Most terminals take CR (\r) to be "Move to column 1". This allows you to keep over-writing the previous output.

This is a script dwc that will read from stdin (which could be a tail -f from a file, or from an output process substitution). There is a sample test-bed in the script.

It takes a -l option. The default is to print the input line number to the current row in the terminal, as six numeric digits. With -l, it also prints the first 60 chars of the last line received, clearing any previous text. When it sees EOF, it emits newline so the prompt appears under the last output.

#! /bin/bash

#.. dwc [-l]

AWK_SHORT='
BEGIN { Fmt = "\r%6d "; }
{ printf (Fmt, NR); }
END { printf ("\n"); }
'

AWK_LONG='
BEGIN { Fmt = "\r%6d  %.60s"; Clr = sprintf ("%60s", ""); }
{ printf (Fmt, NR, $0 Clr); }
END { printf ("\n"); }
'
    if [[ "${1}" = "-l" ]]; then
        awk "${AWK_LONG}"
    else
        awk "${AWK_SHORT}"
    fi

    exit

    #.. Test method.
    man ls | head -n 40 |
         while IFS='' read X; do
             printf '%s\n' "${X}"; sleep 0.75
         done |
         tee >( ./dwc -l ) > foo.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.