9

I use awk a fair bit for parsing logs; I have never seen anything like this: I have six file containing a number of lines; I want the ones containing "100", and to choose which columns to print

me:~/tmp> grep 100 *.dl.tst

outputs what I expect:

100  139M  100  139M    0     0  6376k      0  0:00:22  0:00:22 --:--:-- 6539k
100  139M  100  139M    0     0  6677k      0  0:00:21  0:00:21 --:--:-- 6579k
100  139M  100  139M    0     0  6022k      0  0:00:23  0:00:23 --:--:-- 6093k
100  139M  100  139M    0     0  13.9M      0  0:00:10  0:00:10 --:--:-- 14.3M
100  139M  100  139M    0     0  14.3M      0  0:00:09  0:00:09 --:--:-- 14.7M
100  139M  100  139M    0     0  13.2M      0  0:00:10  0:00:10 --:--:-- 13.3M

as does:

me:~/tmp> grep 100 *.dl.tst|awk '{print$0}'
100  139M  100  139M    0     0  6376k      0  0:00:22  0:00:22 --:--:-- 6539k
100  139M  100  139M    0     0  6677k      0  0:00:21  0:00:21 --:--:-- 6579k
100  139M  100  139M    0     0  6022k      0  0:00:23  0:00:23 --:--:-- 6093k
100  139M  100  139M    0     0  13.9M      0  0:00:10  0:00:10 --:--:-- 14.3M
100  139M  100  139M    0     0  14.3M      0  0:00:09  0:00:09 --:--:-- 14.7M
100  139M  100  139M    0     0  13.2M      0  0:00:10  0:00:10 --:--:-- 13.3M

Why then does $1 become the file name:

me:~/tmp> grep 100 *.dl.tst|awk '{print$1}'
shpr002.20201124_141036.dl.tst:
shpr003.20201124_141036.dl.tst:
shpr004.20201124_141036.dl.tst:
hipr002.20201124_141036.dl.tst:
hipr003.20201124_141036.dl.tst:
hipr004.20201124_141036.dl.tst:

And $2:

me:~/tmp> grep 100 *.dl.tst|awk '{print$2}'
0
0
0
0
0
0

I logged out and back in in case my shell (bash) was screwed up; no change... what am I doing wrong?

Output from grep 100 *.dl.tst | awk '{print$1}' | head -n1 | od -c (some of the alpha characters have been substituted by x; the list above had been edited/obfuscated)

0000000   x  s   h   p   r   0   0   2   x   x   x  .   x   x   x   .
0000020   x   x   x   x   .   c   o   m   .   2   0   2   0   -   1   1
0000040   -   2   4   _   1   4   1   0   3   6   .   d   l   .   t   s
0000060   t   :  \r  \n
0000064
Improve this question
  • 5
    Please edit your question and add the output of grep 100 *.dl.tst | awk '{print$1}' | head -n1 | od -c. The file name is normal, that's grep adding the file name to the output, but it should also be there when you print $0. I'm thinking you might have some non-printing characters which is why I asked for that output. – terdon♦ Nov 24 '20 at 16:39
  • 1
    Linux. Output from "grep 100 *.dl.tst | awk '{print$1}' | head -n1 | od -c" (some of the alpha characters have been substituted by "x"; the list above had been edited/obfuscated) ``` 0000000 x s h p r 0 0 2 x x x . x x x . 0000020 x x x x . c o m . 2 0 2 0 - 1 1 0000040 - 2 4 _ 1 4 1 0 3 6 . d l . t s 0000060 t : \r \n 0000064 ``` – Iain Nov 24 '20 at 16:53
  • 2
    I couldn't resist: You're saying that awk doesn't know the value of $1 – Ross Presser Nov 25 '20 at 2:10
  • 1
    @Iain, I think he meant that just as a pun. Awk not knowing the value of one dollar. – ilkkachu Nov 25 '20 at 14:41
  • 2
    @Iain Sorry if my humor was too obscure .. there's an old catchphrase "know the value of a dollar" (as ikkachu mentioned). It's often a criticism from a parent to a spendthrift child :) – Ross Presser Nov 25 '20 at 18:35
16

Those files contain the output from curl downloading files, and curl updates its progress information during downloads by outputting a carriage return (commonly represented as \r, the escape used to produce it in a number of contexts), which causes the cursor to return to the start of the line.

When you run grep 100 *.dl.tst, each line that’s output starts with the file name, but that’s followed by multiple updates which return the cursor to the start of the line, so you don’t see the file name — it’s overwritten by subsequent output. In more detail, the output looks like

shpr002.20201124_141036.dl.tst:

followed by a carriage return, followed by the first progress output from curl,

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

followed by a carriage return, etc., until the percentage reaches 100. Because all this is only separated by carriage returns, not line feeds, it counts as a single line, and grep matches that in its entirety.

The same effect explains the output of grep 100 *.dl.tst|awk '{print$0}'.

When you ask AWK to output $1, it outputs the first field, and now you can see it: it contains the file name, a colon, a carriage return, and that’s it — the start of curl’s output then starts with a space (to leave room for the percentage count), which is a field separator. When you ask it to output $2, it outputs the second field, which is the first percentage count, 0:

shpr002.20201124_141036.dl.tst:\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

<--          Field 1          -->  !     !    !     !  ...
                                   $2    $3   $4    $5 ...
Improve this answer
  • They are indeed output from curl, but if I cat one, all I get is the headers and final, 100%, output: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 139M 100 139M 0 0 6022k 0 0:00:23 0:00:23 --:--:-- 6093k Where's the rest of it? – Iain Nov 24 '20 at 16:59
  • 1
    It’s overwritten, every time the cursor returns to the start of the line. Open the files with less to see what’s going on; it will show the carriage returns as ^M. – Stephen Kitt Nov 24 '20 at 17:00
  • 1
    In the absence of a fix (as opposed to a diagnosis: awk '{ sub ("^.*\r", ""); print; }' – Paul_Pedant Nov 24 '20 at 19:13
  • 1
    "_Where's the rest of it?" You could pass a log-file through tr '\r' '\n' to "see the rest of it". – TripeHound Nov 25 '20 at 16:30
4

Working on Stephen's description of the issue, a simple way to make the output easier to process would be just translate all the carriage returns into newlines, leaving you with curl's progress report as a bunch of individual lines, which you can then use awk on:

$ for f in *.dl; do < "$f" tr '\r' '\n' | awk '$1 == "100" {print $0}' ; done
100  720k  100  720k    0     0  22.5M      0 --:--:-- --:--:-- --:--:-- 22.7M
100 23.6M  100 23.6M    0     0   372M      0 --:--:-- --:--:-- --:--:--  369M

(Though, if curl rounds the percentage it prints to nearest integer instead of down, huge files might show multiple lines with 100 in the first column.)

On the other hand, if it's known that the files contain nothing but the output from curl, then we might as well just pick the last line instead of looking at the contents:

$ for f in *.dl; do < "$f" tr '\r' '\n' | tail -n1  ; done
100  720k  100  720k    0     0  22.5M      0 --:--:-- --:--:-- --:--:-- 22.7M
100 23.6M  100 23.6M    0     0   372M      0 --:--:-- --:--:-- --:--:--  369M
Improve this answer
  • Is there any advantage to using sed -n '$p' over tail -n 1?  (tail is probably faster if the input is a file; maybe not if it’s a pipe.) – Scott Nov 25 '20 at 23:19
  • 1
    @Scott, no, other than the fact that I managed to completely forget about tail when writing that. Thanks for the reminder. Yes, an optimized implementation of tail will definitely be much faster on a (large) file, and even from a pipe, tail was 2-3 times faster than sed on my Debian. Given a large enough input that the difference is visible, that is. – ilkkachu Nov 26 '20 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.