I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted according to various columns. Without the headers, the cutting and sorting is easily accomplished via the -k
option to sort
along with cut
or awk
to view a subset of the columns. However, this method of sorting mixes the column headers in with the rest of the lines of output. Is there an easy way to keep the headers at the top?
Stealing Andy's idea and making it a function so it's easier to use:
# print the header (the first line of input)
# and then run the specified command on the body (the rest of the input)
# use it in a pipeline, e.g. ps | body grep somepattern
body() {
IFS= read -r header
printf '%s\n' "$header"
"$@"
}
Now I can do:
$ ps -o pid,comm | body sort -k2
PID COMMAND
24759 bash
31276 bash
31032 less
31177 less
31020 man
31167 man
...
$ ps -o pid,comm | body grep less
PID COMMAND
31032 less
31177 less
-
ps -C COMMAND
may be more appropriate thangrep COMMAND
, but it's just an example. Also, you can't use-C
if you also used another selection option such as-U
. – Mikel Apr 23 '11 at 0:51 -
Or maybe it should be called
body
? As inbody sort
orbody grep
. Thoughts? – Mikel Apr 23 '11 at 0:57 -
4Renamed from
header
tobody
, because you're doing the action on the body. Hopefully that makes more sense. – Mikel Apr 23 '11 at 1:02 -
3Remember to call
body
on all subsequent pipeline participants:ps -o pid,comm | body grep less | body sort -k1nr
– bishop Nov 7 '16 at 20:02 -
2@Tim You can just write
<foo body sort -k2
orbody sort -k2 <foo
. Just one extra character from what you wanted. – Mikel Sep 4 '17 at 13:49
You can keep the header at the top like this with bash:
command | (read -r; printf "%s\n" "$REPLY"; sort)
Or do it with perl:
command | perl -e 'print scalar (<>); print sort { ... } <>'
-
2
-
2+1, any reason why a subshell is preferable, or is
{}
ok instead of()
? – jonderry Apr 23 '11 at 0:57 -
3
IFS=
disables word splitting when reading the input. I don't think it's necessary when reading to$REPLY
.echo
will expand backslash escapes ifxpg_echo
is set (not the default);printf
is safer in that case.echo $REPLY
without quotes will condense whitespace; I thinkecho "$REPLY"
should be okay.read -r
is needed if the input may contain backslash escapes. Some of this might depend on bash version. – Andy Apr 23 '11 at 1:50 -
1@Andy: Wow, you're right, different rules for
read REPLY; echo $REPLY
(strips leading spaces) andread; echo $REPLY
(doesn't). – Mikel Apr 23 '11 at 2:44 -
1@Andy: IIRC, the default value of
xpg_echo
depends on your system, e.g. on Solaris I think it defaults to true. This is why Gilles likesprintf
so much: it's the only thing with predictable behavior. – Mikel Apr 23 '11 at 2:47
I found a nice awk version that works nicely in scripts:
awk 'NR == 1; NR > 1 {print $0 | "sort -n"}'
-
2I like this, but it requires a bit of explanation - the pipe is inside the awk script. How does that work? Is it calling the
sort
command externally? Does anyone know of at least a link to a page explaining pipe use within awk? – Wildcard Nov 7 '15 at 1:24 -
-
Hackish but effective: prepend 0
to all header lines and 1
to all other lines before sorting. Strip the prefix after sorting.
… |
awk '{print (NR <= 2 ? "0 " : "1 ") $0}' |
sort -k 1 -k… |
cut -b 3-
Here's some magic perl line noise that you can pipe your output through to sort everything but keep the first line at the top: perl -e 'print scalar <>, sort <>;'
I tried the command | {head -1; sort; }
solution and can confirm that it really screws things up--head
reads in multiple lines from the pipe, then outputs just the first one. So the rest of the output, that head
did not read, is passed to sort
--NOT the rest of the output starting from line 2!
The result is that you are missing lines (and one partial line!) that were in the beginning of your command output (except you still have the first line) - a fact that is easy to confirm by adding a pipe to wc
at the end of the above pipeline - but that is extraordinarily difficult to trace down if you don't know this! I spent at least 20 minutes trying to work out why I had a partial line (first 100 bytes or so cut off) in my output before solving it.
What I ended up doing, which worked beautifully and didn't require running the command twice, was:
myfile=$(mktemp)
whatever command you want to run > $myfile
head -1 $myfile
sed 1d $myfile | sort
rm $myfile
If you need to put the output into a file, you can modify this to:
myfile=$(mktemp)
whatever command you want to run > $myfile
head -1 $myfile > outputfile
sed 1d $myfile | sort >> outputfile
rm $myfile
-
You can use ksh93's
head
builtin or theline
utility (on systems that still have one) orgnu-sed -u q
orIFS=read -r line; printf '%s\n' "$line"
, that read the input one byte at a time to avoid that. – Stéphane Chazelas Jan 11 '18 at 21:58
I think this is easiest.
ps -ef | ( head -n 1 ; sort )
or this which is possibly faster as it does not create a sub shell
ps -ef | { head -n 1 ; sort ; }
Other cool uses
shuffle lines after header row
cat file.txt | ( head -n 1 ; shuf )
reverse lines after header row
cat file.txt | ( head -n 1 ; tac )
-
2See unix.stackexchange.com/questions/11856/…. This is not actually a good solution. – Wildcard Nov 6 '15 at 21:43
-
1Not working,
cat file | { head -n 1 ; sort ; } > file2
only show head – Peter Krauss Jul 6 '18 at 19:19
command | head -1; command | tail -n +2 | sort
-
4This starts
command
two times. Therefore it is limited to some specific commands. However, for the requestedps
command in the example, it would work. – jofel May 20 '14 at 12:00
Simple and straightforward!
<command> | head -n 1; <command> | sed 1d | sort <....>
- sed nd ---> 'n' specifies line no., and 'd' stands for delete.
-
1Just as jofel commented a year and a half ago on Sarva's answer, this starts
command
twice. So not really suitable for use in a pipeline. – Wildcard Nov 6 '15 at 2:36
I came here looking for a solution for the command w
. This command shows details of who is logged in and what they are doing.
To show the results sorted, but with the headers kept at the top (there are 2 lines of headers), I settled on:
w | head -n 2; w | tail -n +3 | sort
Obviously this runs the command w
twice and therefore may not be suitable for all situations. However, to its advantage it is substantially easier to remember.
Note that the tail -n +3
means 'show all lines from the 3rd onwards' (see man tail
for details).
Try doing:
wc -l file_name | tail -n $(awk '{print $1-1}') file_name | sort
-
4
{ head -1; sort; }
to work. It always deletes a bunch of the text after the first line. Does anyone know why this happens? – jonderry Apr 23 '11 at 1:02head
is reading more than one line into a buffer and throwing most of it away. Mysed
idea had the same problem. – Andy Apr 23 '11 at 1:09lseek
able input so it won't work when reading from a pipe. It will work if you redirect to a file>outfile
and then run{ head -n 1; sort; } <outfile
– don_crissti Sep 26 '15 at 13:40