77

I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted according to various columns. Without the headers, the cutting and sorting is easily accomplished via the -k option to sort along with cut or awk to view a subset of the columns. However, this method of sorting mixes the column headers in with the rest of the lines of output. Is there an easy way to keep the headers at the top?

Improve this question
4
  • 1
    I came across the following link. However, I can't get this technique of { head -1; sort; } to work. It always deletes a bunch of the text after the first line. Does anyone know why this happens? – jonderry Apr 23 '11 at 1:02
  • 2
    I suspect it's because head is reading more than one line into a buffer and throwing most of it away. My sed idea had the same problem. – Andy Apr 23 '11 at 1:09
  • @jonderry - that technique only works with lseekable input so it won't work when reading from a pipe. It will work if you redirect to a file >outfile and then run { head -n 1; sort; } <outfile – don_crissti Sep 26 '15 at 13:40
  • @jonderry I wonder if a specific line ending is observed in your particular tool. Some "Windows" command line tools are still coded for text processing of Linux line endings – Sun Feb 4 '20 at 3:25

11 Answers 11

Active Oldest Votes
77

Stealing Andy's idea and making it a function so it's easier to use:

# print the header (the first line of input)
# and then run the specified command on the body (the rest of the input)
# use it in a pipeline, e.g. ps | body grep somepattern
body() {
    IFS= read -r header
    printf '%s\n' "$header"
    "$@"
}

Now I can do:

$ ps -o pid,comm | body sort -k2
  PID COMMAND
24759 bash
31276 bash
31032 less
31177 less
31020 man
31167 man
...

$ ps -o pid,comm | body grep less
  PID COMMAND
31032 less
31177 less
Improve this answer
10
  • ps -C COMMAND may be more appropriate than grep COMMAND, but it's just an example. Also, you can't use -C if you also used another selection option such as -U. – Mikel Apr 23 '11 at 0:51
  • Or maybe it should be called body? As in body sort or body grep. Thoughts? – Mikel Apr 23 '11 at 0:57
  • 4
    Renamed from header to body, because you're doing the action on the body. Hopefully that makes more sense. – Mikel Apr 23 '11 at 1:02
  • 3
    Remember to call body on all subsequent pipeline participants: ps -o pid,comm | body grep less | body sort -k1nr – bishop Nov 7 '16 at 20:02
  • 2
    @Tim You can just write <foo body sort -k2 or body sort -k2 <foo. Just one extra character from what you wanted. – Mikel Sep 4 '17 at 13:49
46

You can keep the header at the top like this with bash:

command | (read -r; printf "%s\n" "$REPLY"; sort)

Or do it with perl:

command | perl -e 'print scalar (<>); print sort { ... } <>'
Improve this answer
11
  • 2
    +1 awesome. Worth bundling up as a shell function I think. – Mikel Apr 23 '11 at 0:42
  • 2
    +1, any reason why a subshell is preferable, or is {} ok instead of ()? – jonderry Apr 23 '11 at 0:57
  • 3
    IFS= disables word splitting when reading the input. I don't think it's necessary when reading to $REPLY. echo will expand backslash escapes if xpg_echo is set (not the default); printf is safer in that case. echo $REPLY without quotes will condense whitespace; I think echo "$REPLY" should be okay. read -r is needed if the input may contain backslash escapes. Some of this might depend on bash version. – Andy Apr 23 '11 at 1:50
  • 1
    @Andy: Wow, you're right, different rules for read REPLY; echo $REPLY (strips leading spaces) and read; echo $REPLY (doesn't). – Mikel Apr 23 '11 at 2:44
  • 1
    @Andy: IIRC, the default value of xpg_echo depends on your system, e.g. on Solaris I think it defaults to true. This is why Gilles likes printf so much: it's the only thing with predictable behavior. – Mikel Apr 23 '11 at 2:47
37

I found a nice awk version that works nicely in scripts:

awk 'NR == 1; NR > 1 {print $0 | "sort -n"}'
Improve this answer
4
  • 2
    I like this, but it requires a bit of explanation - the pipe is inside the awk script. How does that work? Is it calling the sort command externally? Does anyone know of at least a link to a page explaining pipe use within awk? – Wildcard Nov 7 '15 at 1:24
  • @Wildcard you can check the official manual page or this primer. – lapo Nov 2 '16 at 19:52
  • This code fails when I use these arguments to sort: sort -n -k 2b,2 -t $'\t'. The problem is nesting '\t' inside 'NR...{print...}'. The explanation of how to escape the 's is here – Josh Mar 28 '20 at 17:30
  • For fixed-width output, use the -b option, as it will make sort ignore leading blanks in the sort key. The default field separator is non-blank-to-blank transitions, so fields will start with leading blanks. For example, this command lists installed Python packages first by location, then by package name: pip list -v | awk 'NR <= 2; NR > 2 { print $0 | "sort -b -k 3,3 -k 1,1" };' – aparkerlue yesterday
5

Hackish but effective: prepend 0 to all header lines and 1 to all other lines before sorting. Strip the prefix after sorting.

… |
awk '{print (NR <= 2 ? "0 " : "1 ") $0}' |
sort -k 1 -k… |
cut -b 3-
Improve this answer
0
4

Here's some magic perl line noise that you can pipe your output through to sort everything but keep the first line at the top: perl -e 'print scalar <>, sort <>;'

Improve this answer
1
  • could you pls explain why this works? – törzsmókus Sep 7 '20 at 9:44
2

I tried the command | {head -1; sort; } solution and can confirm that it really screws things up--head reads in multiple lines from the pipe, then outputs just the first one. So the rest of the output, that head did not read, is passed to sort--NOT the rest of the output starting from line 2!

The result is that you are missing lines (and one partial line!) that were in the beginning of your command output (except you still have the first line) - a fact that is easy to confirm by adding a pipe to wc at the end of the above pipeline - but that is extraordinarily difficult to trace down if you don't know this! I spent at least 20 minutes trying to work out why I had a partial line (first 100 bytes or so cut off) in my output before solving it.

What I ended up doing, which worked beautifully and didn't require running the command twice, was:

myfile=$(mktemp)
whatever command you want to run > $myfile

head -1 $myfile
sed 1d $myfile | sort

rm $myfile

If you need to put the output into a file, you can modify this to:

myfile=$(mktemp)
whatever command you want to run > $myfile

head -1 $myfile > outputfile
sed 1d $myfile | sort >> outputfile

rm $myfile
Improve this answer
1
  • You can use ksh93's head builtin or the line utility (on systems that still have one) or gnu-sed -u q or IFS=read -r line; printf '%s\n' "$line", that read the input one byte at a time to avoid that. – Stéphane Chazelas Jan 11 '18 at 21:58
2

I think this is easiest.

ps -ef | ( head -n 1 ; sort )

or this which is possibly faster as it does not create a sub shell 

ps -ef | { head -n 1 ; sort ; }

Other cool uses

shuffle lines after header row

cat file.txt |  ( head -n 1 ; shuf )

reverse lines after header row

cat file.txt |  ( head -n 1 ; tac )
Improve this answer
2
0
command | head -1; command | tail -n +2 | sort
Improve this answer
1
  • 4
    This starts command two times. Therefore it is limited to some specific commands. However, for the requested ps command in the example, it would work. – jofel May 20 '14 at 12:00
0

Simple and straightforward! 

<command> | head -n 1; <command> | sed 1d | sort <....>
  • sed nd ---> 'n' specifies line no., and 'd' stands for delete.
Improve this answer
1
  • 1
    Just as jofel commented a year and a half ago on Sarva's answer, this starts command twice. So not really suitable for use in a pipeline. – Wildcard Nov 6 '15 at 2:36
0

I came here looking for a solution for the command w. This command shows details of who is logged in and what they are doing.

To show the results sorted, but with the headers kept at the top (there are 2 lines of headers), I settled on:

w | head -n 2; w | tail -n +3 | sort

Obviously this runs the command w twice and therefore may not be suitable for all situations. However, to its advantage it is substantially easier to remember.

Note that the tail -n +3 means 'show all lines from the 3rd onwards' (see man tail for details).

Improve this answer
-2

Try doing:

wc -l file_name | tail -n $(awk '{print $1-1}') file_name | sort
Improve this answer
1

Not the answer you're looking for? Browse other questions tagged or ask your own question.