17

I have a two-column file; the file is sorted the way I want it on column 1 already. I would like to sort on column 2, within each column 1 category. However, sort does not understand the sort order of column 1.

The normal way (from similar questions here on stack) would be this:

sort --stable -k1,1 -k2,2n

But I cannot specify the sort on k1, because it is arbitrary.

Example input:

C 2
C 1
A 2
A 1
B 2 
B 1

and output:

C 1
C 2
A 1
A 2
B 1 
B 2
0

2 Answers 2

19

You could use awk to start a new sort for each block:

% awk -v cmd="sort -k2,2" '$1 != prev {close(cmd); prev=$1} {print | cmd}' foo
C 1
C 2
A 1
A 2
B 1
B 2
  • $1 != prev {close(cmd); prev=$1} - when the saved value is different, we have a new block, so we close any previously started sort
  • {print | "sort -k2,2"}' pipes the output to sort, starting it if it isn't already running (awk can keep track of commands it starts)
6
  • 2
    awk is truly incredible. I like this a lot more than what I was expecting, which was an awk decorate-sort-undecorate!
    – Evan Benn
    Jul 30, 2019 at 4:08
  • I tried to compare perf of this vs the other answer, not sure why this one uses more resources... Any ideas? gist.github.com/EvanTheB/5b64eafb84eeaf51c289295ac06e1b0b
    – Evan Benn
    Jul 31, 2019 at 1:46
  • How many runs did you average across?
    – muru
    Jul 31, 2019 at 1:54
  • I did no averaging, but am seeing consistent runtimes as I repeat and investigate.
    – Evan Benn
    Jul 31, 2019 at 2:12
  • Here is a similar file to what I am using if you want to investigate: seq 30 | xargs -L1 bash -cs 'yes $1 | head -1000000 | paste - <(seq 1000000) | shuf' bash
    – Evan Benn
    Jul 31, 2019 at 2:17
12

You could use a Schwartzian transform (this is basically the decorate-sort-undecorate approach you alluded to in a comment, but likely more performant than muru's fine answer due to using a single sort invocation as opposed to multiple) - using awk add a prefix column that increments with a change in value in the first column, sort by the prefix column followed by the "second" column(whose ordinal position has temporarily shifted to 3 due to the presence of the prefix column), and finally get rid of the prefix column

awk '{print ($1 in a? c+0: ++c)"\t" $0; a[$1]}' file | sort -k1,1n  -k3,3 | cut -f 2-
3
  • I'm surprised, but you are correct, this was faster than the other answer! 3 minutes vs 2 minutes on my 100 million line file (~30 uniq first columns).
    – Evan Benn
    Jul 31, 2019 at 0:35
  • 1
    No need to keep an array of the unique key from the first column. I think it ought to be enough to compare the current line's first column against the previous.
    – Kusalananda
    Jul 31, 2019 at 10:11
  • Something like awk -v OFS="\t" '$1 != prev { key++ } { print key, $0; prev = $1 } (untested).
    – Kusalananda
    Jul 31, 2019 at 10:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .