17

I have a two-column file; the file is sorted the way I want it on column 1 already. I would like to sort on column 2, within each column 1 category. However, sort does not understand the sort order of column 1.

The normal way (from similar questions here on stack) would be this:

sort --stable -k1,1 -k2,2n

But I cannot specify the sort on k1, because it is arbitrary.

Example input:

C 2
C 1
A 2
A 1
B 2 
B 1

and output:

C 1
C 2
A 1
A 2
B 1 
B 2
0

2 Answers 2

19

You could use awk to start a new sort for each block:

% awk -v cmd="sort -k2,2" '$1 != prev {close(cmd); prev=$1} {print | cmd}' foo
C 1
C 2
A 1
A 2
B 1
B 2
  • $1 != prev {close(cmd); prev=$1} - when the saved value is different, we have a new block, so we close any previously started sort
  • {print | "sort -k2,2"}' pipes the output to sort, starting it if it isn't already running (awk can keep track of commands it starts)
6
  • 2
    awk is truly incredible. I like this a lot more than what I was expecting, which was an awk decorate-sort-undecorate!
    – Evan Benn
    Jul 30, 2019 at 4:08
  • I tried to compare perf of this vs the other answer, not sure why this one uses more resources... Any ideas? gist.github.com/EvanTheB/5b64eafb84eeaf51c289295ac06e1b0b
    – Evan Benn
    Jul 31, 2019 at 1:46
  • How many runs did you average across?
    – muru
    Jul 31, 2019 at 1:54
  • I did no averaging, but am seeing consistent runtimes as I repeat and investigate.
    – Evan Benn
    Jul 31, 2019 at 2:12
  • Here is a similar file to what I am using if you want to investigate: seq 30 | xargs -L1 bash -cs 'yes $1 | head -1000000 | paste - <(seq 1000000) | shuf' bash
    – Evan Benn
    Jul 31, 2019 at 2:17
12

You could use a Schwartzian transform (this is basically the decorate-sort-undecorate approach you alluded to in a comment, but likely more performant than muru's fine answer due to using a single sort invocation as opposed to multiple) - using awk add a prefix column that increments with a change in value in the first column, sort by the prefix column followed by the "second" column(whose ordinal position has temporarily shifted to 3 due to the presence of the prefix column), and finally get rid of the prefix column

awk '{print ($1 in a? c+0: ++c)"\t" $0; a[$1]}' file | sort -k1,1n  -k3,3 | cut -f 2-
3
  • I'm surprised, but you are correct, this was faster than the other answer! 3 minutes vs 2 minutes on my 100 million line file (~30 uniq first columns).
    – Evan Benn
    Jul 31, 2019 at 0:35
  • 1
    No need to keep an array of the unique key from the first column. I think it ought to be enough to compare the current line's first column against the previous.
    – Kusalananda
    Jul 31, 2019 at 10:11
  • Something like awk -v OFS="\t" '$1 != prev { key++ } { print key, $0; prev = $1 } (untested).
    – Kusalananda
    Jul 31, 2019 at 10:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.