Sorting lines in a file by a numeric value burried inside

Question

I'm writing a Bash script to test USB flash memory for errors using f3 tool. I have text like this (created by grepping logs from f3read program):

2017-10-25_09:30:22/sdf.log:Data LOST: 4.00 KB (8 sectors)
2017-10-25_09:30:22/sdi.log:Data LOST: 5.00 KB (10 sectors)
2017-10-25_09:30:22/sdj.log:Data LOST: 2.35 MB (4822 sectors)
2017-10-25_09:30:22/sdn.log:Data LOST: 5.00 KB (10 sectors)
2017-10-25_09:30:22/sdo.log:Data LOST: 4.00 KB (8 sectors)
2017-10-25_09:30:22/sdp.log:Data LOST: 4.00 KB (8 sectors)
2017-10-25_09:30:22/sdq.log:Data LOST: 2.00 KB (4 sectors)
2017-10-25_14:37:03/sdb.log:Data LOST: 5.00 KB (10 sectors)
2017-10-25_14:37:03/sdc.log:Data LOST: 3.00 KB (6 sectors)
2017-10-26_09:17:59/sdd.log:Data LOST: 3.00 KB (6 sectors)
2017-10-26_09:17:59/sde.log:Data LOST: 2.00 KB (4 sectors)
2017-10-26_09:17:59/sdf.log:Data LOST: 3.00 KB (6 sectors)
2017-10-26_09:17:59/sdg.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdh.log:Data LOST: 611.29 MB (1251918 sectors)
2017-10-26_09:17:59/sdi.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdl.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdo.log:Data LOST: 3.00 KB (6 sectors)
2017-10-26_09:17:59/sdp.log:Data LOST: 2.00 KB (4 sectors)
2017-10-26_09:17:59/sdq.log:Data LOST: 414.60 MB (849106 sectors)
2017-10-26_09:17:59/sdr.log:Data LOST: 65.29 MB (133712 sectors)
2017-10-26_09:17:59/sds.log:Data LOST: 5.00 KB (10 sectors)

I would like to sort the lines by the number of bad sectors noted at the end of the line. I tried using sort, but I don't know how to use it's --key option to make it do what I want.

I cannot cut the lines first, becasue I need the drive name (sda, sdb, etc) to be then extracted for a report.

Stéphane Chazelas · Accepted Answer · 2017-11-15 17:58:05Z

10

POSIXly, you can do:

sort -t '(' -k2n < file

That is set the field separator to ( and sort on the second field (or rather the portion of the line starting with the second field) numerically.

Alternatively, you could keep the default field separator (the transition from a non-blank to a blank) where the 5^th field would be like " (12" and use:

sort -k5.3n < file

(that is, sort numerically on the portion of the line starting with the 3^rd character of the 5^th field).

For ties, the last-resort sorting order comes into play, and that's lexical comparison of the full line (which conveniently here will give you a chronological order).

If you wanted to sort ties on drive name instead, you could use:

sort -t '(' -k2n -k1.21

(second key being the portion of line starting with the 21^st character, lexical comparison)

edited Nov 15, 2017 at 17:58

answered Nov 15, 2017 at 12:22

Stéphane Chazelas

525k91 gold badges1015 silver badges1503 bronze badges

Why is it sort < file rather than just sort file?
– dessert
Nov 15, 2017 at 12:24
3

@dessert, why sort file rather than just sort < file? The shell can open the file, no need to pass it to sort. See also When should I use input redirection?
– Stéphane Chazelas
Nov 15, 2017 at 12:29

Add a comment |

Community · Accepted Answer · 2020-06-11 14:16:50Z

Use sort -V if that option is available

-V, --version-sort

natural sort of (version) numbers within text

$ <cmd> | sort -k5,5V
2017-10-25_09:30:22/sdq.log:Data LOST: 2.00 KB (4 sectors)
2017-10-26_09:17:59/sde.log:Data LOST: 2.00 KB (4 sectors)
2017-10-26_09:17:59/sdp.log:Data LOST: 2.00 KB (4 sectors)
2017-10-25_14:37:03/sdc.log:Data LOST: 3.00 KB (6 sectors)
2017-10-26_09:17:59/sdd.log:Data LOST: 3.00 KB (6 sectors)
2017-10-26_09:17:59/sdf.log:Data LOST: 3.00 KB (6 sectors)
2017-10-26_09:17:59/sdo.log:Data LOST: 3.00 KB (6 sectors)
2017-10-25_09:30:22/sdf.log:Data LOST: 4.00 KB (8 sectors)
2017-10-25_09:30:22/sdo.log:Data LOST: 4.00 KB (8 sectors)
2017-10-25_09:30:22/sdp.log:Data LOST: 4.00 KB (8 sectors)
2017-10-25_09:30:22/sdi.log:Data LOST: 5.00 KB (10 sectors)
2017-10-25_09:30:22/sdn.log:Data LOST: 5.00 KB (10 sectors)
2017-10-25_14:37:03/sdb.log:Data LOST: 5.00 KB (10 sectors)
2017-10-26_09:17:59/sds.log:Data LOST: 5.00 KB (10 sectors)
2017-10-26_09:17:59/sdg.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdi.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdl.log:Data LOST: 6.00 KB (12 sectors)
2017-10-25_09:30:22/sdj.log:Data LOST: 2.35 MB (4822 sectors)
2017-10-26_09:17:59/sdr.log:Data LOST: 65.29 MB (133712 sectors)
2017-10-26_09:17:59/sdq.log:Data LOST: 414.60 MB (849106 sectors)
2017-10-26_09:17:59/sdh.log:Data LOST: 611.29 MB (1251918 sectors)

unfa · Accepted Answer · 2017-11-15 12:22:12Z

3

I've found it.

sort -n -t '(' -k2V

-n tells sort to read numeric values in the strings

-t '(' tells it to use ( character as field delimiter. Becasue the word sectors) is always the same later on it won't affect the sorting order.

-k2V defines a custom key, using the second column - text after first ( character - for sorting.

answered Nov 15, 2017 at 12:22

unfa

1,7254 gold badges23 silver badges34 bronze badges

Note that you don't need the (GNU-specific) V flag here (which by the way makes -n redundant). That would be useful if the key didn't start with a digit like in @sundeep's approach where the key starts with " (". V is for version sort where v3.4 sorts before v3.20 (but after u5 or V5) which are not numbers and would be considered as 0 with the n flag. Same for " (2" vs " (10".
– Stéphane Chazelas
Nov 16, 2017 at 8:52

Add a comment |

drl · Accepted Answer · 2017-11-15 12:44:25Z

If your standard sort version does not offer option -V, then you might be able to use an alternate sorting program msort. It allows the use of a hybrid option:
Hybrid comparison is like lexicographic comparison except for the fact that strings of digits are treated as numbers.

So a snippet like this:

# Utility functions: print-as-echo, print-line-with-visual-space.
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
pl " Input data file $FILE:"
tail -v $FILE

pl " Expected output:"
tail -v $E

pl " Results:"
msort -q -j -l -n 5,5 -c hybrid $FILE |
tail -v

produces this:

-----
 Input data file data1:
==> data1 <==
2017-10-26_09:17:59/sdf.log:Data LOST: 3.00 KB (6 sectors)
2017-10-26_09:17:59/sdg.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdh.log:Data LOST: 611.29 MB (1251918 sectors)
2017-10-26_09:17:59/sdi.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdl.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdo.log:Data LOST: 3.00 KB (6 sectors)
2017-10-26_09:17:59/sdp.log:Data LOST: 2.00 KB (4 sectors)
2017-10-26_09:17:59/sdq.log:Data LOST: 414.60 MB (849106 sectors)
2017-10-26_09:17:59/sdr.log:Data LOST: 65.29 MB (133712 sectors)
2017-10-26_09:17:59/sds.log:Data LOST: 5.00 KB (10 sectors)

-----
 Expected output:
==> expected-output1 <==
2017-10-25_09:30:22/sdn.log:Data LOST: 5.00 KB (10 sectors)
2017-10-25_14:37:03/sdb.log:Data LOST: 5.00 KB (10 sectors)
2017-10-26_09:17:59/sds.log:Data LOST: 5.00 KB (10 sectors)
2017-10-26_09:17:59/sdg.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdi.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdl.log:Data LOST: 6.00 KB (12 sectors)
2017-10-25_09:30:22/sdj.log:Data LOST: 2.35 MB (4822 sectors)
2017-10-26_09:17:59/sdr.log:Data LOST: 65.29 MB (133712 sectors)
2017-10-26_09:17:59/sdq.log:Data LOST: 414.60 MB (849106 sectors)
2017-10-26_09:17:59/sdh.log:Data LOST: 611.29 MB (1251918 sectors)

-----
 Results:
==> standard input <==
2017-10-25_09:30:22/sdi.log:Data LOST: 5.00 KB (10 sectors)
2017-10-25_09:30:22/sdn.log:Data LOST: 5.00 KB (10 sectors)
2017-10-25_14:37:03/sdb.log:Data LOST: 5.00 KB (10 sectors)
2017-10-26_09:17:59/sdi.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdl.log:Data LOST: 6.00 KB (12 sectors)
2017-10-26_09:17:59/sdg.log:Data LOST: 6.00 KB (12 sectors)
2017-10-25_09:30:22/sdj.log:Data LOST: 2.35 MB (4822 sectors)
2017-10-26_09:17:59/sdr.log:Data LOST: 65.29 MB (133712 sectors)
2017-10-26_09:17:59/sdq.log:Data LOST: 414.60 MB (849106 sectors)
2017-10-26_09:17:59/sdh.log:Data LOST: 611.29 MB (1251918 sectors)

Some details for msort:

msort   sort records in complex ways (man)
Path    : /usr/bin/msort
Version : 8.53
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYS ...)
Help    : probably available with -h,--help
Repo    : Debian 8.9 (jessie) 
Home    : http://www.billposer.org/Software/msort.html (pm)

On a system like:

OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.9 (jessie) 
bash GNU bash 4.3.30

Best wishes ... cheers, drl

I managed to do it with sort, even without the -V option, and msort is not by default installed on Linux Mint 18.2. Thanks anyway! — unfa, Nov 15, 2017 at 13:22
Hi. OK, good to see that you solved it, but keep msort in mind for tasks that sort might not be able to handle, and it's available at least on systems like Linux Mint 17.1 Rebecca ... cheers — drl, Nov 15, 2017 at 13:29

Stéphane Chazelas · Accepted Answer · 2017-11-16 08:58:30Z

3

A slightly more flexible solution extracts the sector count, duplicates it at the beginning of the line, sorts, and finally removes the duplicated part again:

sed 's/^.*(\([0-9]\{1,\} \)sectors)$/\1&/' |
  sort -n |
  cut -d ' ' -f 2-

This solution is more flexible because it works for arbitrary patterns that identify the sort key in each record.

edited Nov 16, 2017 at 8:58

Stéphane Chazelas

525k91 gold badges1015 silver badges1503 bronze badges

answered Nov 15, 2017 at 20:15

David Foerster

1,5151 gold badge11 silver badges18 bronze badges

Add a comment |

Stack Exchange Network

Sorting lines in a file by a numeric value burried inside

5 Answers 5

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
sort
.

Hot Network Questions

Sorting lines in a file by a numeric value burried inside

5 Answers 5

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged sort.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
sort
.