Hot answers tagged

107

If you have GNU coreutils (common in most Linux distributions), you can use du -sh * | sort -h. The -h option tells sort that the input is the human-readable format (number with unit). This feature was added to GNU Core Utilities 7.5 in Aug 2009. Note: If you are using Mac OSX, you need install coreutils with brew install coreutils, then use ...


99

Why not to use built-in ls feature for this particular case, namely -v natural sort of (version) numbers within text For example ls -1v log*


61

sort | uniq existed before sort -u, and is compatible with a wider range of systems, although almost all modern systems do support -u -- it's POSIX. It's mostly a throwback to the days when sort -u didn't exist (and people don't tend to change their methods if the way that they know continues to work, just look at ifconfig vs. ip adoption). The two were ...


50

sort has the -o, --output option that takes a filename as argument. If it is the same as the input file, it writes the result to a temporary file, then overwrites the original input file (exactly the same thing as what sed -i does). From GNU sort info page: `-o OUTPUT-FILE' `--output=OUTPUT-FILE' Write output to OUTPUT-FILE instead of standard output....


44

A key specification like -k2 means to take all the fields from 2 to the end of the line into account. So Villamor 44 ends up before Villamor 50. Since these two are not equal, the first comparison in sort -k2 -k1 is enough to discriminate these two lines, and the second sort key -k1 is not invoked. If the two Villamors had had the same age, -k1 would have ...


44

It is not just GNU sort that has it. BSD sort has it too. And as to why? (I thought it was a good question too...) From the man page: "The argument given is the name of an output file to be used instead of the standard output. This file can be the same as one of the input files." You can't go to the same file with redirection, the output redirection ...


43

sort -k 3,3 myFile would display the file sorted by the 3rd column assuming the columns are separated by sequences of blanks (ASCII SPC and TAB characters in the POSIX/C locale), according to the sort order defined by the current locale. Note that the leading blanks are included in the column (the default separator is the transition from a non-blank to a ...


36

The sort that you find on Linux comes from the coreutils package and implements an External R-Way merge. It splits up the data into chunks that it can handle in memory, stores them on disc and then merges them. The chunks are done in parallel, if the machine has the processors for that. So if there was to be a limit, it is the free disc space that sort can ...


35

Some versions of sort have a -z option, which allows for null-terminated records. find folder1 folder2 -name "*.txt" -print0 | sort -z | xargs -r0 myCommand Additionally, you could also write a high-level script to do it: find folder1 folder2 -name "*.txt" -print0 | python -c 'import sys; sys.stdout.write("\0".join(sorted(sys.stdin.read().split("\0"))))' ...


32

One difference is that uniq has a number of useful additional options, such as skipping fields for comparison and counting the number of repetitions of a value. sort's -u flag only implements the functionality of the unadorned uniq command.


30

Stealing Andy's idea and making it a function so it's easier to use: # print the header (the first line of input) # and then run the specified command on the body (the rest of the input) # use it in a pipeline, e.g. ps | body grep somepattern body() { IFS= read -r header printf '%s\n' "$header" "$@" } Now I can do: $ ps -o pid,comm | body ...


28

With POSIX compliant sorts and uniqs (GNU uniq is currently not compliant in that regard), there's a difference in that sort uses the locale's collating algorithm to compare strings (will typically use strcoll() to compare strings) while uniq checks for byte-value identity (will typically use strcmp()). That matters for at least two reasons. In some ...


27

Try using the -k flag to count 1K blocks intead of using human-readable. Then, you have a common unit and can easily do a numeric sort. du -ck | sort -n You don't explictly require human units, but if you did, then there are a bunch of ways to do it. Many seem to use the 1K block technique above, and then make a second call to du. http://serverfault....


26

| sort | uniq -c As stated in the comments. Piping the output into sort organises the output into alphabetical/numerical order. This is a requirement because uniq only matches on repeated lines, ie a b a If you use uniq on this text file, it will return the following. This is because the two as are separated by the b - they are not consecutive lines. ...


25

My shortest method uses zsh: print -rl **/*(.Om) If you have GNU find, make it print the file modification times and sort by that. I assume there are no newlines in file names. find . -type f -printf '%T@ %p\n' | sort -k 1 -n | sed 's/^[^ ]* //' If you have Perl (again, assuming no newlines in file names): find . -type f -print | perl -l -ne ' $_{$...


25

This is very similar to this question. The trouble is that you have an alphanumeric field that you are sorting on, and -n doesn't treat that sensibly, however version sort (-V) does. Thus use: sort -V Note that this is currently, AFAIK, a GNU sort only solution.


25

That's the last resort comparison. When comparing two lines, if all the keys compare equal, then as a last resort, a basic string comparison of the whole lines is performed (-r still applies but not the other options). That behavior is specified by POSIX: Except when the -u option is specified, lines that otherwise compare equal shall be ordered as if ...


24

You can keep the header at the top like this with bash: command | (read -r; printf "%s\n" "$REPLY"; sort) Or do it with perl: command | perl -e 'print scalar (<>); print sort { ... } <>'


24

With GNU ls (i.e. on Linux, Cygwin, or other systems that have GNU ls specifically installed): ls -v In zsh: echo *(n) In other shells: echo log?.gz log??.gz log???.gz Replace echo by printf '%s\n' if you want each file name on a separate line. If you want file metadata as well (ls -l) and you don't have GNU ls, you'll need to call ls separately ...


22

GNU sort (which is the default on most Linux systems), has a --parallel option. From http://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html: ‘--parallel=n’ Set the number of sorts run in parallel to n. By default, n is set to the number of available processors, but limited to 8, as there are diminishing performance gains after ...


22

Using the sort command will probably be the fastest option. But you'll probably want to fix the locale to C. sort -u doesn't report unique lines, but one of each set of lines that sort the same. In the C locale, 2 different lines necessarily don't sort the same, but that's not the case in most UTF-8 based locales on GNU systems. Also, using the C locale ...


21

Sorting the history This command works like sort|uniq, but keeps the lines in place nl|sort -k 2|uniq -f 1|sort -n|cut -f 2 Basically, prepends to each line its number. After sort|uniq-ing, all lines are sorted back according to their original order (using the line number field) and the line number field is removed from the lines. This solution has ...


19

bash's braces, {}, will enumerate them in order: for file in log{1..164}.gz; do process "$file" done


19

Specify the sort keys separately with the criteria: sort -k1,1nr -k2,2 inputfile This specifies that the first key is sorted numerically in reverse order while the second is sorted as per the default sort order. Quoting from POSIX sort: -k keydef The keydef argument is a restricted sort key field definition. The format of this definition is: ...


19

Izkata's comment revealed the answer: locale-specific comparisons. The sort command uses the locale indicated by the environment, whereas Python defaults to a byte order comparison. Comparing UTF-8 strings is harder than comparing byte strings. $ time (LC_ALL=C sort <numbers.txt >s2.txt) real 0m5.485s user 0m14.028s sys 0m0.404s How about ...


19

sort -k 1.2bn < file Sorts numerically on a key starting with the 2nd character of the 1st field ignoring leading blanks (and ending at the end of the line, but that doesn't matter for a numerical sort which only considers the initial sequence of decimal digits). Note that if there's a tie, like in between these two lines: F91HE*-K92HA 7.242 ...


18

Pipe the lines through sort -n -r -k2. Edited to sort from largest to smallest.


18

With awk, you could do: awk -F, -vOFS=, '{l=$0; $3=""}; ! ($0 in seen) {print l; seen[$0]}'


17

Let's see, !a[$0]++ first a[$0] we look at the value of a[$0] (array a with whole input line ($0) as key). If it does not exist ( ! is negation in test will eval to true) !a[$0] we print the input line $0 (default action). Also, we add one ( ++ ) to a[$0], so next time !a[$0] will evaluate to false. Nice, find!! You should have a look at code ...


16

I would use tr instead of awk: echo "Lorem ipsum dolor sit sit amet." | tr [:space:] '\n' | grep -v "^\s*$" | sort | uniq -c | sort -bnr tr just replaces spaces with newlines grep -v "^\s*$" trims out empty lines sort to prepare as input for uniq uniq -c to count occurrences sort -bnr sorts in numeric reverse order while ignoring whitespace wow. it ...



Only top voted, non community-wiki answers of a minimum length are eligible