Sorting text files or other data, for example with the sort utility.

learn more… | top users | synonyms

90
votes
13answers
60k views

how do you sort du output by size?

How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong. [~]# du -sh /var/* | sort -n 0 /var/mail 1.2M ...
56
votes
5answers
53k views

What is the difference between “sort -u” and “sort | uniq”?

Everywhere I see someone needing to get a sorted, unique list, they always pipe to sort | uniq. I've never seen any examples where someone uses sort -u instead. Why not? What's the difference, and why ...
41
votes
4answers
12k views

sort inplace - like sed --in-place - exists?

Am I blind or is there no option like --in-place for sort? In order to save results to the input file, sed uses -i (--in-place). Redirecting the output of sort to the input file sort < f > f ...
39
votes
8answers
34k views

List files sorted numerically

I have a bunch of files from log1 to log164. I'm trying to LIST the directory (sorted) in a UNIX terminal but the sort functions are only providing the format like this: home:logs Home$ ls -1 | sort ...
34
votes
9answers
19k views

sort but keep header line at the top

I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
32
votes
3answers
81k views

Sort based on the third column

I'm facing a huge 4-columns file. I'd like to display the sorted file in stdout based on its 3rd column: cat myFile | sort -u -k3 Is that enough to perform the trick?
31
votes
2answers
44k views

Trying to sort on two fields, second then first

I am trying to sort on multiple columns. The results are not as expected. Here's my data (people.txt): Simon Strange 62 Pete Brown 37 Mark Brown 46 Stefan Heinz 52 Tony Bedford 50 John Strange 51 ...
31
votes
5answers
38k views

Sorting the output of “find”?

I need to be able to alphabetically sort the output of find before piping it to a command. Entering | sort | between didn't work, so what could I do? find folder1 folder2 -name "*.txt" -print0 | ...
29
votes
2answers
1k views

Why is sort -o useful?

UNIX philosophy says: do one thing and do it well. Make programs that handle text, because that is a universal interface. The sort command, at least GNU sort, has an -o option to output to a file ...
27
votes
4answers
27k views

How to list files sorted by modification date recursively (no stat command available!)

How can I get the list of all files under current directory along with their modification date and sorted by that date? Now I know how to achieve that with find, stat and sort, but for some weird ...
23
votes
1answer
1k views

Why is sort changing the order of lines with identical sort keys?

Here is the data: D 2 B 2 A 2 When I run this command: sort -k2,2 file it outputs: A 2 B 2 D 2 My question is that when I specify only the second column -k2,2, why is it that it sorts by the ...
21
votes
5answers
7k views

How can I remove duplicates in my .bash_history, preserving order?

I really enjoying using control+r to recursively search my command history. I've found a few good options I like to use with it: # ignore duplicate commands, ignore commands starting with a space ...
20
votes
3answers
2k views

Why is coreutils sort slower than Python?

I wrote the following script to test the speed of Python's sort functionality: from sys import stdin, stdout lines = list(stdin) lines.sort() stdout.writelines(lines) I then compared this to the ...
18
votes
3answers
1k views

Scalability of 'sort -u' for gigantic files

What is reasonable scalability limit of 'sort -u' ? (in dimensions of "line length", "amount of lines", "total file size"?) What is Unix alternative for files exceeding this in dimension of "amount ...
16
votes
2answers
3k views

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets ...
15
votes
3answers
26k views

Sort and count number of occurrence of lines

I have Apache logfile, access.log, how to count number of line occurrence in that file? for example the result of cut -f 7 -d ' ' | cut -d '?' -f 1 | tr '[:upper:]' '[:lower:]' is a.php b.php a.php ...
15
votes
5answers
7k views

case-insensitive search of duplicate file-names

I there a way to find all files in a directory with duplicate filenames, regardless of the casing (upper-case and/or lower-case)?
14
votes
3answers
10k views

How to sort the string which combined with string + numeric using bash script?

This is the data what I want to sort. But sort treats the numeric to string, the data it no sorted as I expected. /home/files/profile1 /home/files/profile10 /home/files/profile11 ...
14
votes
1answer
43k views

Sort data in descending order of first column, for equal values, use second column in ascending order

Allow me to clarify: Assume I have some keywords with frequency of their usage: 12 Hi 7 Hash 7 C++ 9 Superuser 17 Stackoverflow 9 LaTeX 42 Life 9 Ubuntu What I want, is to sort this data ...
14
votes
2answers
5k views

Why does ls sorting ignore non-alphanumeric characters?

When sorting file names, ls ignores characters like -,_. I expected it to use those characters in sorting as well. An example: touch a1 a2 a-1 a-2 a_1 a_2 a.1 a.2 a,1 a,2 Now display these files ...
13
votes
2answers
12k views

How to sort big files?

I have a PC with Intel(R) Pentium(R) CPU G640 @ 2.80GHz and 8 GByte RAM. Installed Scientific Linux 6.5 on it with EXT3 filesystem. Q: What is the fastest way I can do a "sort -u list.txt -o ...
13
votes
6answers
20k views

Get text-file word occurrence count of all words & print output sorted

I had a command which would work through a text file, count all the occurrences of the words and print it out like this: remy@box $˜ magic-command-i-forgot | with grep | and awk | sort ./textfile.txt ...
13
votes
2answers
367 views

Fastest way to delete duplicates in large wordlist?

I need to deduplicate a large wordlist. I tried several commands and did some research here and here where they explain that the fastest way to deduplicate a wordlist seems to be using awk. awk ...
12
votes
4answers
758 views

Sort lines by number of words per line

Given input: hello: world foo bar baz bar: baz: bin boop bop fiz bang beep bap: bim bam bop boatkeeper: poughkeepsie I would like to sort it into most words at the top, to least at the end, like ...
12
votes
4answers
8k views

How do I make ls sort underscore characters first?

I like being able to name files and directories with an underscore prefix if it's something I want to keep separate from other files and directories at the same level. On Windows and Mac, for example, ...
11
votes
3answers
759 views

Sorting data faster approach

I need to sort a bed file randomly 10000 times and take the top 1000 rows each time. Currently, I am using the following code: for i in {1..100}; do for j in {1..100}; do sort -R ...
11
votes
4answers
1k views

Shuffle file randomly with some additional constraints

I have a huge music playlist and, while some artists have many albums, others have just one song. I wanted to sort the playlist so the same artist won't play twice in a row, or his songs won't end up ...
10
votes
6answers
4k views

Can I parallelize sort?

For example for bzip there is pbzip, a parallel version of bzip. Is there any such parallelization tools for sort to improve performance?
10
votes
1answer
235 views

How to sort a file based on a section of a field

I would like to sort a file : D104HN-D104HA 8.320 4.521 1.69e+05 -- D104HN-D104HB* 8.320 2.823 2.93e+05 -- A90HB#-A90HA 1.655 4.207 7.12e+05 -- ...
9
votes
2answers
1k views

sort by hex value

Using coreutils sort, how can I sort numerically by a hexadecimal value (field)? I was expecting something along the lines of sort -k3,3x file_to_sort however, such an x does not exist. Edit: ...
8
votes
3answers
11k views

How is it possible to sort ps command's CPU field?

I'm trying to make a small command that will find the processes that use the most CPU power. Firstly, I use ps aux > file.txt and then cut -c 16-20 file.txt | sort -n | tail -5. The result I get ...
8
votes
2answers
235 views

Why non-number record shows after “0” in sorting?

I want to sort files according to the number in the filename. Here are the files: $ ls *.f 0.f 13.f 1.f 22.f 4.f abc.f The sorting result: $ ls *.f | sort -t. -k1n 0.f abc.f # note this file! ...
8
votes
7answers
3k views

How to sort (by whatever key) BibTex entries in `.bib` file?

I work with LaTeX and do versioning with Git. For bibliography management I use Mendeley. The problem is that each time Mendeley synchronizes it's .bib exports, they are in different order, what ...
8
votes
1answer
753 views

Is gnu coreutils sort broken?

Consider the following input to sort: cat > foo <<EOM D,,5014978 DD,,25 D,I,1972765530 D,Y,4223624 -,Y,71285059 YA,I,2 EOM Now try running sort foo The output is not sorted when trying ...
8
votes
3answers
1k views

sort at jobs chronologically

Possibly a simple solution I misplaced. How can I get the output of atq sorted chronologically, so I can easily see who's due to run next? The man page for sort doesn't have anything built-in to ...
8
votes
5answers
623 views

How intelligent are unix utilities?

Unix utilities like sort, find, grep, diff et al are very handy to perform quick tasks, sometimes without writing any code at all. I wanted to know what algorithms do they internally use and how do ...
7
votes
4answers
2k views

uniq a csv file ignoring a column, awk maybe?

Given this file (annotations are not part of file, but form part of explanation)... x,a,001,b,c,d,y x,a,002,b,c,e,yy x,bb,003,b,d,e,y x,c,004,b,d,e,y x,c,005,b,d,e,y # nb - dupe of row 4 ...
7
votes
2answers
6k views

Counting number of times each IP address appears in log file

I have a file in the format as follows: $ cat file.txt 27.33.65.2 27.33.65.2 58.161.137.7 121.50.198.5 184.173.187.1 184.173.187.1 184.173.187.1 What's the best way to parse the file file.txt into ...
7
votes
2answers
5k views

sort output by column

I'd like to take this command find -maxdepth 1 -type d | while read -r dir; do printf "%s:\t" "$dir"; find "$dir" | wc -l; done ( from here ). which has an output of basically ./kennel: 11062 ...
7
votes
4answers
793 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
7
votes
1answer
15k views

Sorting file by second column

How can I manipulate a two columns tab-separated text file by sorting by the first element of the second column (only if the first column element is the same)? Exemple: Input File 1 A 1-2 A ...
7
votes
2answers
262 views

Sorting a file with integer in parentheses with no space

I am trying to sort a file like this (which is a list of subroutine names) cfn=(1370) __lib_file_MOD_file_open fn=(1368) __universe_MOD_general_boot fn=(916) __universe_MOD_general_main fn=(6142) ...
7
votes
3answers
667 views

How to sort vertically and horizontally?

Before (INPUT.txt): Foo#1 Foo#2 Foo#3 Foo#4 Foo#4 Foo#5 SUM Bar#1 0 0 0 0 3 0 3 Bar#2 2 0 1 0 0 0 3 Bar#3 0 0 0 2 2 0 4 Bar#4 0 0 1 ...
7
votes
2answers
1k views

Sort files by modification time

I've seen many questions and answers here using a construction along the lines of list_dir=`ls -t /path/to/dir/` for i in $list_dir; do or ls -t | while read i; do Now, I know that you shouldn't ...
7
votes
3answers
7k views

What's the default order of Linux sort?

For a long period I thought the default behavior of the sort program was using ASCII order. However, when I input the following lines into sort without any arguments: # @ I got: @ # But ...
7
votes
3answers
1k views

Unix sort out of memory

I'm sorting a 25 GB file containing 200M lines on Fedora 18 using: sort -S 10G -T /bigdisk bigfile and I am getting process killed due to no memory. The process isn't maxing out my RAM before death ...
7
votes
1answer
3k views

comm: file is not in sorted order

I used comm to compare two sorted files. Each line in these files are positive integer numbers. But the results show comm: file 1 is not in sorted order comm: file 2 is not in sorted order How ...
7
votes
2answers
191 views

Sort command inconsistent behavior

I want to sort two files but I cannot get consistent results. It seems there are problems with collation but I cannot understand the reason. In sample files separator is a single space: file1: a b B ...
6
votes
5answers
1k views

Grouped sorting of continuous paragraphs (separated by blank line)?

I think I'm pretty experienced now in sorting by columns; however, I haven't found anything so far how to sort continuous rows. Supposing we have a text file that looks like this: (very simplified, ...
6
votes
3answers
428 views

Print random lines respecting the order of the source file

I have a large text file (~500K lines) with short sentences (couple of words long). Additionally, there is some XML markup in most of the lines. Finally, the text file has been sorted before the ...