Sorting bigger files gives segmentation fault

Question

mkdir tmp
sort -u *.txt -T tmp/ -o output.txt

there are just a few few ten GByte sized files that needs to be sorted..

The problem is that after a few minutes sort gives a segmentation fault.

The OS is an up-to-date Scientific Linux 6.6.

Question: How can I "sort -u" the files? Sort cannot handle larger files?.. The RAM isn't half full when it segfaults.. only one core is on 100%

Answer 1

In the past, I've had to sort files that are too big for sort. I assume this is also your problem, though if you provide more information, perhaps we can better diagnose your issue.

The solution to my issue was to break up the file with grep as a pre-processor. Take a look at your data to see where the clumps will be. I'm assuming it is decently spread out over the alphanumeric space, but I'll discuss how to deal with clumps later.

for char1 in other {0..9} {a..z}; do
  out="/tmp/sort.$char1"
  echo "Extracting lines starting with '$char1'"
  if [ "$char1" = "other" ]; then char1='[^a-z0-9]'; fi
  grep -i "^$char1" *.txt > "$out"
  sort -u "$out" >> output.txt || break
done

(This uses a bashism. To avoid it, explicitly name each of the 37 characters, like for char1 in other 0 1 2 3 4 5 6 7 8 9 0 a b c d e f ...)

Clumps: It is completely possible that some of these looped sort calls will segfault due to having too much data. Merely revise that iteration to be more broken into parts. This might be as simple as removing grep's -i flag and calling out each capital letter (don't forget to change the other to [^a-zA-Z0-9]), or it might require digging into the data. If this is a list of software packages, you may have too many lines that start with "lib" and so the /tmp/sort.l iteration will fail. The || break part of this loop will stop processing at this point so you can fix it and resume where you left off. Following the "lib" example, you might want to resume with something like this:

for char1 in 'l[a-h]' 'lia' lib{a..z} lib{0..9} 'li[c-z]' 'l[j-z]' {m..z}; do
  ...

This breaks the l list into parts before and after the lib* portion. It's a bit ugly, but it should work. Just pay attention to the desired order so you can preserve it.

Answer 2

Another possibility would be to sort each of the files separately and then merge them:

for f in *txt; do
    sort -u "$f" -T tmp/ > "$f".sorted
done
sort -mu *sorted

The -m option causes sort to merge already sorted files instead of attempting to sort them. That should result in a far lower memory usage and should avoid the segfault.

Answer 3

Combining the different answers: When you want to sort the file in pieces, try using split:

LARGETMP=/var/tmp
mkdir ${LARGETMP}
N_LINES=100000 # Adjust when to still too large or too small
split --lines=${N_LINES} bigfile splitted_
for small in splitted*; do
   sort -u -T ${LARGETMP} ${small} > sorted_${small}
   rm ${small}
done
echo "Done with sorting the splitted files, now concate the stuff"
sort -um -T ${LARGETMP} sorted_* > bigfile.sorted

Edit: As @ua2b commented, splitting on size mostly will split in the middle of lines.
(When the file doesn't has linefeeds, the whole idea of sorting is a bit strange).

Answer 4

Sort will need a harddisk location to store temporary data. /tmp is not the best place. The sort command stores working data in temporary disk files.
You can use the flag -T to specify a large temp dir (on a partition with a lot of free space).

Answer 5

I found a way on OpenBSD:

http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/sort.1?query=sort&sec=1

 -H
    Use a merge sort instead of a radix sort. This option should be used for files larger than 60MB.

But this isn't a fully OK solution, since it takes too much space... x>100 GByte isn't enough for it..