4
mkdir tmp
sort -u *.txt -T tmp/ -o output.txt

there are just a few few ten GByte sized files that needs to be sorted..

The problem is that after a few minutes sort gives a segmentation fault.

The OS is an up-to-date Scientific Linux 6.6.

Question: How can I "sort -u" the files? Sort cannot handle larger files?.. The RAM isn't half full when it segfaults.. only one core is on 100%

  • what is tmp free disk size: df -h tmp – jet Mar 6 '15 at 22:58
  • You've mentioned a few tens of GB. Approximately how many lines, please? – roaima Mar 7 '15 at 23:41
  • It doesn't matter in this context, but in the event a reader is wondering what Scientific Linux is: Like CentOS, Scientific Linux is an unbranded fully compatible clone of the Red Hat Enterprise Linux of the same version. The default configuration of Scientific Linux is, unsurprisingly, tailored to scientific pursuits. It is also minimalistic. – Adam Katz Mar 13 '15 at 23:28
3
+75

In the past, I've had to sort files that are too big for sort. I assume this is also your problem, though if you provide more information, perhaps we can better diagnose your issue.

The solution to my issue was to break up the file with grep as a pre-processor. Take a look at your data to see where the clumps will be. I'm assuming it is decently spread out over the alphanumeric space, but I'll discuss how to deal with clumps later.

for char1 in other {0..9} {a..z}; do
  out="/tmp/sort.$char1"
  echo "Extracting lines starting with '$char1'"
  if [ "$char1" = "other" ]; then char1='[^a-z0-9]'; fi
  grep -i "^$char1" *.txt > "$out"
  sort -u "$out" >> output.txt || break
done

(This uses a bashism. To avoid it, explicitly name each of the 37 characters, like for char1 in other 0 1 2 3 4 5 6 7 8 9 0 a b c d e f ...)

Clumps: It is completely possible that some of these looped sort calls will segfault due to having too much data. Merely revise that iteration to be more broken into parts. This might be as simple as removing grep's -i flag and calling out each capital letter (don't forget to change the other to [^a-zA-Z0-9]), or it might require digging into the data. If this is a list of software packages, you may have too many lines that start with "lib" and so the /tmp/sort.l iteration will fail. The || break part of this loop will stop processing at this point so you can fix it and resume where you left off. Following the "lib" example, you might want to resume with something like this:

for char1 in 'l[a-h]' 'lia' lib{a..z} lib{0..9} 'li[c-z]' 'l[j-z]' {m..z}; do
  ...

This breaks the l list into parts before and after the lib* portion. It's a bit ugly, but it should work. Just pay attention to the desired order so you can preserve it.

  • A lot of other answers here try to run sort in parts as I have, but because their output is not itself sorted, they need to run sort on the collected output of the smaller sorts. This will often fail for the same reason you couldn't sort the original data in one pass. Because I'm using grep as a pre-processor, the smaller $out pieces are already sorted relative to each other, so when they are sorted themselves, the result can be concatenated rather than sorted once more. – Adam Katz Mar 13 '15 at 23:20
  • Your script is try and error (when all lines start with a space all lines come in 1 file), and uses a lot time grepping again and again. It is the last resort when all other solutions fail. – Walter A Mar 14 '15 at 10:00
  • @WalterA is right in that there can be some trial and error with this method, especially if you really don't know your data, but this will work very well for most data without any failures, and if you can anticipate things (like leading spaces), you can reduce the chance of those errors. Other answers here will fail without possible recovery if sorting the pieces is too much of a task. – Adam Katz Mar 14 '15 at 14:07
3

Another possibility would be to sort each of the files separately and then merge them:

for f in *txt; do
    sort -u "$f" -T tmp/ > "$f".sorted
done
sort -mu *sorted

The -m option causes sort to merge already sorted files instead of attempting to sort them. That should result in a far lower memory usage and should avoid the segfault.

2

Combining the different answers: When you want to sort the file in pieces, try using split:

LARGETMP=/var/tmp
mkdir ${LARGETMP}
N_LINES=100000 # Adjust when to still too large or too small
split --lines=${N_LINES} bigfile splitted_
for small in splitted*; do
   sort -u -T ${LARGETMP} ${small} > sorted_${small}
   rm ${small}
done
echo "Done with sorting the splitted files, now concate the stuff"
sort -um -T ${LARGETMP} sorted_* > bigfile.sorted

Edit: As @ua2b commented, splitting on size mostly will split in the middle of lines.
(When the file doesn't has linefeeds, the whole idea of sorting is a bit strange).

  • 1
    split --bytes does not care about linebreaks IIRC, so you better use split --lines=NUMBER for this purpose. This will not save you from using wc a number of times to get a feeling of how many lines make up a reasonable split size. Otherwise a charming solution, my favorite for now. – ua2b Mar 12 '15 at 3:59
  • @ua2b is right, I edited the answer. – Walter A Mar 12 '15 at 9:00
1

Sort will need a harddisk location to store temporary data. /tmp is not the best place. The sort command stores working data in temporary disk files.
You can use the flag -T to specify a large temp dir (on a partition with a lot of free space).

  • The asker has already specified -T tmp (not -T /tmp) – roaima Mar 7 '15 at 23:40
  • I can not see the free diskspace on the partition where the tmp dir was created. I mentioned /tmp since that is a default. – Walter A Mar 8 '15 at 10:45
0

I found a way on OpenBSD:

http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/sort.1?query=sort&sec=1

 -H
    Use a merge sort instead of a radix sort. This option should be used for files larger than 60MB.

But this isn't a fully OK solution, since it takes too much space... x>100 GByte isn't enough for it..

  • I hope this BSD solution one works on your Scientific Linux 6.6. If not, please try another solution and award the bounty. – Walter A Mar 14 '15 at 14:16
  • One way or the other, there is a sort option to compress temporary files that can be used in this kind of situations: --compress-program=PROG compress temporaries with PROG; decompress them with PROG -d – JJoao Apr 8 '15 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.