Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have tried this shell script on a SUSE 10 server, kernel 2.6.16.60, ext3 filesystem.

The script has a line like this:

cat file | awk '{print $1" "$2" "$3}' | sort -n > result

The file's size is about 3.2G, and I get the following error message: File size limit exceeded.

In this shell, ulimit -f is unlimited.

After I change script into this:

cat file | awk '{print $1" "$2" "$3}' >tmp
sort -n tmp > result

the problem is gone.

I don't know why, can anyone help me with an explanation?

share|improve this question

closed as too localized by Michael Mrozek Sep 20 '12 at 15:15

This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

    
what happens if you get rid of the cat? awk is quite capable of reading a file without cat's help. awk '{print $1" "$2" "$3}' file | sort -n > result. How much RAM & swap do you have? 64-bit or 32-bit system? –  cas Sep 20 '12 at 11:08
    
Don't crosspost, stuff like this tends to happen. Closing here –  Michael Mrozek Sep 20 '12 at 15:15
add comment

1 Answer 1

See duplicate question in serverfault:

The pipe version needs many more temporary files. You can inspect this quickly with the strace utility.

The pipe version use a quick exploding number of temporary files:

for i in {1..200000} ; do echo $i ; done |strace sort -n |& grep -e 'open.*/tmp/'
open("/tmp/sortb9Mhqd", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
open("/tmp/sortqKOVvG", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
open("/tmp/sortb9Mhqd", O_RDONLY)       = 3
open("/tmp/sortqKOVvG", O_RDONLY)       = 4

The file version doesn't use temporary files for the same data set. For bigger data sets it use extremely less temporary files.

for i in {1..200000} ; do echo $i ; done >/tmp/TESTDATA ; strace sort -n /TMP/TESTDATA |& grep -e 'open.*/tmp/'
share|improve this answer
    
But how does more files relate to "File (singular) size exceeded"? –  jw013 Sep 20 '12 at 13:11
    
More temporary files mean also more stress to the partition containing /tmp. –  H.-Dirk Schmitt Sep 20 '12 at 13:20
    
even if i use the sort -T option to specify a big-enough-directory ,the error still happen –  yboren Sep 21 '12 at 3:58
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.