Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

let me first explain my problem in detail. It's actually pretty simple. I have a huge .txt file, 300GB to be more preciese, and I would like to put all the distinct strings from the first column, that match my pattern into a different .txt file.

awk '{print $1}' file_name | grep -o '/ns/.*' | awk '!seen[$0]++' > test1.txt

This is what I've tried, and as far as I can see it works fine but the problem is that after some time I get the following error:

awk: program limit exceeded: maximum number of fields size=32767
    FILENAME="file_name" FNR=117897124 NR=117897124

Any suggestions for parsing through a file this big?

share|improve this question
What operating system? Is GNU awk (gawk) available? groups.google.com/a/broadinstitute.org/forum/m/#!topic/… – Mikel May 29 '14 at 13:33
cross-posting is deemed a cardinal sin around here – iruvar May 29 '14 at 15:05

Sounds to me like awk hits a huge line which would result in 32767 or more fields. I cannot reproduce that with my awk, though:

> echo | awk 'BEGIN {for(i=1;i<100000;i++) printf "%d ",i}; { print ""; }' >file
> awk '{ print $50000; }' too_long_line_for_awk.txt

> awk --version
GNU Awk 4.1.0, API: 1.0

You could use a tool that is more robust against long lines. You have to decide what the maximum length for the first field is. If we assume 100 then you can try this:

cut -b -100 file | awk ...

Furthermore (but this is unrelated to your problem) your awk | grep | awk construct doesn't make sense. That would be done this way:

awk '$1 ~ "/ns/" {sub("^.*/ns/","/ns/",$1); if( !seen[$1]++ ) print $1}' \
  file_name >test1.txt

debug suggestion

As Ramesh pointed out: It might be interesting to find the line causing the problem. The number of the problem line should be one of those following the number printed (or written to a file) by this command:

awk '{ print NR;}' | tail -n 1 >crashline.txt

If awk empties its buffers before "crashing" then it should be the next number (+1).

share|improve this answer
not sure. But will awk '{ if (length($1) > 32000) print $1;'} filename help identify the big line? – Ramesh May 29 '14 at 13:55
grep -o '/ns/.*' Gives me only the text that matches the pattern from every line. Which is exactly what I want. Is your awk command doing the same thing? So for example I have something like: afhdafsdfagasg/ns/importantstuff asgafhasdfahafdasfdhadf/ns/improtantstuff1 . . . My grep command will return only the /ns/importantstuff from every line. – Jovan Andonov May 29 '14 at 13:59
@Ramesh No. But awk '{ print NR;}' | tail -n 1 >crashline.txt will (maybe not precisely though, due to buffering). – Hauke Laging May 29 '14 at 14:03
@HaukeLaging, good suggestion. Probably you can add this as a debug method in the answer too :) – Ramesh May 29 '14 at 14:04
The line causing a problem is cited in the error message : " FNR=117897124 " – Olivier Dulac May 30 '14 at 7:03

It seems that your awk implement has limit the number of fields.

Example in mawk:


/*------- more than 1 fbank needed  ------------*/                              

  compute the address of a field with index                                     
  > MAX_SPLIT                                                                   

CELL *                                                                          
slow_field_ptr(int i)                                                           
    if (i > MAX_FIELD)                                                          
        rt_overflow("maximum number of fields", MAX_FIELD);

rt_overflow (define in error.c) is a function to generate error message at run time:

/* run time */                                                                  
rt_overflow(const char *s, unsigned size)                                       
    errmsg(0, "program limit exceeded: %s size=%u", s, size);                   

And in file size.h:

#define  FBANK_SZ    256                                                        
#define  FB_SHIFT      8    /* lg(FBANK_SZ) */                                  
#define  FBANK_SZ   1024                                                        
#define  FB_SHIFT     10    /* lg(FBANK_SZ) */                                  
#define  NUM_FBANK   128    /* see MAX_FIELD below */                           

#define  MAX_SPLIT  (FBANK_SZ-1)    /* needs to be divisble by 3 */             

You can see, MAX_FIELD default is 256*128 - 1 = 32767.

Using gawk can solve this problem.

share|improve this answer

Generally speaking, the more specialized a tool is, the better it is at coping with very large files. You could deal with that file in you awk, mind you — you'd just need to extract the first field manually instead of using the built-in field processing. You could combine the grep call and the second awk call into a single awk call, too.

awk -F '\n' '
    { sub(/[\t ].*/,"");
      if (match($0, "/ns/")) $0 = substr($0,RSTART); else next; }

However, a pipeline through specialized tools is likely to be faster. If your fields always use a tab as the separator, you can use cut to isolate the first field. If the separator is a space, make that cut -d ' '.

cut -f 1 | grep … | …

Alternatively, you can use sed for the first two steps. Whether this is faster than cut … | grep … depends on your data and on your implementation. In the sed call, replace \t by a literal tab character if your implementation doesn't understand \t; if your implementation doesn't understand \n in an s replacement, replace it by backslash-newline.

sed -n -e 's/[ \t].*//' \
    -e 's!/ns/!\n&!' -e 'b' \
    -e 's/^.*\n//p'

If there is always a single occurrence of /ns/ in the first field, you can simplify this to the following, which matches the last occurrence of /ns:

sed -n -e 's/[ \t].*//' -e 's!.*/ns/!/ns/!p'

Turning to the last step, if there are a lot of matches, then the awk command will use a lot of memory. If changing the order of the lines in the output is acceptable, you can use sort -u instead.

cut -f 1 | grep -o '/ns/.*' | sort -u
sed -n -e 's/[ \t].*//' -e 's!.*/ns/!/ns/!p' | sort -u
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.