Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have large text file (1.5 G),

I want to know what the fastest and more reliable tool in linux is.

I usualy use:

awk '!x[$0]++' file.txt

But when I use htop command I see my memory usage is increasing...

I want to know whats is the fastest and more reliable one...(for huge files)

uniq?
sort?
sed?
awk?

And why?

Thank you

share|improve this question
    
Have you tried running them, possibly with time? –  choroba May 10 at 12:08
    
time is important and also memory usage and reliability(I mean which one does his job accurately) –  MortezaLSC May 10 at 12:10
    
Not yet ... But I did some tests before...and ask somewhere, some guys told me awk is the best..but in htop...I see memory usage is increasing –  MortezaLSC May 10 at 12:11
3  
@MortezaLSC: It is a trade off. The faster the program is, the more memory is used. –  cuonglm May 10 at 12:33

1 Answer 1

up vote 11 down vote accepted

Let's consider how each solution works.

  • uniq This requires that the file already be sorted. If not, you have to pipe it through sort first, which means that sort has to read the entire file into memory, reorder it (O(n log n)), and then write it into the pipe. The work of uniq is very cheap, since it only has to compare adjacent lines of its input.

  • sort -u This combines the work of sort | uniq. This has to collect all the unique inputs into memory like the awk script does, but it also then wastes time sorting them before producing the output. This is O(n log n), although in this case n is the number of unique items, not all the inputs. So it's better than the pipe.

  • sed I'm not sure why you listed this, as I can't think of a good way to do this with sed at all. Maybe if you first sort it and pipe to a sed script, there's a way to compare adjacent lines. So sed would just be doing what uniq does, and uniq probably does it about as efficiently as possible.

  • awk This is likely the best because it only does the minimal amount of work necessary. As it reads each line, it does an efficient hash lookup to see if the line is already in its memory, and only stores the unique lines. When it's done reading the input, it just dumps the hash table in its internal order, without needing to sort it. This uses O(n) time and O(uniq n) memory.

Every method will use a considerable amount of memory, either for sorting the input or keeping track of which inputs have seen so they can remove duplicates.

share|improve this answer
    
Thank you very much....seems this is the answer –  MortezaLSC May 10 at 13:06
1  
+1 The explanation regarding awk also explains why it uses increasing amounts of memory. Anything which does a sort will end up doing this as well, only 1) it will probably use it all at once, 2) it may use slightly more, depending on the number of unique vs. duplicated keys. –  goldilocks May 10 at 13:09
    
@Barmar pardon, But when I have a large file(16 G) with memory capacity 8G , So what is going to happen with my memory? –  MortezaLSC May 10 at 13:30
5  
@goldilocks, sort resorts to temporary files (in an intelligent way) to avoid filling up the memory. Its memory usage is bound. The boundary is customisable with some sort implementations. It's more efficient that letting the system swapping memory randomly to disk (which also affects also applications on the system). –  Stéphane Chazelas May 10 at 15:19
    
That's true. So if you run into a case where awk runs out of memory, sort may be the only solution because it has been designed to deal with this. On the other hand, all that disk reading and writing will slow it down, so it will probably take a long time to complete. If you're dealing with such large amounts of data, you should probably be using a DBMS rather than text files. –  Barmar May 10 at 19:11

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.