Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have an 11gb wordlist file which is already sorted as each word is on its own line.

I need to remove duplicates and lines starting from 077.

I guess I need to run sed and sort -u together but I also want a live output (display whats happening in terminal) and if possible display the time left. All of this in one command and it must be able to run optimally at full performance under a LiveCD or possibly installed BackTrack 5 rc3.

Time is not very important but if there is a way for me to calculate the ETA, I might be able to borrow my dad's i7 based CPU which should process it faster obviously otherwise I'll have to use an older core 2 CPU.

The problem I'm facing with sort command is that under a VMWare player running it live, it doesnt have enough space so I have to specify temp files on my 32gb USB by using the -T command. I guess this won't be a problem had I installed Linux.

So please give me the complete command, be it sed/sort/awk to do this (whichever is most optimal).

share|improve this question
You might want to include a sample of the file, so people can see what you are dealing with. And any approaches you have tried yourself. – jasonwryan Nov 27 '12 at 0:59
@jasonwryan umm how do i give a sample of 11gb file lol. Anyways just imagine the file contains 1 password on each line like a list. You would be familiar if you ever used a password dictionary file. Also i know how to run those command individually but they take a lot of time since its a huge file, i just want simply it and get the additional options listed in one command. – promicin Nov 27 '12 at 3:00

Use pv for progress. You don't need a tempfile since you're only removing text. Just overwrite the file in place. If the file is already sorted, you don't need sort -u, just uniq.

pv file | {
  uniq | grep -v '^077'
  perl -e 'truncate STDOUT, tell STDOUT'
} 1<> file

The perl line is to truncate the file at the point were grep finished to write.

Note that since you're writing the file in-place, if you make a mistake, you won't be able to go back.

share|improve this answer
do i copy this whole script and paste into terminal after changing the input/output file ? I also need to install pv for this to work. Is this script any better then what Leonid has suggested below ? Because i rather not sacrifice performance. – promicin Nov 27 '12 at 11:24
Yes. Mine is a simplified version of Leonid's (uniq < infile | greo -v '^077' > outfile) with the addition of the progress bar (pv), and inplace editing (so you don't need extra free space). – Stéphane Chazelas Nov 27 '12 at 12:00
Thank you Stephane, i will try your script tonight. – promicin Nov 27 '12 at 15:33

I'm not sure you can have ETA or any progress display without significantly increasing the complexity of the sed/awk/shell script (and slowing the whole thing as a result). If you just want it to be as fast as possible, just try cat source_file | uniq | sed -n -e '/^077/!p' > dest_file. For an approximation of the progress display, you might want to watch the growing size of the dest_file with that command running in background or in another terminal.

share|improve this answer
Thanks, this one looks pretty straight forward, its running uniq and then piping into sed to remove numbers starting from 077 and saving the output in the destination file. I have also heard of awk with this command: awk '!x[$0]++' is it any good compared to uniq ? – promicin Nov 27 '12 at 11:19
If the file is already sorted, you won't be able to beat uniq. That awk command will end up storing all unique lines in memory which for so big a file is probably not a good idea and has no benefit otherwise. That sed line is a convoluted (double-negation) way to write sed '/^077/d' though you don't need sed here as grep is more than enough. You don't need cat either (UUOC). – Stéphane Chazelas Nov 27 '12 at 11:57
I completely agree with Stephane: just like it's hard to beat uniq in doing its specific job, grep shall be faster than sed in just filtering out the '^077'-s. – Leonid Nov 27 '12 at 13:44
Thats great, i will run this script and see how it performs. Thank you Stephane and Leonid for your time and information :) – promicin Nov 27 '12 at 15:30

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.