I have an 11GB word-list file which is already sorted as each word is on its own line.

I need to remove duplicates and lines starting from 077.

I guess I need to run sed and sort -u together but I also want a live output display what's happening in terminal and if possible display the time left.

All of this in one command and it must be able to run optimally at full performance under a Live CD or possibly installed Backtrack 5 rc3.

Time is not very important but if there is a way for me to calculate the ETA, I might be able to borrow my dad's i7 based CPU which should process it faster obviously otherwise I'll have to use an older core 2 CPU.

The problem I'm facing with sort command is that under a VMware player running it live, it doesn't have enough space so I have to specify temp files on my 32GB USB by using the -T command. I guess this won't be a problem had I installed Linux.

So please give me the complete command, be it sed,sort,awk to do this (whichever is most optimal).

  • 2
    You might want to include a sample of the file, so people can see what you are dealing with. And any approaches you have tried yourself. – jasonwryan Nov 27 '12 at 0:59
  • @jasonwryan umm how do i give a sample of 11gb file lol. Anyways just imagine the file contains 1 password on each line like a list. You would be familiar if you ever used a password dictionary file. Also i know how to run those command individually but they take a lot of time since its a huge file, i just want simply it and get the additional options listed in one command. – promicin Nov 27 '12 at 3:00

Use pv for progress. You don't need a tempfile since you're only removing text. Just overwrite the file in place. If the file is already sorted, you don't need sort -u, just uniq.

pv file | {
  uniq | grep -v '^077'
  perl -e 'truncate STDOUT, tell STDOUT'
} 1<> file

The perl line is to truncate the file at the point were grep finished to write.

Note that since you're writing the file in-place, if you make a mistake, you won't be able to go back.

  • do i copy this whole script and paste into terminal after changing the input/output file ? I also need to install pv for this to work. Is this script any better then what Leonid has suggested below ? Because i rather not sacrifice performance. – promicin Nov 27 '12 at 11:24
  • Yes. Mine is a simplified version of Leonid's (uniq < infile | greo -v '^077' > outfile) with the addition of the progress bar (pv), and inplace editing (so you don't need extra free space). – Stéphane Chazelas Nov 27 '12 at 12:00

I'm not sure you can have ETA or any progress display without significantly increasing the complexity of the sed/awk/shell script (and slowing the whole thing as a result). If you just want it to be as fast as possible, just try cat source_file | uniq | sed -n -e '/^077/!p' > dest_file. For an approximation of the progress display, you might want to watch the growing size of the dest_file with that command running in background or in another terminal.

  • Thanks, this one looks pretty straight forward, its running uniq and then piping into sed to remove numbers starting from 077 and saving the output in the destination file. I have also heard of awk with this command: awk '!x[$0]++' is it any good compared to uniq ? – promicin Nov 27 '12 at 11:19
  • If the file is already sorted, you won't be able to beat uniq. That awk command will end up storing all unique lines in memory which for so big a file is probably not a good idea and has no benefit otherwise. That sed line is a convoluted (double-negation) way to write sed '/^077/d' though you don't need sed here as grep is more than enough. You don't need cat either (UUOC). – Stéphane Chazelas Nov 27 '12 at 11:57
  • I completely agree with Stephane: just like it's hard to beat uniq in doing its specific job, grep shall be faster than sed in just filtering out the '^077'-s. – Leonid Nov 27 '12 at 13:44
  • Thats great, i will run this script and see how it performs. Thank you Stephane and Leonid for your time and information :) – promicin Nov 27 '12 at 15:30
awk '!a[$0]++' "filename" > /tmp/dup
mv -f /tmp/dup "filename"
  • please include a short description of what your commands do. It looks like you do not take the requirement '077' into account. Why is filename quoted? – Sebastian Dec 24 '17 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.