1

As you can see in the picture attached screen shot

After I used the sort command to sort a VERY large wordlist file I then split them up using the Split command. From there I confirmed that the listed words are in a-z order.

I then ran the sort -u command and noticed that it was not removing unique words. (It was removing some as I could see the files were a little smaller, but not all.)

What am I doing wrong?

Overall goal: my over all goal is to take all my wordlists and put them into one large file (25gig), then sort and remove any unique words (cutting this by 40% or so) and then split up the files into manageable sizes. No Windows programs or Linux commands worked.

Improve this question
2
  • 6
    Copy and paste output instead of posting screenshots. Text is much easier to deal with than images, especially when your question has nothing to do with graphics.
    – jw013
    Sep 20, 2012 at 20:52
  • 1
    Are you sure that there are no invisible characters at the end of the words? Like spaces.
    – Karlson
    Sep 20, 2012 at 20:57

1 Answer 1

Reset to default
4

sort -u removes unique lines. So, the a potential problem is that these three lines are not the same, and sort -u will leave all of them:

foo
foo 
foo 

No matter how closely you look, its hard to notice why. That is, unless you take a hex dump, with xxd for example:

0000000: 666f 6f0a 666f 6f20 0a66 6f6f e280 820a  foo.foo .foo....

0x0a is newline, if you're not familiar with hex dumps. So the three "foo"s are:

666f 6f         0a
666f 6f20       0a
666f 6fe2 8082  0a

Aha! That's actually foo, foo<SPACE> (the 0x20), and foo<EN-SPACE> (the 0xe28082, which is U+2002 encoded in UTF-8).

You've probably got something similar going on. You need to use a hex editor, or a text editor set to show invisible characters. For example, here is what it looks like in gvim with :set list. I've just typed in the ga command to see what the character under the cursor is, revealing that its U+2002. You can also see how the end of line ($) isn't where you expect on the two with spaces after them:

screenshot of gvim demonstrating :set list

Improve this answer
6
  • well this is what i thought too... so i loaded it in a text editor and saw nothing (checking so see if there were spaces). HOWEVER, i did NOT turn on show hidden characters. so heres the followup question.... is there any script that would delete these after the word? I dont know how to intelligently ask that question however you understand my overall goal. Any suggestion would be great. :( I really want to make this the most efficient wordlist for running through audits.
    – Alex
    Sep 20, 2012 at 21:35
  • @Alex I'm unsure from your comment, when you tried it with show hidden characters (or even better, a hex editor) did it become clear what is going on?
    – derobert
    Sep 20, 2012 at 21:37
  • ^ sorry, had to edit the comment. it submitted and cut me off
    – Alex
    Sep 20, 2012 at 21:39
  • @Alex I think you could do that fairly easily with sed or perl. For example, piping through perl -p -e 's/\s+$//'
    – derobert
    Sep 20, 2012 at 21:41
  • @Alex also, you'd ask that as "trimming whitespace from the end of each line" or similar. Also, as a final note, if you're doing matches against this huge wordlist, and need it to be efficient, there are algorithms for that. Probably a question for Stack Overflow.
    – derobert
    Sep 20, 2012 at 21:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.