As you can see in the picture attached screen shot
After I used the sort command to sort a VERY large wordlist file I then split them up using the Split command. From there I confirmed that the listed words are in a-z order.
I then ran the sort -u
command and noticed that it was not removing unique words. (It was removing some as I could see the files were a little smaller, but not all.)
What am I doing wrong?
Overall goal: my over all goal is to take all my wordlists and put them into one large file (25gig), then sort and remove any unique words (cutting this by 40% or so) and then split up the files into manageable sizes. No Windows programs or Linux commands worked.