I've searched around the internet and stackexchange for this. Even though there are lots of similar topics, I haven't found a solution yet.
So, I have a quite large list (approx. 20GB), which contains around 5% duplicate lines. I want to filter this list, sothat one of the duplicates is deleted. Example:
test123 Test123 test test123 test 123
test123 Test123 test test 123
Whether the list gets sorted or not doesn't matter.
sort -u -o output.txt, also
sort -us -o output.txt. It works fine for smaller files, but when I try to do files of more than approx. 4GB, the resulting file is suspiciously small and, instead of a .txt file, it has apparently become an "emacs-lisp-source-text".
I'd be very grateful if someone could help me out!