Tagged Questions

votes

1answer

1k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...

asked Aug 29 '11 at 2:40

greatwolf
1387

newest uniq linux questions feed

Technology			Life / Arts	Culture / Recreation	Science	Other
Stack Overflow Server Fault Super User Web Applications Ask Ubuntu Webmasters Game Development TeX - LaTeX	Programmers Unix & Linux Ask Different (Apple) WordPress Answers Geographic Information Systems Electrical Engineering Android Enthusiasts IT Security	Database Administrators Drupal Answers SharePoint User Experience Mathematica more (14)	Photography Science Fiction & Fantasy Seasoned Advice (cooking) Home Improvement more (13)	English Language & Usage Skeptics Mi Yodeya (Judaism) Travel Arqade (gaming) Bicycles Role-playing Games more (22)	Mathematics Cross Validated (stats) Theoretical Computer Science Physics more (7)	Stack Apps Meta Stack Overflow Area 51 Stack Overflow Careers

rev 2013.5.17.690

Linux is a registered trademark of Linus Torvalds. UNIX is a registered trademark of The Open Group.
This site is not affiliated with Linus Torvalds or The Open Group in any way.

Tagged Questions

How to remove duplicate lines in a large multi-GB textfile?

Related Tags