I have a big text file (>500GB), all the ways I can find (sed/tail and others) all require write the 500GB content to disk. Is there anyway to quickly remove the first a few lines in place without writing 500GB to disk?
You can use
sed to delete lines in place with the
$ cat foo.txt bar baz lorem $ sed -i '1d' foo.txt $ cat foo.txt baz lorem
You can also delete a range of lines; for example
sed -i '1,4d' foo.txt will remove lines 1-4.
EDIT: as don pointed out in the comments, the
-i option still creates a copy.
By using the tail command in that way:
# tail -n +<lines to skip> filename
tail -n +1000 hugefile.txt > hugefile-wo-the-first-1000-lines.txt
And that's all.- For more information: https://es.wikipedia.org/wiki/Tail
BTW: Don't be fooled if someone tell you this is exactly the opposite what you want to do, I've tested it:
$ tail -n +3 /tmp/test 3 4 5 $ cat /tmp/test 1 2 3 4 5