I have a huge (70GB), one line, text file and I want to replace a string (token) in it.
I want to replace the token
<unk>, with another dummy token (glove issue).
sed 's/<unk>/<raw_unk>/g' < corpus.txt > corpus.txt.new
but the output file
corpus.txt.new has zero-bytes!
I also tried using perl:
perl -pe 's/<unk>/<raw_unk>/g' < corpus.txt > corpus.txt.new
but I got an out of memory error.
For smaller files, both of the above commands work.
How can I replace a string is such a file? This is a related question, but none of the answers worked for me.
What about splitting the file in chunks of 10GBs (or whatever) each and applying
sed on each one of them and then merging them with
cat? Does that make sense? Is there a more elegant solution?