I have a big text file (roughly 2GB). I have a csv file that has the following fields:


My interest is: replace the content of the big file as follows. Whenever a string in the big file matches a "pattern" in my CSV (second field), it will replace that string by the corresponding "rowID" (first field).

This is what I have tried using sed, which is extremely slow (also due to in-place replacement of the file). But, is there any faster solution?

while read f1 f2 f3; 
    sed -i "s/$f2/$f1/g" bigfile; 
done < map.csv

Note that map.csv contains over 500000 rows.

  • please show some samples line of map.csv and expected output applied based on given given sample big.txt file Nov 23 '20 at 17:52

Note that sed replaces regular expressions not strings - I'm going to assume that is not an issue for your use case, since your current solution uses it.

One way would be to pre-process map.csv into a sequence of expressions to pass to a single invocation of sed -f:

awk -F, 'NR>1 {printf "s/%s/%s/g\n", $2, $1}' map.csv | sed -f - bigfile

(add -i only once you are satisfied that it is doing the right thing).

  • Isn't it too expensive? The CSV itself has >500000 rows. Sorry, I forgot to mention this in my question.
    – Coder
    Nov 23 '20 at 17:45
  • @Coder, this solution will process your big file once. Your while loop will process the big file N times (where N is the number of records in the map.csv). This is about as efficient as you can get. Nov 23 '20 at 17:50
  • @glennjackman the result of awk is a big string that goes into sed. This froze my computer for 5 hours. This is not feasible in my computer that is 4GB ram. Also, NR>=1.
    – Coder
    Nov 24 '20 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.