I have a file of 10GB data, I want to execute 2048 substitution expressions of sed -i 's/pattern/replace_pattern/g' on that file.

Is there a way to do it faster and effectively.

Currently, I am launching sed or perl in sequential order.

sed -i  "s/VSS:F31128015/VSS_F31128015/g" file.data
sed -i  "s/VSS:F31127969/VSS_F31127969/g" file.data
perl -pi -e  's/VSS\:F31128015/VSS_F31128015/g' file.data
perl -pi -e  's/VSS\:F31127969/VSS_F31127969/g' file.data

I did notice very less difference in speed of execution by perl or sed Is there a better way to do this? I have almost 2048 expressions of replacement.

will opening 10GB file via Tcl/tk and using regexp be a better idea?

Should I use lex?

  • Maybe you can compact these patterns somehow. s/VSS:/VSS_/g is the extreme example, it may match too many fragments though. Something with [:digit:]? It strongly depends on your data and the patterns. – Kamil Maciorowski Apr 17 at 7:03
  • I have a list of the expressions on which I need to perform substitution, giving s/VSS:/VSS_/g will result in the greedy fashion. – JigarGandhi Apr 17 at 7:04
  • 1
    Build a file so you can sed -f. In what format are these "2048 substitution expressions". Are they in a file. How does the file look like? Maybe all you need is sed -f subst data. – Kamil Maciorowski Apr 17 at 7:15
  • I have a file with the list of all 2048 sed or perl expressions. – JigarGandhi Apr 17 at 9:11
  • Are you wanting to hack an RC extracted netlist? – Rakesh Sharma Apr 19 at 3:40

Some things that should work (untested)

Use a more general regex

sed -i -e 's/(VSS):(F[0-9]+)/\1_\2/g' file.data

Use more than one expression, but one sed invocation.

sed -i -e "s/VSS:F31128015/VSS_F31128015/g" -e "s/VSS:F31127969/VSS_F31127969/g" file.data
  • I cannot use the first suggestion as I want to avoid the greedy expression. I will use the second expression, Is using perl instead of sed advisable in this context? – JigarGandhi Apr 17 at 7:27
  • I have no idea perl vs sed. – ctrl-alt-delor Apr 17 at 7:28

perl is good for simple text replacements like this. Following up on your comment thread on ctrl-alt-delor's answer, there are ways to avoid greediness. In this case, are you wanting to match exactly 8 digits? If yes,

perl -i.bak -pe 's/(?<=VSS):(?=F\d{8})/_/' bigfile

This uses lookarounds to find the colon you want to change to underscore.

People often add the g flag even when there's only 1 match per line. Add it if necessary.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.