0

I have a CSV file with tuples of values where I need to change occurrences of one into another in a different (large) file.

Thus far, I have done a while read line [...] < foo.csv, essentially running sed one time for each line in the CSV file.

This takes quite some amount of time, so I wondered if I should change the while loop to construct a very long string of multiple -e statements, and then running that with eval.

I could try, obviously, but if someone can tell me if sed will just, essentially, do the same that I have done thus far, namely running through the file for each -e statement, meaning that no performance gain is to be had, then I don't think I shall bother.

Edit after comments:

Basically, I do the following:

while read line
do
  old_user=echo $line | cut -d \; -f 2|tr -d \"
  new_user=echo $line | cut -d \; -f 4|tr -d \"

  if [ "$old_user" != "$new_user" ]
  then
    sed -i -e "s/^(.*ri:username=\")$old_user(\".*)$/\1$new_user\2/g" confluence/entities_converted.xml
  fi
done < usernames.csv

If you notice that it's an XML file, the reason is that there are a number of instances where XML parsing and rewriting is cumbersome, hence sed. I just wonder if, instead of running sed multiple times, I should construct multiple -e arguments to sed.

usernames.csv looks like

    "Full name";"Username";"Email";"New username"
    "Sune Mølgaard";"sune.molgaard";"foo@bar.baz";"smo"

There may be any number of lines along the way of the second line, hence the looping. I am aware that the first line probably won't match, but that is insignificant.

  • 1
    Please, tell us more about the change. How exactly do you want to change the input? – choroba Sep 8 '15 at 8:33
  • do not reply in comment, you should edit your post. – Archemar Sep 8 '15 at 8:35
  • Could you also post a sample of the input? – choroba Sep 8 '15 at 8:49
  • multiple -e, not while loop. – mikeserv Sep 8 '15 at 9:15
3

No need to eval or construct multiple -e's. Sed can read its "program" from a file or pipe which you can in fact generate in sed, too:

cut -f2,4 -d\; usernames.csv \
    | sed -e 's/^/s%ri:username=/' -e 's/;/%ri:username=/' -e 's/$/%/' \
    | sed -i~ -f- confluence/entities_converted.xml

To check the generated program, remove the last line.

If you want to skip the lines where no change is needed (might speed it up), remove them by inserting grep between the seds:

   | grep -v '"\(.*\)".*"\1"' \
  • Hmmm, first two lines seem to generate lines a la s%ri:username="sune.molgaard"%ri:username="smo"% Is that some other sed syntax that I don't know of, as opposed to s/foo/bar/? – Sune Mølgaard Sep 8 '15 at 9:20
  • clever move, keeping quotes !! ( s%foo%bar% is equivalent to s/foo/bar/ or syfooybary ) – Archemar Sep 8 '15 at 9:22
  • @SuneMølgaard: You can use any character with substitution, not just /. Here, it avoids backslashing. – choroba Sep 8 '15 at 9:24
  • Almost 8x times faster than my naïve approach - thank you very much :-) – Sune Mølgaard Sep 8 '15 at 9:46
0

you sould use awk to parse usernames.csv (where field 2 and 4 are different), and generate sed file.

 tr -d \" username.csv |
 awk -F\; '$2 != $4 { printf "s/^(.*ri:username=%c)%s(%c.*)$/\\1%s\\2/g\n",34,$2,34,$4 ; }' |
 sed -i -f - confluence/entities_converted.xml

some trick

  • use printf "..%c..",34 to generate quotes.
  • you can skip the sed line in debug part to make sure all sed instructions are properly generated.
  • do you need /g in substitution ?

on my test file

;foo;;foo;;
;fubar;;mr X;;
;bar;;bistro;;
    "Full name";"Username";"Email";"New username"
    "Sune Mølgaard";"sune.molgaard";"foo@bar.baz";"smo"

this generate

s/^(.*ri:username=")fubar(".*)$/\1mr X\2/g
s/^(.*ri:username=")bar(".*)$/\1bistro\2/g
s/^(.*ri:username=")Username(".*)$/\1New username\2/g
s/^(.*ri:username=")sune.molgaard(".*)$/\1smo\2/g

don't bother to remove Username's line, if not found, no substitute.

  • For some reason, I had to change tr -d \" usernames.csv to cat usernames.csv | tr -d \", but then it seems to generate correct lines to pipe further into sed - will time a run shortly to see if it improves speed - thank you! Also, I do believe that I need /g, since multiple occurrences a possible, or does /g just pertain to multiple occrrences per line? – Sune Mølgaard Sep 8 '15 at 9:17
  • I went with choroba's approach, but thank you very much! – Sune Mølgaard Sep 8 '15 at 9:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.