I have this line in my file:

select concat_ws('|',column1,'${hiveconf:RUNDATE}'), '${hiveconf:RUNDATE}' as date Group by concat_ws('|',column,'${hiveconf:RUNDATE}')

I need to replace '${hiveconf:RUNDATE}' with string('${hiveconf:RUNDATE}') but only when it is within concat_ws.

So I have 2 concat_ws in the same line, therefore if I give g in sed, it replaces all the RUNDATE and if I remove g, it replaces only the first occurrence.

After changing, my line should look like:

select concat_ws('|',column1,string('${hiveconf:RUNDATE}')), '${hiveconf:RUNDATE}' as date Group by concat_ws('|',column,string('${hiveconf:RUNDATE}'))

So the various patterns i have gathered till now are : concat_ws('|',column1,coalesce(to_date(timestamp),"") concat_ws('|',columnA,to_date(a.timestamp),'') concat_ws('|',column1,max(to_date(timestamp)),substr(id,1,9)) concat_ws('|',column,timestamp_d,'01.03.02',column2).

Need to have the timestamp column cast as string. The output file would look like: concat_ws('|',column1,coalesce(string(to_date(timestamp)),"") concat_ws('|',columnA,string(to_date(a.timestamp)),'') concat_ws('|',column1,string(max(to_date(timestamp))),substr(id,1,9)) concat_ws('|',column,string(timestamp_d),'01.03.02',column2).

1 Answer 1


If your concat_ws(...) is never nested, that is if there is never something like concat_ws(... foo() ...), so there is never a second set of parentheses within the concat_ws(), then you can match all occurrences of concat_ws(, followed by non-( until a ${hiveconf:RUNDATE} and replace only those. Like this:

$ sed "s/\(concat_ws([^)]*\)\('\${hiveconf:RUNDATE}\)/\1string(\2/g" file
select concat_ws('|',column1,string('${hiveconf:RUNDATE}'), '${hiveconf:RUNDATE}' as date Group by concat_ws('|',column,string('${hiveconf:RUNDATE}')
  • Thank you, but cannot tell for sure that concat_ws(..) will not be nested. Wanted to see an approach where brackets are not considered because it is too dynamic.
    – Jin
    Oct 23, 2020 at 11:11
  • @Jin I fear that would require a full-blown script where you keep track of how many ( you've seen and how many ) you've seen. I can't think of any other way of knowing if you are within the concat_ws(..) block :/
    – terdon
    Oct 23, 2020 at 14:51
  • oh ok...thank you for looking into this though.. because of the sheer size of files I need to make this change in and the different patterns which i am not able to generalize, I wanted to try this..to replace once after every pattern match, pattern being concat_ws....Will keep looking...Thank you :) @ terdon♦
    – Jin
    Oct 23, 2020 at 15:06
  • @Jin what would the pattern be though? I mean, you can use something like perl which understands regex lookarounds, but you have one case of concat_ws('|',column1,'${hiveconf:RUNDATE}') and another of concat_ws('|',column,'${hiveconf:RUNDATE}'). What can we rely on? What can we know won't change? Will '${hiveconf:RUNDATE}' always be immediately after the second comma of concat_ws(? We would need to see more of your file to come up wit any clever solutions.
    – terdon
    Oct 23, 2020 at 15:09
  • i found the pattern to substitute if the column ( i.e. '${hiveconf:RUNDATE}') I need to cast as string is right next to concat_ws('|',. But the problem occurred when the column was not right at the beginning.
    – Jin
    Oct 23, 2020 at 15:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .