I have this line in my file:

select concat_ws('|',column1,'${hiveconf:RUNDATE}'), '${hiveconf:RUNDATE}' as date Group by concat_ws('|',column,'${hiveconf:RUNDATE}')

I need to replace '${hiveconf:RUNDATE}' with string('${hiveconf:RUNDATE}') but only when it is within concat_ws.

So I have 2 concat_ws in the same line, therefore if I give g in sed, it replaces all the RUNDATE and if I remove g, it replaces only the first occurrence.

After changing, my line should look like:

select concat_ws('|',column1,string('${hiveconf:RUNDATE}')), '${hiveconf:RUNDATE}' as date Group by concat_ws('|',column,string('${hiveconf:RUNDATE}'))

So the various patterns i have gathered till now are : concat_ws('|',column1,coalesce(to_date(timestamp),"") concat_ws('|',columnA,to_date(a.timestamp),'') concat_ws('|',column1,max(to_date(timestamp)),substr(id,1,9)) concat_ws('|',column,timestamp_d,'01.03.02',column2).

Need to have the timestamp column cast as string. The output file would look like: concat_ws('|',column1,coalesce(string(to_date(timestamp)),"") concat_ws('|',columnA,string(to_date(a.timestamp)),'') concat_ws('|',column1,string(max(to_date(timestamp))),substr(id,1,9)) concat_ws('|',column,string(timestamp_d),'01.03.02',column2).

1 Answer 1


If your concat_ws(...) is never nested, that is if there is never something like concat_ws(... foo() ...), so there is never a second set of parentheses within the concat_ws(), then you can match all occurrences of concat_ws(, followed by non-( until a ${hiveconf:RUNDATE} and replace only those. Like this:

$ sed "s/\(concat_ws([^)]*\)\('\${hiveconf:RUNDATE}\)/\1string(\2/g" file
select concat_ws('|',column1,string('${hiveconf:RUNDATE}'), '${hiveconf:RUNDATE}' as date Group by concat_ws('|',column,string('${hiveconf:RUNDATE}')
  • Thank you, but cannot tell for sure that concat_ws(..) will not be nested. Wanted to see an approach where brackets are not considered because it is too dynamic.
    – Jin
    Oct 23, 2020 at 11:11
  • @Jin I fear that would require a full-blown script where you keep track of how many ( you've seen and how many ) you've seen. I can't think of any other way of knowing if you are within the concat_ws(..) block :/
    – terdon
    Oct 23, 2020 at 14:51
  • oh ok...thank you for looking into this though.. because of the sheer size of files I need to make this change in and the different patterns which i am not able to generalize, I wanted to try this..to replace once after every pattern match, pattern being concat_ws....Will keep looking...Thank you :) @ terdon♦
    – Jin
    Oct 23, 2020 at 15:06
  • @Jin what would the pattern be though? I mean, you can use something like perl which understands regex lookarounds, but you have one case of concat_ws('|',column1,'${hiveconf:RUNDATE}') and another of concat_ws('|',column,'${hiveconf:RUNDATE}'). What can we rely on? What can we know won't change? Will '${hiveconf:RUNDATE}' always be immediately after the second comma of concat_ws(? We would need to see more of your file to come up wit any clever solutions.
    – terdon
    Oct 23, 2020 at 15:09
  • i found the pattern to substitute if the column ( i.e. '${hiveconf:RUNDATE}') I need to cast as string is right next to concat_ws('|',. But the problem occurred when the column was not right at the beginning.
    – Jin
    Oct 23, 2020 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.