1

I have a variable in bash holding a string "<topic>...<topic>"representing topics that need to be mapped into a database table name, replacing illegal characters, via a mapping configuration:

The required mapping format is "topic1:table1,topic2:table2" – that is, this is the output I need.

For context, this is a configuration entry for Snowflake Kafka Connector which helps streaming data from a topic into a table, and importantly, table names are much more restricted in the characters allowed.

In this most simple case, the illegal character is a hyphen which should be converted to an underscore.

For example, for "foo-bar,boo-baz" as the input string, the required answer would be:

"foo-bar:foo_bar,boo-baz:boo_baz".

In Python, this is straight-forward:

import sys
s = sys.argv[1]
print(','.join(p + ':' + p.replace('-', '_') for p in s.split(',')))

I'm looking for a solution based on shell scripting tools to avoid installing additional software.

I know that sed for example has labels which I think can help with this, but I haven't been able to work out a solution.

8
  • 2
    We encourage questioners to show what they have tried so far to solve the problem themselves.
    – Cyrus
    Commented Jun 25, 2023 at 14:20
  • @Cyrus good point – I have been looking at sed's label and not really been able to work out anything. I added a Python solution which is what I'm using right now, but ideally I would be able to solve this without adding additional tools to the mix (trying to keep a container image within a reasonable size).
    – malthe
    Commented Jun 25, 2023 at 17:22
  • Shells, and many shell tools (e.g. grep) do not handle binary data well. Other languages, e.g. perl has features that make binary data handling easy. Start with man perl, and there's a book "Programming Perl".
    – waltinator
    Commented Jun 25, 2023 at 17:33
  • 1
    @malthe show us an example. Is the string in question within a file or a database? Is it a single string or multiple lines? .. Commented Jun 25, 2023 at 20:17
  • 1
    Does it also need to include mapping for topics that don't include the character you want to replace?
    – aviro
    Commented Jun 26, 2023 at 8:25

3 Answers 3

1

Assuming you have the input in a variable called TOPICS:

Using sed:

sed 's/[^,]\+/\0:\0/g; :a s/:\([^-,]\+\)-/:\1_/g; ta' <<<"$TOPICS"
  1. s/[^,]\+/\0:\0/g - Take each topic (a word that doesn't contain a comma), and add the same word after it with a colon. The result after this part would be: foo-bar:foo-bar,boo-baz:boo-baz
  2. :a s/:\([^-,]\+\)-/:\1_/g - replace the hyphen after every colon with underscore.
  • At this point it would only replace the first hyphen within each topic, so for instance if one of the topics is foo-bar-baz, and after the first step with have foo-bar-baz:foo-bar-baz, after this step the result would be foo-bar-baz:foo_bar-baz.
  1. ta - if the latest substitution was successful (a hyphen was replaced with an underscore) - go back to the :a label to check if there are more substitutions to be made. This is in case there's a topic that has more than one hyphen. If no substitution was made in step 2, don't branch back to the label - continue to the next line.
  • For instance: foo-bar-baz:foo_bar-baz from the previous example will now become foo-bar-baz:foo_bar_baz.

Using awk:

awk 'BEGIN {ORS=RS=","} { if (gsub( /\n$/, "" )) ORS="\n"; NEW=$0; gsub("-", "_", NEW); print $0":"NEW}' <<<"$TOPICS"
  1. RS (Input Record Separator) and ORS (Output Record Separator) are set to ,. That way awk will treat every topic as a separate line.
  2. if (gsub( /\n$/, "" )) ORS="\n" - If the last character of the word is \n (a new line), remove it. gsub will return the number of replacements (1), and then on the last word it will not print the new line after the read word, but only as the last (Output Record Separator).
  3. gsub("-", "_", NEW) - replace hyphens with underscores.

Using awk + sed + tr:

We have additional commands here, but it might be a bit easier to read:

echo "$TOPICS" \
 | tr ',' '\n' \
 | awk '{NEW=$0; gsub("-", "_", NEW); print $0":"NEW}' \
 | tr '\n' ',' \
 | sed 's/,$/\n/'

  1. tr ',' '\n' - First separate the words with a new line.
  2. awk to print the mapping.
  3. tr '\n' ',' - Replace newlines back with commas.
  4. sed 's/,$/\n/' - Except for the last comma, which should be replaced with a newline.
1

Assuming you want to manipulate the string only, one way of achieving this would be as follows:

#!/usr/bin/env bash

p="foo-bar,boo-baz"
IFS=',' read -ra arr <<< "$p"
result=()

for item in "${arr[@]}"; do
  result+=("${item}:${item//-/_}")
done

end_result=$(printf '%s,' "${result[@]}")
echo "${end_result%,*}"

If you want to know about string manipulation , see https://mywiki.wooledge.org/BashFAQ/100

5
  • Nitpick (I can't make such a small edit), the two colons appearing in the last two lines should be commas.
    – malthe
    Commented Jun 26, 2023 at 7:02
  • @malthe does the above not satisfy your requirement? It is a variable which holds just that string and then some string manipulation. Is there anything not covered in the snippet? Commented Jun 26, 2023 at 8:09
  • I think this is a good answer and it certainly satisfies the requirement. It's a little verbose though and I was kind of fishing for an answer that's essentially a one-liner.
    – malthe
    Commented Jun 26, 2023 at 8:13
  • Yes, the mapping are supposed to be separated by commas. The colons represent the mapping themselves. In your case you're using colons for both the mapping and the separation. Your result should be foo-bar:foo_bar,boo-baz:boo_baz and not foo-bar:foo_bar:boo-baz:boo_baz.
    – aviro
    Commented Jun 26, 2023 at 8:17
  • Oh that is sharp aviro. I will update the answer later on today as I don't have access to a compurer right now. Thanks for pointing it out Commented Jun 26, 2023 at 9:08
1

Using any awk in any shell on every Unix box:

$ echo 'foo-bar,boo-baz' |
    awk -F',' '{for (i=1; i<=NF; i++) {t=$i; gsub(/-/,"_",t); printf "%s:%s%s", $i, t, (i<NF ? FS : ORS)}}'
foo-bar:foo_bar,boo-baz:boo_baz

or if you prefer:

$ echo 'foo-bar,boo-baz' |
    awk -v RS=',' '{t=$1; gsub(/-/,"_",t); printf "%s:%s%s", $1, t, (sub(/\n$/,"") ? ORS : RS)}'
foo-bar:foo_bar,boo-baz:boo_baz

The second one will fail if the input doesn't end in a newline (which it should to be a valid text file per POSIX).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .