1

I want to split a csv, but ignore any commas within matching curly brace groups, and loop through each of the list members. The code below works great but does not consider commas within curly brace groups.

Assumptions:

  • There will always be matching pairs of curly braces. That is, input such as {{ {a,b,c}, x will not occur.

Expected Output:

Word='{0,1}'
Word='alpha'
Word='{(x,y,z)}'
Word='{{1,2,3}, {a,b,c}}'

References:

Code:

#!/bin/bash

#TEST_STRING="alpha, beta, gamma" ## <--- works great for simple case
TEST_STRING="{0,1}, alpha, {(x,y,z)}, {{1,2,3}, {a,b,c}}"

echo "${TEST_STRING}" | sed -n 1'p' | tr ',' '\n' | while read Extracted_Word; do
    printf "Word='%s'\n" "${Extracted_Word}"
done

I attemped to adapt 123's (now deleted) solution:

#!/bin/bash

#TEST_STRING="alpha, beta, gamma" ## <--- works great for simple case
TEST_STRING="{0,1}, alpha, {(x,y,z)}, {{1,2,3}, {a,b,c}}"

echo "${TEST_STRING}" \
    | sed -n 1'p' \
    | sed 's/\({[^}]*\({[^}]*}[^}]*\)*} *\)\(,\|$\) */\1\n/g;:1;s/\(\n[^{}]*\), */\1\n/;t1' \
    | tr ',' '\n' \
    | while read Extracted_Word; do
    printf "Word='%s'\n" "${Extracted_Word}"
done

but this produces the following error message for me:

./testcsv.sh
sed: 1: "s/\({[^}]*\({[^}]*}[^}] ...": bad flag in substitute command: ':'
./testcsv.sh: line 18: {{ {a,b,c}, x: command not found
  • Judging by your test string: although you don't show expected output, it appears that you want to match nesting of braces. That's beyond the capabilities of regular expressions, and so you need the capabilities of a context-free grammar; it's likely that sed is the wrong tool for the job. – Toby Speight Feb 2 '16 at 9:39
  • @TobySpeight: Expected output shown. I thought sed could do matching brace patterns? What would you suggest I use for this job? – Peter Grill Feb 2 '16 at 9:43
  • "I attemped to adapt 123's solution" - looks like you line-folded the sed script without adding ; where the newlines were: sed 's/\({[^}]*\({[^}]*}[^}]*\)*} *\)\(,\|$\) */\1\n/g;:1;s/\(\n[^{}]*\), */\1\n/;t1' . (Actually, given that you're on MacOS, you might need newlines or separate -e expressions) BTW, you could combine the two sed commands by adding 1q to the end of the second one, to finish after the first line. And drop the tr, since that's done within the sed command. – Toby Speight Feb 2 '16 at 9:43
  • If you you have a fixed, small limit on the depth of nesting, you should be okay with regular expressions - and hence with sed. For arbitrary nesting, you'll need something more, but I'm still thinking what that might be... – Toby Speight Feb 2 '16 at 9:45
  • @TobySpeight: As I don't have the rep on this SE site to see the deleted answer, I tried copying your code in the comment (updated question to show that), but that did not produce the desired output (which was shown in the deleted answer). Also, for my purposes, I am pretty sure I won't need a nesting depth more than what I have shown in the MWE. – Peter Grill Feb 2 '16 at 9:48
3

Try a pure bash

#!/bin/bash
TEST_STRING="{0,1}, alpha, {(x,y,z)}, {{1,2,3}, {a,b,c}}"
TEST_STRING="$TEST_STRING"","
count=0
newword=''
while [ "${TEST_STRING::1}" ] ; do 
    l="${TEST_STRING::1}"
    TEST_STRING=${TEST_STRING:1}
    [ "$l" = '{' ] && ((count++))
    [ "$l" = '}' ] && ((count--))
    if [ "$l" = ',' ] && ! ((count)) ; then
        echo "Word='$newword'"
        newword=''
    else
        if [ "$newword" ] || [ "$l" != " " ] ; then
            newword="$newword""$l"
        fi
    fi
done
  • This breaks up the last one {{1,2,3}, {a,b,c}} – 123 Feb 2 '16 at 8:56
  • @123 You are right! Have to think. – Costas Feb 2 '16 at 9:03
  • @123: Hey, what happened to your answer? I was trying to adapt it and an into difficulty. See updated question. – Peter Grill Feb 2 '16 at 9:35
  • @Costas: Unforutantely, there isn't a character that I can guarantee that won't be in there. But, I do know that if there is a # character that it will only appear as \#. – Peter Grill Feb 2 '16 at 9:36
  • @PeterGrill Deleted it, it didn't work with more embedded brackets. Also you need to put ; where the newlines were if you are putting the command on a single line. – 123 Feb 2 '16 at 9:37
2

Here's a sed script that will split your example:

#!/bin/sed -Ef

# replace all commas with newlines
s/,/\
/g

# Do we need to re-join any lines?
:loop
# Unmatched brace containing possibly another (matched) level of
# braces:
s/(\{([^{}]|\{[^{}]*\})*)\
/\1,/
tloop

# remove any leading space
s/\n */\
/g

# At first line, print result, then exit.
1q

Caveat: it will only handle two levels of braces (as per the comments to the question).

Tests:

$ ./259252.sed <<<'{0,1}, alpha, {(x,y,z)}, {{1,2,3}, {a,b,c}}'
{0,1}
alpha
{(x,y,z)}
{{1,2,3}, {a,b,c}}

And to show that it exits after the first line is processed:

$ ./259252.sed <<<$'a,b,c\nd,e,f'
a
b
c

I'm running this on Linux, and using the answers to Differences between sed on Mac OSX and other “standard” sed? to port it to MacOS. If that doesn't work, then this answer suggests you can install GNU sed with brew install gnu-sed, then use gsed instead of sed to invoke it.

In use:

#!/bin/bash

TEST_STRING="{0,1}, alpha, {(x,y,z)}, {{1,2,3}, {a,b,c}}"

echo "${TEST_STRING}" | sed -E -f 259252.sed | while read Extracted_Word; do
    printf "Word='%s'\n" "${Extracted_Word}"
done

which gives:

Word='{0,1}'
Word='alpha'
Word='{(x,y,z)}'
Word='{{1,2,3}, {a,b,c}}'
  • Nice answer ᠌ ᠌ ᠌ – 123 Feb 2 '16 at 10:19
  • That looks promising, but can't figure out how to get it to work in my script so that I can loop through each of the entires. I also tried to run it as a standalone script and – Peter Grill Feb 2 '16 at 10:19
  • @TobySpeight: The output I get is Word='{0n1}n alphan {(xnynz)}n {{1n2n3}n {anbnc}}'. I think the extra n in the output are the clue... – Peter Grill Feb 2 '16 at 10:48
  • Does your sed not interpret \n as newline? Check sed -e 's/m/\n/g' <<<'...m...' - you should get two rows of dots. – Toby Speight Feb 2 '16 at 11:15
  • Not sure what the three <<< mean. Three <<< gives me Missing name for redirect.. With echo "...m..." | sed -e 's/m/\n/g' I get a single line: ...n.... – Peter Grill Feb 2 '16 at 11:24
1
str='{0,1},alpha,{(x,y,z)},{{1,2,3},{a,b,c}}'
OPTIND=1 l=0 r=0; set ""
while   getopts : na -"$str"
do      [ "$l" -gt "$r" ]
        case    $?$OPTARG  in
        (1,)  ! l=0 r=0    ;;
        (0})    r=$((r+1)) ;;
        (?{)    l=$((l+1)) ;;
        esac    &&
        set -- "$@$OPTARG" ||
        set -- "$@" ""
done;   printf  %s\\n "$@"

dash has a bug which requires something like:

set -- "$@" ""; str=${str#?}

...but aside from that the above should be pretty quick as these things go, and work in basically any POSIX shell, besides being pretty simple all around. It should also handle mismatched pairs (even if you don't need it) by neglecting to specially recognize a } that occurs before a leading {.


{0,1}
alpha
{(x,y,z)}
{{1,2,3},{a,b,c}}

To get your prefixed string and the surrounding quotes you can substitute the following...

printf "Word='%s'\n" "$@"

...for the printf %s\\n "$@" used above. Given the example value of $str here it would print:

Word='{0,1}'
Word='alpha'
Word='{(x,y,z)}'
Word='{{1,2,3},{a,b,c}}'

More robustly you might do...

for W do alias "Word=$W" Word; done

...which would render...

Word='{0,1}'
Word=alpha
Word='{(x,y,z)}'
Word='{{1,2,3},{a,b,c}}'

...quoted as necessary, and would correctly quote embedded hard-quotes as well (though, if using a bash, you might want to do set --posix first).

And so, for the sake of demonstration...

str="{0,1

}}, {,}alph}'a, {(x,y,z)}, {{1,2,3}, {a,b,c}}" 
OPTIND=1 l=0 r=0; set ""
while   getopts : na -"$str"
do      [ "$l" -gt "$r" ]
        case    $?$OPTARG  in
        (1,)  ! l=0 r=0    ;;
        (0})    r=$((r+1)) ;;
        (?{)    l=$((l+1)) ;;
        esac    &&
        set -- "$@$OPTARG" ||
        set -- "$@" ""
done;   for W do alias "Word=${W# }" Word
done

Word='{0,1

}}'
Word='{,}alph}'\''a'
Word='{(x,y,z)}'
Word='{{1,2,3}, {a,b,c}}'

...where even the leading spaces are handled quite simply...

  • This doesn't seem to work for me on MacOS 10.9.5. I just get the original string {0,1} alpha {(x,y,z)} {{1,2,3},{a,b,c}} printed. To bad as I was hoping that this might actually ignore escaped curly braces, `{, (not necessarily matching). – Peter Grill Feb 5 '16 at 8:41
  • @PeterGrill - what is your shell? is it bash 2 or whatever Apple ships? I think it also ships zsh, where it does work for me. It also works in bash 4 at least, and mksh and ksh93. bash is just a bad shell, though it is marginally better these days than it was in the GPL 2 days. It uses POSIX syntax though, and so if it doesn't work on your system then it is because your system does not meet minimum unix standards. – mikeserv Feb 5 '16 at 8:44
  • @PeterGrill - but nowhere did I indicate that it would ignore escaped curly braces - though you could do that easily enough with a very little extra logic - at least to handle backslash escapes would be very simple and would require only one or two more case patterns in the test there. – mikeserv Feb 5 '16 at 8:47
  • Am on 3.2.53. I guess I really should update to 4.x bash... – Peter Grill Feb 5 '16 at 11:51
0

An additional bash solution:

  • It will handle unmatched pairs of braces {.
  • Won't accept a closing brace until one or more opening braces appear.
  • Will reset the count of braces to 0 at the end of the line.
  • Will accept a comma as valid after more closing braces than opening braces.
  • Will remove one space in front of solution.
  • Will quote the resulting word.

Code:

str="}}{0,1}}, {,}alph}'a"

            fin='false' d='0'
until  $fin
do     IFS=   read -r -d '' -n 1 a || fin='true'
       if     [[ $a == '{' ]] ; then (( d++ )) ; fi ### count openning braces.
       if     [[ $a == ',' ]] && (( d<1 )) || $fin  ### ',' out of braces or end.
       then   $fin && s="${s%$'\n'}"                ### removing a last newline.
              set -- "$@" "$s"                      ### store in an array.
              unset a s d                           ### unset working variables.
       fi
       if [[ $a == '}' ]] && ((d>0)); then ((d--)); fi  ### close braces.
       s="$s$a"
done <<<"$str"
printf 'Word=%q\n' "${@# }"       ### print a quoted value removing front space.

Output:

Word=\}\}\{0\,1\}\}
Word=\{\,\}alph\}\'a

Or somewhat more cryptic:

str="{0,1

}}, {,}alph}'a, {(x,y,z)}, {{1,2,3}, {a,b,c}}"

        fin='false' d='0'
until  $fin
do     IFS=   read -r -d '' -n 1 a || fin='true'
       [[ $a == '{' ]] && (( d++ ))                 ### count openning braces.
       [[ $a == ',' ]] && (( d<1 )) || $fin && {    ### ',' no braces (or end).
              $fin && s="${s%$'\n'}"                ### removing a last newline.
              set -- "$@" "$s"                      ### store in an array.
              unset a s d                           ### unset working variables.
       }
       [[ $a == '}' ]] && (( d>0 )) && ((d--))      ### substract closing braces.
       s="$s$a"
done <<<"$str"
printf 'Word=%q\n' "${@# }"    ### print a quoted value with front space removed.

Result:

Word=$'{0,1\n\n}}'
Word=\{\,\}alph\}\'a
Word=\{\(x\,y\,z\)\}
Word=\{\{1\,2\,3\}\,\ \{a\,b\,c\}\}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.