Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

As an example, this is the pronunciation of "when" according to Wiktionary. enPR, IPA and X-SAMPA are different schemes for showing pronunciations.

when:* {{a|US}} {{enPR|wĕn|hwĕn}}, {{IPA|/wɛn/|/ʍɛn/}}, {{X-SAMPA|/wEn/|/WEn/}}

I'd like to pull out the keyword when and its two IPA pronunciations, and put them on separate lines:

when wɛn
when ʍɛn

There may be one, two or more IPA pronunciations of a word, and there may or may not be enPR or X-SAMPA pronunciations.

I'm thinking PHP, lists within lists, but this seems to be overkill, and I don't want the users to have to install it if possible. Is there a way to do this in awk, sed, cut or other standard Unix command line utility?

share|improve this question

3 Answers 3

up vote 5 down vote accepted

With sed, you could write it as:

sed '/\([^:]*\):.*{IPA|\([^}]*\).*/!d;s//\1 \2/;s,/,,g;:1
     s/\(\([^ ]*\).*\)|/\1\n\2 /;t1'

Breakdown (by @slm, thanks)

The above command can be broken down as follows:

  1. Parse input into when: ... {IPA|...} and delete non-matching lines.

    In /pattern/!d; s//repl/

    We [d]iscard the lines that do not[!] match the pattern, and then reuse the same pattern in the next [s]substitution command (an empty pattern means reusing the last pattern). Instead of [d]eleting the non-matching lines, we could have left them untouched by using b instead of d, or if we know that all the lines match the pattern, we could use s/pattern/repl/ directly.

    /\([^:]*\):.*{IPA|\([^}]*\).*/
    

    That pattern splits the data up into 2 chunks. The first chunk is when:. This bit of code, \([^:]*\): says to take all the characters until you encounter a : and to save it in a temp. variable (\1).

    All the characters between the : up to and including the {IPA| are skipped. The next bit that's saved is everything after the IPA|. This is done by this block of code, \([^}]*\), which says to save all the code until a } is encountered. This is saved in variable (\2).

    NOTE: In sed any time you want to save a chunk of the string you can wrap it in parentheses. They need to be escaped with a \ so that sed knows you don't mean a literal paren. Like so: \( savethis \).

    example

    $ sed 's/\([^:]*\):.*{IPA|\([^}]*\).*/\1 \2/;' sample.txt
    when /wɛn/|/ʍɛn/
    
  2. Remove all forward slashes (/)

    This one looks more complicated because it's using an alternate separator. You'd typically use the form s///g, but sed let's you make up separators on the fly, so we're using commas instead (s,,,g). This block searches for / and replaces them with nothing.

    example

    $ sed '/\([^:]*\):.*{IPA|\([^}]*\).*/!d;s//\1 \2/;s,/,,g;' sample.txt
    when wɛn|ʍɛn
    
  3. Iterate through each IPA

     :1 s/\(\([^ ]*\).*\)|/\1\n\2 /;t1
    

    This is by far the most complicated component of this solution. It's hard to see what's going on but this block is a conditional branch.

     :label command(s) t label
    

    The label is :1 the command(s) are s/\(\([^ ]*\).*\)|/\1\n\2 /; and the t label is the "test" that sees if the previous command modified the pattern space. If so then jump to label 1, hence the t1.

  4. The command inside the loop

    If we take the label ... loop out for a sec, and increase our IPA example so that it has 3, you can see what's going on a little bit better.

    {{IPA|/wɛn/|/ʍɛn/|/blah/}}
    

    We'll end up with this, using the previous commands to this point.

    when wɛn|ʍɛn|blah
    

    If we now run this:

    $ echo "when wɛn|ʍɛn|blah" | sed 's/\(\([^ ]*\).*\)|/\1 \2 /;'
    

    We get this:

    when wɛn|ʍɛn
    when blah
    

    Can you see what it's doing now? Yeah me neither, so let's simplify a little more, and take the newline (\n) out and swap in some shorter strings.

    simpler example

    $ echo "X C1|C2|C3" | sed 's/\(\([^ ]*\).*\)|/\1 \2 /;'
    X C1|C2 X C3
    

    Now what's going on here is that the code \(\([^ ]*\).*\)| is clever in the sense that it's nesting the parens so that they're like this ( ( ) ). What's being matched on the inside parens is anything that's not a space. This get's the when string. The outer parens match everything up to the last pipe (|).

    The other interesting thing with this code snippet is that the parens are ordered so that the outer ones get stored in \1 while the inner ones are \2. This is because sed numbers them based on the order in which they're encountered.

    You can convince yourself of this by extending the snippet with additional \1's and \2's.

    $ echo "X C1|C2|C3" | sed 's/\(\([^ ]*\).*\)|/\1 \1 \1 /;'
    X C1|C2 X C1|C2 X C1|C2 C3
    
    $ echo "X C1|C2|C3" | sed 's/\(\([^ ]*\).*\)|/\1 \2 \2 /;'
    X C1|C2 X X C
    

    So the command inside the loop is basically taking the X 2 times. Once as part of the whole X C1|C2 (outside parens) and a second time as anything up to the space (inside parens).

  5. Back to the conditional branch

    OK so the branch is basically going to call the command in #5, for IPA's where there are more than 2. sed's branch construct will keep re-running the command until the command no longer modifies the substitution, at which point it stops.

    example

    $ echo "X C1|C2|C3" | sed ':1 s/\(\([^ ]*\).*\)|/\1\n\2 /; t1'
    X C1
    X C2
    X C3
    

Hopefully the above will help other passerby's to this answer in the future.

share|improve this answer
1  
Don't know how you figured this out, and can't make heads or tails of it, but it works! –  Yimin Rong Aug 6 '13 at 20:54
    
@YiminRong - yes Stephane's got some of the best answers on the site. He and Gilles. –  slm Aug 6 '13 at 21:03
    
@YiminRong - if you start to break it down you can see how it works. –  slm Aug 6 '13 at 21:06
    
Do you care if someone expands on your answers to explain how they work? –  slm Aug 6 '13 at 21:06
    
Why does this also work? sed '/\([^:]*\):.*{IPA|\([^}]*\).*/!d;s//\1 \2/;s,/,,g;s/\(\([^ ]*\).*\)|/\1\n\2 /; What role does :1 and t1 play? –  slm Aug 6 '13 at 21:12

With perl inside a perl script (processing STDIN)

while(<>) {
    if(/^([^:]+):.*{{IPA\|([^}]+)}}/) { 
        print "$1 $_\n" foreach(split /\|/, $2); 
    }
}

or on the command line (piping)

perl -ne ' if(/^([^:]+):.*{{IPA\|([^}]+)}}/) { print "$1 $_\n" foreach(split /\|/, $2); }'
share|improve this answer

With bash and grep

line='when:* {{a|US}} {{enPR|wĕn|hwĕn}}, {{IPA|/wɛn/|/ʍɛn/}}, {{X-SAMPA|/wEn/|/WEn/}}'
IFS=$': \t' read -ra words <<< "$line"
for item in "${words[@]}"; do
    if [[ $item == "{{IPA|"* ]]; then
        grep -o '/[^/]\+/' <<< "$item" | while read -r pronunc; do
             echo "${words[0]} ${pronunc//\//}"
        done
    fi
done
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.