Extracting tokens from a line of text

Question

Using bash scripting and grep/awk/sed, how can I split a line matching a known pattern with a single character delimiter into an array, e.g. convert token1;token2;token3;token4 into a[0] = token1 … a[3]=token4 ?

Answer 1

UPDATE Please note that making an array this way is suitable only when IFS is a single non-whitespace character and there are no multiple-consecutive delimiters in the data string.
For a way around this issue, and a similar solution, go to this Unix & Linux question ... (and it is worth the read just to get more of an insight into IFS.

---

Use bash (and other POSIX shells, e.g. ash, ksh, zsh)'s IFS (Internal Field Seperator).

Using IFS avoids an external call, and it simply allows for embeded spaces.

# ==============
  A='token0:token1:token2.y   token2.z '
  echo normal. $A
# Save IFS; Change IFS to ":" 
  SFI=$IFS; IFS=:     ##### This is the important bit part 1a 
  set -f              ##### ... and part 1b: disable globbing
  echo changed $A
  B=($A)  ### this is now parsed at :  (not at the default IFS whitespace) 
  echo B...... $B
  echo B[0]... ${B[0]}
  echo B[1]... ${B[1]}
  echo B[2]... ${B[2]}
  echo B[@]... ${B[@]}
# Reset the original IFS
  IFS=$SFI             ##### Important bit part 2a
  set +f               ##### ... and part 2b
  echo normal. $A

# Output
normal. token0:token1:token2.y token2.z
changed token0 token1 token2.y   token2.z 
B...... token0
B[0]... token0
B[1]... token1
B[2]... token2.y   token2.z 
B[@]... token0 token1 token2.y   token2.z 
normal. token0:token1:token2.y token2.z

Answer 2

There are major two approaches. One is IFS, demonstrated by fred.bear. This has the advantage of not requiring a separate process, but it can be tricky to get right when your input might have characters that have special meaning to the shell. The other approach is to use a text processing utility. Field splitting is built into awk.

input="token1;token2;token3;token4"
awk -vinput="$input" 'BEGIN {
    count = split(input, a, ";");
    print "first field: " a[1];
    print "second: field" a[2];
    print "number of fields: " count;
    exit;
}'

Awk is particularly appropriate when processing multiple inputs.

command_producing_semicolon_separated_data |
awk -F ';' '{
    print "first field: " $1;
    print "second field: " $2;
    print "number of fields: " NF;
}'

Answer 3

$ str="token1;token2;token3;token4"
$ echo $str
token1;token2;token3;token4
$ echo $str | tr ';' ' '
token1 token2 token3 token4
$ arr=( $(echo $str | tr ';' ' ') ) # Populate the tokens into an array
$ echo ${arr[0]}  # Access items by index
token1
$ echo ${arr[2]}
token3
$ echo ${arr[1]}
token2
$ echo ${#arr[@]}  # Length of the array
4