10

Parsing an array using IFS with non-whites space values creates empty elements.
Even using tr -s to shrink multiple delims to a single delim isn't enough.
An example may explain the issue more clearly..
Is there a way to achieve "normal" results via a tweaking of IFS (is there an associated setting to change IFS's behaviour? .... ie. To act the same as the default whitespace IFS.

var=" abc  def   ghi    "
echo "============== IFS=<default>"
arr=($var)
for x in ${!arr[*]} ; do
   echo "# arr[$x] \"${arr[x]}\""
done
#
sfi="$IFS" ; IFS=':'
set -f # Disable file name generation (globbing)
       # (This  data won't "glob", but unless globbing     
       #  is actually needed, turn if off, because   
       #  unusual/unexpected combinations of data can glob!
       #  and they can do it in the most obscure ways...  
       #  With IFS, "you're not in Kansas any more! :)  
var=":abc::def:::ghi::::"
echo "============== IFS=$IFS"
arr=($var)
for x in ${!arr[*]} ; do
   echo "# arr[$x] \"${arr[x]}\""
done
echo "============== IFS=$IFS and tr"
arr=($(echo -n "$var"|tr -s "$IFS"))
for x in ${!arr[*]} ; do
   echo "# arr[$x] \"${arr[x]}\""
done
set +f     # enable globbing 
IFS="$sfi" # re-instate original IFS val
echo "============== IFS=<default>"

Here is the output


============== IFS=<default>
# arr[0] "abc"
# arr[1] "def"
# arr[2] "ghi"
============== IFS=:
# arr[0] ""
# arr[1] "abc"
# arr[2] ""
# arr[3] "def"
# arr[4] ""
# arr[5] ""
# arr[6] "ghi"
# arr[7] ""
# arr[8] ""
# arr[9] ""
============== IFS=: and tr
# arr[0] ""
# arr[1] "abc"
# arr[2] "def"
# arr[3] "ghi"
============== IFS=<default>
3

To remove multiple (non-space) consecutive delimiter chars, two (string/array) parameter expansions can be used. The trick is to set the IFS variable to the empty string for the array parameter expansion.

This is documented in man bash under Word Splitting:

Unquoted implicit null arguments, resulting from the expansion of parameters that have no values, are removed.

(
set -f
str=':abc::def:::ghi::::'
IFS=':'
arr=(${str})
IFS=""
arr=(${arr[@]})

echo ${!arr[*]}

for ((i=0; i < ${#arr[@]}; i++)); do 
   echo "${i}: '${arr[${i}]}'"
done
)
  • Good! A simple and effective method - with no need for a bash loop and no need to call a utility app — BTW. As you mentioned "(non-space)", I'd point out, for clarity, that it works fine with any combination of delimiter chars, including space. – Peter.O May 22 '15 at 18:37
  • In my tests setting IFS=' ' (i.e. a whitespace) behaves the same. I find this less confusing than an explicit null argument ("" or '') of IFS. – Micha Wiedenmann Sep 22 '15 at 15:13
  • That's kind of a terrible solution if your data contains embedded whitespace. This, if your data was 'a bc' instead of 'abc', IFS="" would split 'a' into a separate element from 'bc'. – Dejay Clayton Sep 24 '15 at 15:19
5

From bash manpage :

Any character in IFS that is not IFS whitespace, along with any adjacent IFS whitespace characters, delimits a field. A sequence of IFS whitespace characters is also treated as a delimiter.

It means that IFS whitespace (space, tab and newline) is not treated like the other separators. If you want to get exactly the same behaviour with an alternative separator, you can do some separator swapping with the help of tr or sed :

var=":abc::def:::ghi::::"
arr=($(echo -n $var | sed 's/ /%#%#%#%#%/g;s/:/ /g'))
for x in ${!arr[*]} ; do
   el=$(echo -n $arr | sed 's/%#%#%#%#%/ /g')
   echo "# arr[$x] \"$el\""
done

The %#%#%#%#% thing is a magic value to replace the possible spaces inside the fields, it is expected to be "unique" (or very unlinkely). If you are sure that no space will ever be in the fields, just drop this part).

  • @FussyS... Thanks (see modificaton in my question ) ... You may have given me the answer to my intended question.. and that answer may be (probably is) "There is no way to get IFS to behave in the manner I want"... I intendet the tr examples to show the problem... I want to avoid a system call, so I'll look at a bash option beyond the ${var##:} which I mentioned in my comment to glen's ansewer..... I'll wait for a time.. maybe there is a way to coax IFS, otherwise the first part of your answer is was after.... – Peter.O Feb 23 '11 at 17:31
  • That treatment of IFS is the same in all Bourne-style shells, it's specified in POSIX. – Gilles Feb 23 '11 at 21:25
  • 4-plus years since i asked this question - i found @nazad's answer (posted over a year ago) to be the simplest way to juggle IFS to create an array with any number and combination of IFS chars as delimiter-string. My question was best answered by jon_d, but @nazad's's answer shows a nifty way to use IFS with no loops and no utility apps. – Peter.O May 7 '15 at 2:48
2

As bash IFS does not provide an in-house way to treat consecutive delimiter chars as a single delimiter (for non-whitespace delimiters), I have put together an all bash version (vs.using an external call eg. tr, awk, sed)

It can handle mult-char IFS..

Here are its execution-time resu;ts, along with similar tests for the tr and awk options shown on this Q/A page... The tests are based on 10000 itterations of just building the arrray (with no I/O )...

pure bash     3.174s (28 char IFS)
call (awk) 0m32.210s  (1 char IFS) 
call (tr)  0m32.178s  (1 char IFS) 

Here is the output

# dlm_str  = :.~!@#$%^&()_+-=`}{][ ";></,
# original = :abc:.. def:.~!@#$%^&()_+-=`}{][ ";></,'single*quote?'..123:
# unified  = :abc::::def::::::::::::::::::::::::::::'single*quote?'::123:
# max-w 2^ = ::::::::::::::::
# shrunk.. = :abc:def:'single*quote?':123:
# arr[0] "abc"
# arr[1] "def"
# arr[2] "'single*quote?'"
# arr[3] "123"

Here is the script

#!/bin/bash

# Note: This script modifies the source string. 
#       so work with a copy, if you need the original. 
# also: Use the name varG (Global) it's required by 'shrink_repeat_chars'
#
# NOTE: * asterisk      in IFS causes a regex(?) issue,     but  *  is ok in data. 
# NOTE: ? Question-mark in IFS causes a regex(?) issue,     but  ?  is ok in data. 
# NOTE: 0..9 digits     in IFS causes empty/wacky elements, but they're ok in data.
# NOTE: ' single quote  in IFS; don't know yet,             but  '  is ok in data.
# 
function shrink_repeat_chars () # A 'tr -s' analog
{
  # Shrink repeating occurrences of char
  #
  # $1: A string of delimiters which when consecutively repeated and are       
  #     considered as a shrinkable group. A example is: "   " whitespace delimiter.
  #
  # $varG  A global var which contains the string to be "shrunk".
  #
# echo "# dlm_str  = $1" 
# echo "# original = $varG" 
  dlms="$1"        # arg delimiter string
  dlm1=${dlms:0:1} # 1st delimiter char  
  dlmw=$dlm1       # work delimiter  
  # More than one delimiter char
  # ============================
  # When a delimiter contains more than one char.. ie (different byte` values),    
  # make all delimiter-chars in string $varG the same as the 1st delimiter char.
  ix=1;xx=${#dlms}; 
  while ((ix<xx)) ; do # Where more than one delim char, make all the same in varG  
    varG="${varG//${dlms:$ix:1}/$dlm1}"
    ix=$((ix+1))
  done
# echo "# unified  = $varG" 
  #
  # Binary shrink
  # =============
  # Find the longest required "power of 2' group needed for a binary shrink
  while [[ "$varG" =~ .*$dlmw$dlmw.* ]] ; do dlmw=$dlmw$dlmw; done # double its length
# echo "# max-w 2^ = $dlmw"
  #
  # Shrik groups of delims to a single char
  while [[ ! "$dlmw" == "$dlm1" ]] ; do
    varG=${varG//${dlmw}$dlm1/$dlm1}
    dlmw=${dlmw:$((${#dlmw}/2))}
  done
  varG=${varG//${dlmw}$dlm1/$dlm1}
# echo "# shrunk.. = $varG"
}

# Main
  varG=':abc:.. def:.~!@#$%^&()_+-=`}{][ ";></,'\''single*quote?'\''..123:' 
  sfi="$IFS"; IFS=':.~!@#$%^&()_+-=`}{][ ";></,' # save original IFS and set new multi-char IFS
  set -f                                         # disable globbing
  shrink_repeat_chars "$IFS" # The source string name must be $varG
  arr=(${varG:1})    # Strip leading dlim;  A single trailing dlim is ok (strangely
  for ix in ${!arr[*]} ; do  # Dump the array
     echo "# arr[$ix] \"${arr[ix]}\""
  done
  set +f     # re-enable globbing   
  IFS="$sfi" # re-instate the original IFS
  #
exit
  • Great work, interesting +1! – F. Hauri Feb 10 '13 at 14:43
1

You can do it with gawk too, but it's not pretty:

var=":abc::def:::ghi::::"
out=$( gawk -F ':+' '
  {
    # strip delimiters from the ends of the line
    sub("^"FS,"")
    sub(FS"$","")
    # then output in a bash-friendly format
    for (i=1;i<=NF;i++) printf("\"%s\" ", $i)
    print ""
  }
' <<< "$var" )
eval arr=($out)
for x in ${!arr[*]} ; do
  echo "# arr[$x] \"${arr[x]}\""
done

outputs

# arr[0] "abc"
# arr[1] "def"
# arr[2] "ghi"
  • Thanks... I seem to have not been clear in my main request (modified question)... It's easy enough to do it by just changing my $var to ${var##:} ... I was really after a way to tweak IFS itself.. I want to do this without an external call (I have a feeling that bash can do this more effeciently than any external can.. so I'll keep on that track)... your method works (+1).... As far as modifying the input goes, I'd prefer to try it with bash, rather than awk or tr (it would avoid a system call), but I'm really hanging out for an IFS tweak... – Peter.O Feb 23 '11 at 17:57
  • @fred, as mentioned, IFS only slurps up multiple consecutive delimeters for the default whitespace value. Otherwise, consecutive delimiters results in extraneous empty fields. I expect one or two external calls is exceedingly unlikely to impact performance in any real way. – glenn jackman Feb 23 '11 at 20:23
  • @glen.. (You said your answer is not "pretty".. I think it is! :) However, I have put together an all bash version (vs an external call) and based on 10000 itterations of just building the arrray (no I/O)... bash 1.276s ... call (awk) 0m32.210s ,,, call (tr) 0m32.178s ... Do that a few times and you might think bash is slow! ... Is awk easier in this case? ... not if you've already got the snippet :) ... I'll post it later; must go now. – Peter.O Feb 24 '11 at 3:00
  • Just by the way, re your gawk script... I've basically not used awk before, so I've been looking at it (and others) in detail...I can't pick why, but I'll mention the issue anyhow.. When given quoted data, it looses the quotes, and splits at spaces between the quotes.. and crashes for odd numbers of quotes... Here's the test data: var="The \"X\" factor:::A single '\"' crashes:::\"One Two\"" – Peter.O Feb 24 '11 at 16:28
-1

The simple answer is: collapse all delimiters to one (the first).
That require a loop (which runs less than log(N) times):

 var=':a bc::d ef:#$%_+$$%      ^%&*(*&*^
 $#,.::ghi::*::'                           # a long test string.
 d=':@!#$%^&*()_+,.'                       # delimiter set
 f=${d:0:1}                                # first delimiter
 v=${var//["$d"]/"$f"};                    # convert all delimiters to
 :                                         # the first of the delimiter set.
 tmp=$v                                    # temporal variable (v).
 while
     tmp=${tmp//["$f"]["$f"]/"$f"};        # collapse each two delimiters to one
     [[ "$tmp" != "$v" ]];                 # If there was a change
 do
     v=$tmp;                               # actualize the value of the string.
 done

All that is left to do is to split correctly the string on one delimiter, and print it:

 readarray -td "$f" arr < <(printf '%s%s' "$v"'' "$f")
 printf '<%s>' "${arr[@]}" ; echo

No need for set -f nor to change IFS.
Tested with spaces, newlines, and glob characters. All work. Quite slow (as a shell loop should be expected to be).
But only for bash (bash 4.4+ because of option -d to readarray).


sh

A shell version can not use an array, the only array available are the positional parameters.
Using tr -s is just one line (IFS doesn't change in the script):

 set -f; IFS=$f command eval set -- '$(echo "$var" | tr -s "$d" "[$f*]" )""'

And print it:

 printf '<%s>' "$@" ; echo

Still slow, but not much more.

The command command is invalid in Bourne.
In zsh, command calls only external commands and makes eval fail if command is used.
In ksh, even with command, the value of IFS is changed in the global scope.
And command makes the split fail in mksh related shells (mksh, lksh, posh) Removing the command command makes the code run on more shells. But: removing command will make IFS retain its value in most shells (eval is a special builtin) except in bash (without posix mode) and zsh in default (no emulation) mode. This concept can not be made to work in default zsh either with or without command.


Multiple character IFS

Yes, IFS could be multi character, but each character will generate one argument:

 set -f; IFS="$d" command eval set -- '$(echo "$var" )""'
 printf '<%s>' "$@" ; echo

Will output:

 <><a bc><><d ef><><><><><><><><><      ><><><><><><><><><
 ><><><><><><ghi><><><><><>

With bash, you can omit the command word if not in sh/POSIX emulation. The command will fail in ksh93 (IFS keeps the changed value). In zsh the command command makes zsh try to find eval as an external command (which it doesn't find) and fails.

What happens is that the only IFS characters that are auto collapsed to one delimiter are IFS white space.
One space in IFS will collapse all consecutive spaces to one. One tab will collapse all tabs. One space and one tab will collapse runs of spaces and/or tabs to one delimiter. Repeat the idea with newline.

To collapse several delimiters some juggling around is required.
Assuming ASCII 3 (0x03) is not used in the input var:

 var=${var// /$'\3'}                       # protect spaces
 var=${var//["$d"]/ }                      # convert all delimiters to spaces
 set -f;                                   # avoid expanding globs.
 IFS=" " command eval set -- '""$var""'    # split on spaces.
 set -- "${@//$'\3'/ }"                    # convert spaces back.

Most of the comments about ksh,zsh and bash (about command and IFS) still apply here.

A value of $'\0' would be less probable in text input, but bash variables can not contain NULs (0x00).

There are no internal commands in sh to do the same string operations, so tr is the only solution for sh scripts.

  • Yes, I wrote that for the shell the OP asked for: Bash. In that shell IFS is not kept. And yes, is not portable, to zsh , for example. @StéphaneChazelas – Isaac Jan 8 at 16:29
  • In the case of bash and zsh, they behave as POSIX specifies when invoked as sh – Stéphane Chazelas Jan 8 at 17:05
  • @StéphaneChazelas Added (many) notes about limitations of each shell. – Isaac Jan 9 at 10:58
  • @StéphaneChazelas Why the downvote? – Isaac Jan 9 at 11:24
  • Don't know, wasn't me. BTW, I think there is a dedicated Q&A here about command eval IIRC by Gilles – Stéphane Chazelas Jan 9 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.