2

OS: kernel 2.6.x

Shell: POSIX compatible shell

Utilities: BusyBox 1.25

Question: How do you remove the values in $VAR2 from $VAR1 and output the remainder values to $VAR3 ? Each value in the variables is space separated.

Logic:

VAR1="1 2 3 4 5"
VAR2="1 3 5"
for i in $VAR1
   if $i is not found in $VAR2; do
   append $i to $VAR3
   remove trailing space character
done

Desired Output:

VAR3="2 4"
2

First, simple, but limited variant

VAR3=$(printf "%d\n" $VAR1 $VAR2 | sort | uniq -u | tr '\n' ' ' | sed 's/\s$//)

$ echo "$VAR3"
2 4

The main flaw is: it leaves only unique values from the VAR1 variable. That is, if $VAR1 have one value repeated few times, this value doesn't appear in the $VAR3, because it's not unique.

Example:

VAR1="1 2 2 3 4 4 4 5"
VAR2="1 3 5"
# the resulting VAR3 variable is empty
VAR3 = "" # because it is containing only unique values and `2` and `4` repeated few times in the `VAR1`, therefore, they are not unique.

# The right result should be
VAR3 = "2 2 4 4 4" 

Second, more universal and right variant

VAR3=$(printf "%s\n" $VAR2 | awk -v var1="$VAR1" '
{arr2[$1] = 1;}

END {
    size = split(var1, arr1); 
    for(i = 1; i <= size; i++) {
        if(!arr2[arr1[i]]) 
            printf "%s ", arr1[i];
    }
}' | sed 's/\s$//')

Explanation

  1. printf "%s\n" $VAR2 - converts the $VAR2 to the column - one value per line.
  2. awk ... - removes $VAR2 values from $VAR1.

    • {arr2[$1] = 1;} - puts all VAR2 values (they piped to awk by printf) into the array, where values becomes the array's indexes. The = 1 just mean true - this value exist. This trick give us next behaviour: the first value occurrence create array element, then, if the same value appears again, it goes to the same array index, in other words, the item doesn't changed, when the same value appears few times. Thus, in the end we are having all unique values from the VAR2 variable. If VAR2="one three five", then arr2 will be: arr2[one] = 1, arr2[three] = 1, arr2[five] = 1.
    • END { size = split(var1, arr1); - when input lines ended (VAR2 processing complete), we are splitting the VAR1 into the array - each value goes into the separate item. If VAR1="one two three four five", then we will get the follow array:arr1[1] = one, arr1[2] = two, arr1[3] = three ..., so on. The split function return the size of new, created array.
    • if(!arr2[arr1[i]]) printf "%s ", arr1[i]; - then, iterates through arr1 items and checks, does arr2 have index for this item. For example: i = 1; arr1[1] = "one" then arr2[arr1[i]] is this - arr2[one]. This item exists, don't print it. i = 2; arr1[2] = "two". The arr2[two] doesn't exist, so print it. Thus, we are printing all values from the arr1, which doesn't appear in the arr2.
  3. sed 's/\s$//' - removes trailing space.

Pros of this way compared to the first variant:

    # It can process strings
    VAR1="one two three four five"
    VAR2="one three five"
    # the resulting VAR3 variable
    VAR3 = "two four"

    # It doesn't remove multiple occurrence of one value in the VAR1
    VAR1="1 2 2 3 4 4 4 5"
    VAR2="1 3 5"
    # the resulting VAR3 variable
    VAR3 = "2 2 4 4 4"
  • 1
    Thank you for the detailed answer. In the first example, $VAR3 ends up containing a newline. In the script, echo $VAR shows a single line, but at the command line, the values break to a second line. – uihdff Sep 26 '17 at 11:48
  • 1
    @uihdff Fixed again :) Now trailing space removed by the sed. I removed previous comments, because they contained wrong information (about newlines and quoting). – MiniMax Sep 26 '17 at 12:32
  • 1
    @uihdff No, the command substitution $( ) can be written on multiple lines. In this case, it starts on the first line VAR3=$( and ends on the last line sed 's/\s$//'). Just copy all code to the command line and press Enter. – MiniMax Sep 26 '17 at 13:30
  • 1
    @uihdff Not the first line removes values from $VAR1, but all awk command, written on multiple lines do it. I can add comments inside awk code, if you are needed. – MiniMax Sep 26 '17 at 13:52
  • 1
    @uihdff Instead comments, I did addition into the explanation section. – MiniMax Sep 26 '17 at 14:51
3

It looks like those are meant to be list of strings which you encode by storing them space separated in a scalar variable (assumes the strings don't contain that space character).

It would make more sense to use variables of type list/array, with shells that support them. For instance, with zsh and its ${varX:|varY} array disjunction operator:

VAR1=(1 2 3 4 5)
VAR2=(1 3 5)
VAR3=(${VAR1:|VAR2})

(VAR3=("${(@)VAR1:|VAR2}") to preserve empty elements)

Now, if you're limited to POSIX sh that is without array support except for $@, you'll have to be more creative.

The standard command for list conjunction and disjunction is comm. But the lists have to be provided as a sorted list, newline separated and inside files whose name is passed as arguments (though - can be used for one them to mean stdin).

So here, it becomes awkward to use. If your system supports /dev/fd/<n> special files:

VAR3=$(printf '%s\n' "$VAR1" | tr ' ' '\n' | sort | {
  printf '%s\n' "$VAR2" | tr ' ' '\n' | sort |
    comm -23 /dev/fd/3 -
} 3<&0 | paste -sd ' ' -)

Or:

to_comm() { printf '%s\n' "$@" | tr ' ' '\n' | sort; }
from_comm() { paste -sd ' ' -; }
VAR3=$(to_comm "$VAR1" | { to_comm "$VAR2" | comm -23 /dev/fd/3 -;} 3<&0 |from_comm)

(that also assumes $VAR1 contains at least one element (how would you express a list with one empty elements differently from an empty list, BTW) and that elements don't contain newline characters).

So you might as well implement it by hand. Loop over each element of the first list and look them up in the second list.

In POSIX shells, you could use the split+glob operator:

IFS=' ' # split on space
set -o noglob # we don't want the glob part
VAR3= sep=
for i in $VAR1; do
  case " $VAR2 " in
    (*" $i "*) ;;
    (*) VAR3=$VAR3$sep$i; sep=$IFS;;
  esac
done

That can't be used if there may be empty elements (like in VAR1=' 2 3' or VAR1='1 3'). For that it would be better to use a non-whitespace separator (like | below) for which the splitting rules are different:

VAR1='*|foo bar||blah' VAR2='|blah'
IFS='|' # split on |
set -o noglob # we don't want the glob part
VAR3= sep=
for i in $VAR1''; do
  # that $VAR1 split+glob invocation will split the content of $VAR1
  # into "*", "foo bar", "", "blah" while with IFS=" ", the empty
  # element wouldn't have been there as sequences of spaces would
  # have been seen as a single separator. 
  case "|$VAR2|" in
    (*"|$i|"*) ;;
    (*) VAR3=$VAR3$sep$i; sep=$IFS;;
  esac
done

The '' in $VAR1'' is to make sure foo| is split into "foo" and "" instead of just "foo" for instance in POSIX shells (most, as that's a POSIX requirement) that treat $IFS as a field terminator instead of separator.

Or you could use awk instead:

export VAR1 VAR2
VAR3=$(awk 'BEGIN{
  n = split(ENVIRON["VAR1"], a1, /[ ]/)
      split(ENVIRON["VAR2"], a2, /[ ]/)
  for (i in a2) in_a2[a2[i]]
  for (i = 1; i <= n; i++)
    if (! (a1[i] in in_a2)) $(++NF) = a1[i]
  print}')
  • Thank you for the extremely detailed answer. Unfortunately, this OS does not support special files or the comm command. Is the reason this is so complicated due to having to account for various types of values ? – uihdff Sep 26 '17 at 11:43
  • 1
    @uihdff, what system is that? comm is a standard/POSIX command. If that's using busybox with comm not enabled, the awk ones should work as long as awk has been enabled. If not, the shell only one should work as long as the ash variant is used for the busybox sh. – Stéphane Chazelas Sep 26 '17 at 11:51
  • It's a router firmware that supports BusyBox 1.25. – uihdff Sep 26 '17 at 11:51
  • 1
    @uihdff, Linux has /dev/fd, but your OS might be missing the /dev/fd -> /proc/self/fd symlink (in which case you can use /proc/self/fd instead of /dev/fd in the code above. But in any case, that won't help if you don't have comm. The other solutions should work with the default build of busybox 1.25 – Stéphane Chazelas Sep 26 '17 at 11:55
  • 1
    @uihdff, check the edit, see if that helps. – Stéphane Chazelas Sep 26 '17 at 14:47
1
> echo $VAR1 $VAR2 | tr ' ' '\n' | sort | uniq --unique | tr '\n' ' '
2 4
  • 1
    Unquoted echo is a sure source of (future) problems. Try with VAR1="1 * 3". – Arrow Sep 24 '17 at 23:16
  • Thank you. Your answer accounts for the newline delimiter in MiniMax's answer. However, when executed at the command line, it needs an extra trailing newline to separate from the prompt. In the script, it wouldn't be noticed. – uihdff Sep 26 '17 at 11:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.