Remove duplicate $PATH entries with awk command

Question

I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my path environment.

I was told that it is possible to achieve this with a one line command using the awk command, but I cannot figure out how to do it. Anybody know how? I am new to linux so please don't be too technical.

Answer 1

If you don't already have duplicates in the PATH and you only want to add directories if they are not already there, you can do it easily with the shell alone.

for x in /path/to/add …; do
  case ":$PATH:" in
    *":$x:"*) :;; # already there
    *) PATH="$x:$PATH";;
  esac
done

And here's a shell snippet that removes duplicates from $PATH. It goes through the entries one by one, and copies those that haven't been seen yet.

set -f         # Turn off globbing, to allow unprotected variable substitutions
IFS=:
old_PATH=$PATH:; PATH=
while [ -n "$old_PATH" ]; do
  x=${old_PATH%%:*}       # the first remaining entry
  case $PATH: in
    *:${x}:*) :;;         # already there
    *) PATH=$PATH:$x;;    # not there yet
  esac
  old_PATH=${old_PATH#*:}
done
PATH=${PATH#:}
set +f; unset IFS old_PATH x

Answer 2

Also sed can do the job:

MYPATH=$(echo $MYPATH | sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb')

this one works well only in case first path is . like in dogbane's example.

In general case you need to add yet another s command:

MYPATH=$(echo $MYPATH | sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb;s/^\([^:]*\)\(:.*\):\1/:\1\2/')

It works even on such construction:

$ echo "/bin:.:/foo/bar/bin:/usr/bin:/foo/bar/bin:/foo/bar/bin:/bar/bin:/usr/bin:/bin" \
| sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb;s/^\([^:]*\)\(:.*\):\1/\1\2/'

/bin:.:/foo/bar/bin:/usr/bin:/bar/bin

Answer 3

There has been a similar discussion about this here.

I take a bit of a different approach. Instead of just accepting the PATH that is set from all the different initialization files that get installed, I prefer using getconf to identify the system path and place it first, then add my preferred path order, then use awk to remove any duplicates. This may or may not really speed up command execution (and in theory be more secure), but it gives me warm fuzzies.

# I am entering my preferred PATH order here because it gets set,
# appended, reset, appended again and ends up in such a jumbled order.
# The duplicates get removed, preserving my preferred order.
#
PATH=$(command -p getconf PATH):/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH
# Remove duplicates
PATH="$(printf "%s" "${PATH}" | /usr/bin/awk -v RS=: -v ORS=: '!($0 in a) {a[$0]; print}')"
export PATH

[~]$ echo $PATH
/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/lib64/ccache:/usr/games:/home/me/bin

Answer 4

Use awk to split the path on :, then loop over each field and store it in an array. If you come across a field which is already in the array, that means you have seen it before, so don't print it.

Here is an example:

$ MYPATH=.:/foo/bar/bin:/usr/bin:/foo/bar/bin
$ awk -F: '{for(i=1;i<=NF;i++) if(!($i in arr)){arr[$i];printf s$i;s=":"}}' <<< "$MYPATH"
.:/foo/bar/bin:/usr/bin

(Updated to remove the trailing :.)

Answer 5

Here's a sleek one:

echo -n $PATH | awk -v RS=: -v ORS=: '!arr[$0]++'

Longer (to see how it works):

echo -n $PATH | awk -v RS=: -v ORS=: '{ if (!arr[$0]++) { print $0 } }'

Ok, since you're new to linux, here is how to actually set PATH without a trailing ":"

PATH=`echo -n $PATH | awk -v RS=: '{ if (!arr[$0]++) {printf("%s%s",!ln++?"":":",$0)}}'`

btw make sure to NOT have directories containing ":" in your PATH, otherwise it is gonna be screwed.

some credit to:

• this blog
• this thread

Answer 6

PATH=`perl -e '@A=split(/:/,$ENV{PATH});%H=map {$A[$#A-$_]=>$#A-$_} (0..$#A);@A=join(":",sort{$H{$a} <=> $H{$b} }keys %H);print "@A"'`
export PATH

This uses perl and has several benefits:

1. It removes duplicates
2. It keeps sort order
3. It keeps the earliest appearance (/usr/bin:/sbin:/usr/bin will result in /usr/bin:/sbin)

Answer 7

PATH=`awk -F: '{for (i=1;i<=NF;i++) { if ( !x[$i]++ ) printf("%s:",$i); }}' <<< $PATH`

Explanation of awk code:

1. Separate the input by colons.
2. Append new path entries to associative array for fast duplicate look-up.
3. Prints the associative array.

In addition to being terse, this one-liner is fast: awk uses a chaining hash-table to achieve amortized O(1) performance.

based on Removing duplicate $PATH entries

Answer 8

I would do it just with basic tools such as tr, sort and uniq:

NEW_PATH=`echo $PATH | tr ':' '\n' | sort | uniq | tr '\n' ':'`

If there is nothing special or weird in your path it should work

Answer 9

This is my version:

path_no_dup () 
{ 
    local IFS=: p=();

    while read -r; do
        p+=("$REPLY");
    done < <(sort -u <(read -ra arr <<< "$1" && printf '%s\n' "${arr[@]}"));

    # Do whatever you like with "${p[*]}"
    echo "${p[*]}"
}

Usage: path_no_dup "$PATH"

Sample output:

rany$ v='a:a:a:b:b:b:c:c:c:a:a:a:b:c:a'; path_no_dup "$v"
a:b:c
rany$