Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my path environment.

I was told that it is possible to achieve this with a one line command using the awk command, but I cannot figure out how to do it. Anybody know how? I am new to linux so please don't be too technical.

share|improve this question
    
— An exercise for stackoverflow, as to me. :) –  poige Jun 14 '12 at 12:18
1  
@poige nope, this is the proper place. –  bahamat Jun 14 '12 at 20:52

12 Answers 12

Here's an intelligible one-liner solution that does all the right things: removes duplicates, preserves the ordering of paths, and doesn't add a colon at the end. So it should give you a deduplicated PATH that gives exactly the same behavior as the original:

PATH="$(echo $PATH | perl -e 'print join(":", grep { not $seen{$_}++ } split(/:/, scalar <>))')"

It simply splits on colon (split(/:/, scalar <>)), uses uses grep { not $seen{$_}++ } to filter out any repeated instances of paths except for the first occurrence, and then joins the remaining ones back together separated by colons and prints the result (print join(":", ...)).

If you want some more structure around it, as well as the ability to deduplicate other variables as well, try this snippet, which I'm currently using in my own config:

# Deduplicate path variables
get_var () {
    eval "echo \$$1"
}
set_var () {
    eval "$1=\"$2\""
}
dedup_pathvar () {
    pathvar_name="$1"
    pathvar_value="$(get_var $pathvar_name)"
    deduped_path="$(echo $pathvar_value | perl -e 'print join(":",grep { not $seen{$_}++ } split(/:/, scalar <>))')"
    set_var $pathvar_name "$deduped_path"
}
dedup_pathvar PATH
dedup_pathvar MANPATH

That code will deduplicate both PATH and MANPATH, and you can easily call dedup_pathvar on other variables that hold colon-separated lists of paths (e.g. PYTHONPATH).

share|improve this answer

Recent bash versions (> 4) also of associative arrays, i.e. you can also use a bash 'one liner' for it:

PATH=$(IFS=:; declare -A a; NR=0; for i in $PATH; do NR=$((NR+1)); \
       if [ \! ${a[$i]+_} ]; then if [ $NR -gt 1 ]; then echo -n ':'; fi; \
                                  echo -n $i; a[$i]=1; fi; done)

where:

  • IFS changes the input field separator to :
  • declare -A declares an associative array
share|improve this answer

Here is an AWK one liner.

$ PATH=$(echo -n $PATH \
     | awk -vRS=: -vORS= '!a[$0]++ {if (NR>1) printf(":"); printf("%s", $0) }' )

where:

  • echo -n prints the content of $PATH without a trailing newline
  • RS=: changes the input record delimiter character (default is newline)
  • ORS= changes the output record delimiter to the empty string
  • a the name of an implicitly created array
  • $0 references the current record
  • a[$0] is a associative array dereference
  • ++ is the post-increment operator
  • !a[$0]++ guards the right hand side, i.e. it makes sure that the current record is only printed, if it wasn't printed before
  • NR the current record number, starting with 1

That means that AWK is used to split the PATH content along the : delimiter characters and to filter out duplicate entries without modifying the order.

Since AWK associative arrays are implemented as hash tables the runtime is linear (i.e. in O(n)).

Note that we don't need look for quoted : characters because shells don't provide quoting to support directories with : in its name in the PATH variable.

share|improve this answer

If you don't already have duplicates in the PATH and you only want to add directories if they are not already there, you can do it easily with the shell alone.

for x in /path/to/add …; do
  case ":$PATH:" in
    *":$x:"*) :;; # already there
    *) PATH="$x:$PATH";;
  esac
done

And here's a shell snippet that removes duplicates from $PATH. It goes through the entries one by one, and copies those that haven't been seen yet.

if [ -n "$PATH" ]; then
  old_PATH=$PATH:; PATH=
  while [ -n "$old_PATH" ]; do
    x=${old_PATH%%:*}       # the first remaining entry
    case $PATH: in
      *:"$x":*) ;;         # already there
      *) PATH=$PATH:$x;;    # not there yet
    esac
    old_PATH=${old_PATH#*:}
  done
  PATH=${PATH#:}
  unset old_PATH x
fi
share|improve this answer
    
Probably the best method among those listed here. –  jw013 Sep 19 '12 at 19:42
    
Do you mean -f instead of -g. Using bash 3 and 4 I can't see -g as an option but can see -f which is the equivalent of the noglob option. –  Burhan Ali Apr 24 '13 at 10:50
    
@BurhanAli Yes, indeed. I ainwgiq^Wsomehow managed to get it wrong twice, sorry. –  Gilles Apr 24 '13 at 14:15

This is my version:

path_no_dup () 
{ 
    local IFS=: p=();

    while read -r; do
        p+=("$REPLY");
    done < <(sort -u <(read -ra arr <<< "$1" && printf '%s\n' "${arr[@]}"));

    # Do whatever you like with "${p[*]}"
    echo "${p[*]}"
}

Usage: path_no_dup "$PATH"

Sample output:

rany$ v='a:a:a:b:b:b:c:c:c:a:a:a:b:c:a'; path_no_dup "$v"
a:b:c
rany$
share|improve this answer

I would do it just with basic tools such as tr, sort and uniq:

NEW_PATH=`echo $PATH | tr ':' '\n' | sort | uniq | tr '\n' ':'`

If there is nothing special or weird in your path it should work

share|improve this answer
    
btw, you can use sort -u instead of sort | uniq. –  rush Apr 25 '13 at 18:44
3  
Since the order of the PATH elements is significant, this is not very useful. –  maxschlepzig Apr 13 at 9:13
PATH=`awk -F: '{for (i=1;i<=NF;i++) { if ( !x[$i]++ ) printf("%s:",$i); }}' <<< $PATH`

Explanation of awk code:

  1. Separate the input by colons.
  2. Append new path entries to associative array for fast duplicate look-up.
  3. Prints the associative array.

In addition to being terse, this one-liner is fast: awk uses a chaining hash-table to achieve amortized O(1) performance.

based on Removing duplicate $PATH entries

share|improve this answer
PATH=`perl -e '@A=split(/:/,$ENV{PATH});%H=map {$A[$#A-$_]=>$#A-$_} (0..$#A);@A=join(":",sort{$H{$a} <=> $H{$b} }keys %H);print "@A"'`
export PATH

This uses perl and has several benefits:

  1. It removes duplicates
  2. It keeps sort order
  3. It keeps the earliest appearance (/usr/bin:/sbin:/usr/bin will result in /usr/bin:/sbin)
share|improve this answer

There has been a similar discussion about this here.

I take a bit of a different approach. Instead of just accepting the PATH that is set from all the different initialization files that get installed, I prefer using getconf to identify the system path and place it first, then add my preferred path order, then use awk to remove any duplicates. This may or may not really speed up command execution (and in theory be more secure), but it gives me warm fuzzies.

# I am entering my preferred PATH order here because it gets set,
# appended, reset, appended again and ends up in such a jumbled order.
# The duplicates get removed, preserving my preferred order.
#
PATH=$(command -p getconf PATH):/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH
# Remove duplicates
PATH="$(printf "%s" "${PATH}" | /usr/bin/awk -v RS=: -v ORS=: '!($0 in a) {a[$0]; print}')"
export PATH

[~]$ echo $PATH
/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/lib64/ccache:/usr/games:/home/me/bin
share|improve this answer
    
This is very dangerous because you add a trailing : to the PATH (i.e. an empty string entry), because then the current working directory is part of your PATH. –  maxschlepzig Apr 13 at 10:05

Here's a sleek one:

echo -n $PATH | awk -v RS=: -v ORS=: '!arr[$0]++'

Longer (to see how it works):

echo -n $PATH | awk -v RS=: -v ORS=: '{ if (!arr[$0]++) { print $0 } }'

Ok, since you're new to linux, here is how to actually set PATH without a trailing ":"

PATH=`echo -n $PATH | awk -v RS=: '{ if (!arr[$0]++) {printf("%s%s",!ln++?"":":",$0)}}'`

btw make sure to NOT have directories containing ":" in your PATH, otherwise it is gonna be screwed.

some credit to:

share|improve this answer
    
-1 this doesn't work. I still see duplicates in my path. –  dogbane Jun 14 '12 at 7:34
1  
@dogbane: It removes duplicates for me. However it has a subtle problem. The output has a : on the end which if set as your $PATH, means the current directory is added the path. This has security implications on a multi-user machine. –  camh Jun 14 '12 at 7:42
    
@dogbane, it works and I edited post to have a one line command without the trailing : –  akostadinov Jun 14 '12 at 7:59
    
@dogbane your solution has a trailing : in the output –  akostadinov Jun 14 '12 at 8:12
    
hmm, your third command works, but the first two do not work unless I use echo -n. Your commands don't seem to work with "here strings" e.g. try: awk -v RS=: -v ORS=: '!arr[$0]++' <<< ".:/foo/bin:/bar/bin:/foo/bin" –  dogbane Jun 14 '12 at 8:32

Use awk to split the path on :, then loop over each field and store it in an array. If you come across a field which is already in the array, that means you have seen it before, so don't print it.

Here is an example:

$ MYPATH=.:/foo/bar/bin:/usr/bin:/foo/bar/bin
$ awk -F: '{for(i=1;i<=NF;i++) if(!($i in arr)){arr[$i];printf s$i;s=":"}}' <<< "$MYPATH"
.:/foo/bar/bin:/usr/bin

(Updated to remove the trailing :.)

share|improve this answer

Also sed can do the job:

MYPATH=$(echo $MYPATH | sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb')

this one works well only in case first path is . like in dogbane's example.

In general case you need to add yet another s command:

MYPATH=$(echo $MYPATH | sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb;s/^\([^:]*\)\(:.*\):\1/:\1\2/')

It works even on such construction:

$ echo "/bin:.:/foo/bar/bin:/usr/bin:/foo/bar/bin:/foo/bar/bin:/bar/bin:/usr/bin:/bin" \
| sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb;s/^\([^:]*\)\(:.*\):\1/\1\2/'

/bin:.:/foo/bar/bin:/usr/bin:/bar/bin
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.