I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my PATH environment variable.

I was told that it is possible to achieve this with a one line command using the awk command, but I cannot figure out how to do it. Anybody know how?


20 Answers 20


If you don't already have duplicates in the PATH and you only want to add directories if they are not already there, you can do it easily with the shell alone.

for x in /path/to/add …; do
  case ":$PATH:" in
    *":$x:"*) :;; # already there
    *) PATH="$x:$PATH";;

And here's a shell snippet that removes duplicates from $PATH. It goes through the entries one by one, and copies those that haven't been seen yet.

if [ -n "$PATH" ]; then
  old_PATH=$PATH:; PATH=
  while [ -n "$old_PATH" ]; do
    x=${old_PATH%%:*}       # the first remaining entry
    case $PATH: in
      *:"$x":*) ;;          # already there
      *) PATH=$PATH:$x;;    # not there yet
  unset old_PATH x
  • It would be better, if iterate items in $PATH reversely, because the later ones are newly added usually, and they might have the value up to date.
    – Eric
    Commented Sep 3, 2016 at 6:48
  • 7
    @EricWang I don't understand your reasoning. PATH elements are traversed from front to back, so when there are duplicates, the second duplicate is effectively ignored. Iterating from back to front would change the order. Commented Sep 3, 2016 at 10:50
  • 7
    @EricWang In that case, the added value has no effect so should be ignored. By going backwards, you're making the added value come before. If the added value had been supposed to go before, it would have been added as PATH=x:$PATH. Commented Sep 3, 2016 at 15:42
  • 1
    @DonHatch My own .profile is even more complicated than that (it has complex stuff to sort both existing and added entries), but not everyone needs the complexity. I generally prefer to present possibilities in order of increasing complexity. Commented Nov 24, 2017 at 21:28
  • 2
    @DonHatch I want to keep it because it serves the needs of most people. I do point out that it assumes that there are no duplicates at the beginning, what more do you want? The order of addition is a different issue which is not mentioned in the question and not solved by the duplicate removal code. Commented Nov 24, 2017 at 21:54

Here's an intelligible one-liner solution that does all the right things: removes duplicates, preserves the ordering of paths, and doesn't add a colon at the end. So it should give you a deduplicated PATH that gives exactly the same behavior as the original:

PATH="$(perl -e 'print join(":", grep { not $seen{$_}++ } split(/:/, $ENV{PATH}))')"

It simply splits on colon (split(/:/, $ENV{PATH})), uses uses grep { not $seen{$_}++ } to filter out any repeated instances of paths except for the first occurrence, and then joins the remaining ones back together separated by colons and prints the result (print join(":", ...)).

If you want some more structure around it, as well as the ability to deduplicate other variables as well, try this snippet, which I'm currently using in my own config:

# Deduplicate path variables
get_var () {
    eval 'printf "%s\n" "${'"$1"'}"'
set_var () {
    eval "$1=\"\$2\""
dedup_pathvar () {
    pathvar_value="$(get_var "$pathvar_name")"
    deduped_path="$(perl -e 'print join(":",grep { not $seen{$_}++ } split(/:/, $ARGV[0]))' "$pathvar_value")"
    set_var "$pathvar_name" "$deduped_path"
dedup_pathvar PATH
dedup_pathvar MANPATH

That code will deduplicate both PATH and MANPATH, and you can easily call dedup_pathvar on other variables that hold colon-separated lists of paths (e.g. PYTHONPATH).

  • For some reason I had to add a chomp to remove a trailing newline. This worked for me: perl -ne 'chomp; print join(":", grep { !$seen{$_}++ } split(/:/))' <<<"$PATH" Commented Dec 28, 2014 at 19:05
  • 1
    I can't get this fix to persist. It does clean duplicates from $PATH, but if I open a new Ubuntu WSL2 command prompt window, my $PATH is back to having duplicates. How can I make this permanent? Commented Dec 24, 2020 at 15:39
  • 2
    @KyleVassella Did you add this code to your shell startup file? Commented Dec 25, 2020 at 3:44

Here's a sleek one:

printf %s "$PATH" | awk -v RS=: -v ORS=: '!arr[$0]++'

Longer (to see how it works):

printf %s "$PATH" | awk -v RS=: -v ORS=: '{ if (!arr[$0]++) { print $0 } }'

Ok, since you're new to linux, here is how to actually set PATH without a trailing ":"

PATH=`printf %s "$PATH" | awk -v RS=: '{ if (!arr[$0]++) {printf("%s%s",!ln++?"":":",$0)}}'`

btw make sure to NOT have directories containing ":" in your PATH, otherwise it is gonna be messed up.

some credit to:

  • -1 this doesn't work. I still see duplicates in my path.
    – dogbane
    Commented Jun 14, 2012 at 7:34
  • 5
    @dogbane: It removes duplicates for me. However it has a subtle problem. The output has a : on the end which if set as your $PATH, means the current directory is added the path. This has security implications on a multi-user machine.
    – camh
    Commented Jun 14, 2012 at 7:42
  • 1
    Note that echo -n outputs -n in Unix-compliant echo implementations. The standard way to output a $string without the trailing newline character is printf %s "$string", hence Gilles' edit. Generally you can't use echo for arbitrary data Commented Sep 5, 2016 at 13:47
  • 1
    Problem I ran into, duplicates with and without trailing slashes "/foo/bar:/foo/bar/" will not removed - however, they are equivalent within the PATH variable. Commented Dec 10, 2019 at 19:03
  • 1
    @ChristianHerenz, maybe awk can also split on /: and : at the same time, maybe with regular expression/pattern. Not sure ATM but might be a good thing to explore if you want to improve current solution. Commented Dec 12, 2019 at 8:25

Here is an AWK one liner.

$ PATH=$(printf %s "$PATH" \
     | awk -vRS=: -vORS= '!a[$0]++ {if (NR>1) printf(":"); printf("%s", $0) }' )


  • printf %s "$PATH" prints the content of $PATH without a trailing newline
  • RS=: changes the input record delimiter character (default is newline)
  • ORS= changes the output record delimiter to the empty string
  • a the name of an implicitly created array
  • $0 references the current record
  • a[$0] is a associative array dereference
  • ++ is the post-increment operator
  • !a[$0]++ guards the right hand side, i.e. it makes sure that the current record is only printed, if it wasn't printed before
  • NR the current record number, starting with 1

That means that AWK is used to split the PATH content along the : delimiter characters and to filter out duplicate entries without modifying the order.

Since AWK associative arrays are implemented as hash tables the runtime is linear (i.e. in O(n)).

Note that we don't need look for quoted : characters because shells don't provide quoting to support directories with : in its name in the PATH variable.

Awk + paste

The above can be simplified with paste:

$ PATH=$(printf %s "$PATH" | awk -vRS=: '!a[$0]++' | paste -s -d:)

The paste command is used to intersperse the awk output with colons. This simplifies the awk action to printing (which is the default action).


The same as Python two-liner:

$ PATH=$(python3 -c 'import os; from collections import OrderedDict; \
    l=os.environ["PATH"].split(":"); print(":".join(OrderedDict.fromkeys(l)))' )
  • ok, but does this remove dupes from an existing colon delimited string, or does it prevent dupes from being added to a string? Commented Dec 10, 2016 at 9:59
  • 1
    looks like the former Commented Dec 10, 2016 at 10:00
  • 2
    @AlexanderMills, well, the OP just asked about removing duplicates so this is what the awk call does. Commented Dec 10, 2016 at 18:59
  • 1
    The paste command doesn't work for me unless I add a trailing - to use STDIN.
    – wisbucky
    Commented Apr 24, 2017 at 21:11
  • 2
    Also, I need to add spaces after the -v or else I get an error. -v RS=: -v ORS=. Just different flavors of awk syntax.
    – wisbucky
    Commented Apr 24, 2017 at 21:56

As long as we are adding non-awk oneliners:

PATH=$(zsh -fc "typeset -TU P=$PATH p; echo \$P")

(Could be as simple as PATH=$(zsh -fc 'typeset -U path; echo $PATH') but zsh always reads at least one zshenv configuration file, which can modify PATH.)

It uses two nice zsh features:

  • scalars tied to arrays (typeset -T)
  • and arrays that autoremove duplicate values (typeset -U).
  • nice! shortest working answer, and natively without the colon at the end.
    – jaap
    Commented Dec 19, 2016 at 10:26

There has been a similar discussion about this here.

I take a bit of a different approach. Instead of just accepting the PATH that is set from all the different initialization files that get installed, I prefer using getconf to identify the system path and place it first, then add my preferred path order, then use awk to remove any duplicates. This may or may not really speed up command execution (and in theory be more secure), but it gives me warm fuzzies.

# I am entering my preferred PATH order here because it gets set,
# appended, reset, appended again and ends up in such a jumbled order.
# The duplicates get removed, preserving my preferred order.
PATH=$(command -p getconf PATH):/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH
# Remove duplicates
PATH="$(printf "%s" "${PATH}" | /usr/bin/awk -v RS=: -v ORS=: '!($0 in a) {a[$0]; print}')"
export PATH

[~]$ echo $PATH
  • 3
    This is very dangerous because you add a trailing : to the PATH (i.e. an empty string entry), because then the current working directory is part of your PATH. Commented Apr 13, 2014 at 10:05

As others have demonstrated it is possible in one line using awk, sed, perl, zsh, or bash, depends on your tolerance for long lines and readability. Here's a bash function that

  • removes duplicates
  • preserves order
  • allows spaces in directory names
  • allows you to specify the delimiter (defaults to ':')
  • can be used with other variables, not just PATH
  • works in bash versions < 4, important if you use OS X which for licensing issues does not ship bash version 4

bash function

remove_dups() {
    local D=${2:-:} path= dir=
    while IFS= read -d$D dir; do
        [[ $path$D =~ .*$D$dir$D.* ]] || path+="$D$dir"
    done <<< "$1$D"
    printf %s "${path#$D}"


To remove dups from PATH

PATH=$(remove_dups "$PATH")

If path+="$D$dir" above is replaced with path="$path$D$dir", then this function also deduplicates entries correctly in zsh (in addition to bash). Without this change, a space will be inserted before every colon when this function is used in zsh.

PATH=`perl -e 'print join ":", grep {!$h{$_}++} split ":", $ENV{PATH}'`
export PATH

This uses perl and has several benefits:

  1. It removes duplicates
  2. It keeps sort order
  3. It keeps the earliest appearance (/usr/bin:/sbin:/usr/bin will result in /usr/bin:/sbin)

Recent bash versions (>= 4) also of associative arrays, i.e. you can also use a bash 'one liner' for it:

PATH=$(IFS=:; set -f; declare -A a; NR=0; for i in $PATH; do NR=$((NR+1)); \
       if [ \! ${a[$i]+_} ]; then if [ $NR -gt 1 ]; then echo -n ':'; fi; \
                                  echo -n $i; a[$i]=1; fi; done)


  • IFS changes the input field separator to :
  • declare -A declares an associative array
  • ${a[$i]+_} is a parameter expansion meaning: _ is substituted if and only if a[$i] is set. This is similar to ${parameter:+word} which also tests for not-null. Thus, in the following evaluation of the conditional, the expression _ (i.e. a single character string) evaluates to true (this is equivalent to -n _) - while an empty expression evaluates to false.
  • +1: nice script style, but can you explain the particular syntax: ${a[$i]+_} by editing your answer and adding one bullet. The rest is perfectly understandable but you lost me there. Thank you.
    – Cbhihe
    Commented Jul 12, 2016 at 11:28
  • 1
    @Cbhihe, I've added a bullet point that addresses this expansion. Commented Jul 12, 2016 at 19:23
  • Thank you very much. Very interesting. I did not think that was possible with arrays (non-strings)...
    – Cbhihe
    Commented Jul 12, 2016 at 20:52
PATH=`awk -F: '{for (i=1;i<=NF;i++) { if ( !x[$i]++ ) printf("%s:",$i); }}' <<< "$PATH"`

Explanation of awk code:

  1. Separate the input by colons.
  2. Append new path entries to associative array for fast duplicate look-up.
  3. Prints the associative array.

In addition to being terse, this one-liner is fast: awk uses a chaining hash-table to achieve amortized O(1) performance.

based on Removing duplicate $PATH entries

  • Old post, but could you explain: if ( !x[$i]++ ) . Thanks.
    – Cbhihe
    Commented Jul 12, 2016 at 11:36

Also sed (here using GNU sed syntax) can do the job:

MYPATH=$(printf '%s\n' "$MYPATH" | sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb')

this one works well only in case first path is . like in dogbane's example.

In general case you need to add yet another s command:

MYPATH=$(printf '%s\n' "$MYPATH" | sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb;s/^\([^:]*\)\(:.*\):\1/:\1\2/')

It works even on such construction:

$ echo "/bin:.:/foo/bar/bin:/usr/bin:/foo/bar/bin:/foo/bar/bin:/bar/bin:/usr/bin:/bin" \
| sed ':b;s/:\([^:]*\)\(:.*\):\1/:\1\2/;tb;s/^\([^:]*\)\(:.*\):\1/\1\2/'


This is my version:

path_no_dup () 
    local IFS=: p=();

    while read -r; do
    done < <(sort -u <(read -ra arr <<< "$1" && printf '%s\n' "${arr[@]}"));

    # Do whatever you like with "${p[*]}"
    echo "${p[*]}"

Usage: path_no_dup "$PATH"

Sample output:

rany$ v='a:a:a:b:b:b:c:c:c:a:a:a:b:c:a'; path_no_dup "$v"

The right way of removing duplicates suggested by @ghm1014 and @rush (in the comment) using sort:

PATH=`echo -e ${PATH//:/'\n'} | awk '{printf("%d|%s\n", NR, $0)}' | sort -t '|' -k 2 -u | sort -t '|' -k 1 -g | cut -f2 -d'|'`; export PATH=${PATH//[$'\n']/:}

Since people resorted to suggesting perl oneliners, I'll also throw a coin into the pot. This one is in ruby (and way less cryptic that everything else, IMO):

PATH="$(ruby -e 'puts ENV["PATH"].split(":").uniq.join(":")')"

Use awk to split the path on :, then loop over each field and store it in an array. If you come across a field which is already in the array, that means you have seen it before, so don't print it.

Here is an example:

$ MYPATH=.:/foo/bar/bin:/usr/bin:/foo/bar/bin
$ awk -F: '{for(i=1;i<=NF;i++) if(!($i in arr)){arr[$i];printf s$i;s=":"}}' <<< "$MYPATH"

(Updated to remove the trailing :.)


A solution - not one that is as elegant as those that change the *RS variables, but perhaps reasonably clear:

PATH=`awk 'BEGIN {np="";split(ENVIRON["PATH"],p,":"); for(x=0;x<length(p);x++) {  pe=p[x]; if(e[pe] != "") continue; e[pe] = pe; if(np != "") np=np ":"; np=np pe}} END { print np }' /dev/null`

The entire program works in the BEGIN and END blocks. It pulls your PATH variable from the environment, splitting it into units. It then iterates over the resulting array p (which is created in order by split()). The array e is an associative array that is used to determine whether or not we've seen the current path element (e.g. /usr/local/bin) before, and if not, is appended to np, with logic to append a colon to np if there is already text in np. The END block simply echos np. This could be further simplified by adding the -F: flag, eliminating the third argument to split() (as it defaults to FS), and changing np = np ":" to np = np FS, giving us:

awk -F: 'BEGIN {np="";split(ENVIRON["PATH"],p); for(x=0;x<length(p);x++) {  pe=p[x]; if(e[pe] != "") continue; e[pe] = pe; if(np != "") np=np FS; np=np pe}} END { print np }' /dev/null

Naïvely, I believed that for(element in array) would preserve order, but it doesn’t, so my original solution doesn’t work, as folks would get upset if someone suddenly scrambled the order of their $PATH:

awk 'BEGIN {np="";split(ENVIRON["PATH"],p,":"); for(x in p) { pe=p[x]; if(e[pe] != "") continue; e[pe] = pe; if(np != "") np=np ":"; np=np pe}} END { print np }' /dev/null
export PATH=$(echo -n "$PATH" | awk -v RS=':' '(!a[$0]++){if(b++)printf(RS);printf($0)}')

Only the first occurrence is kept and relative order is well maintained.


(To explain @Michał Politowski 's answer:)

for zsh:

PATH=$(zsh --no-rcs -c "P_scaler=$PATH ; typeset -T P_scaler  P_array ;  typeset -U P_array ; echo \$P_scaler")

Here is a version I created a while back based on @Giles_'so stop being evil' answer above. However, my version ignores the case of the variables. This works for removing duplicates on case insensitive OSes like Windows and MacOS. Thus such things as "C:\Windows" and "C:\WINDOWS" are the same path.

low_PATH="$(tr '[:upper:]' '[:lower:]' <<<$PATH )"

if [ -n "$PATH" ]; then
  old_PATH=$PATH:; PATH= ; lPATH=
  while [ -n "$old_PATH" ]; do
    x=${old_PATH%%:*}       # the first remaining entry
    l=${low_PATH%%:*}       # The lowercase we compare
    case $lPATH: in
      *:"$l":*) ;;          # already there? Do nothing
      *) PATH=$PATH:$x      # not there yet? Add it
    old_PATH=${old_PATH#*:} # Next entry
  unset old_PATH x low_PATH lPATH l


  • Removes duplicate paths
  • Preserves order
  • Ignores case
  • Spaces in paths are allowed

May not work with all versions of Bash.


I would do it just with basic tools such as tr, sort and uniq:

NEW_PATH=`echo $PATH | tr ':' '\n' | sort | uniq | tr '\n' ':'`

If there is nothing special or weird in your path it should work

  • 1
    btw, you can use sort -u instead of sort | uniq.
    – rush
    Commented Apr 25, 2013 at 18:44
  • 12
    Since the order of the PATH elements is significant, this is not very useful. Commented Apr 13, 2014 at 9:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .