Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I really enjoying using control+r to recursively search my command history. I've found a few good options I like to use with it:

# ignore duplicate commands, ignore commands starting with a space
export HISTCONTROL=erasedups:ignorespace

# keep the last 5000 entries
export HISTSIZE=5000

# append to the history instead of overwriting (good for multiple connections)
shopt -s histappend

The only problem for me is that erasedups only erases sequential duplicates - so that with this string of commands:

cd ~

The ls command will actually be recorded twice. I've thought about periodically running w/ cron:

cat .bash_history | sort | uniq > temp.txt
mv temp.txt .bash_history

This would achieve removing the duplicates, but unfortunately the order would not be preserved. If I don't sort the file first I don't believe uniq can work properly.

How can I remove duplicates in my .bash_history, preserving order?

Extra Credit:

Are there any problems with overwriting the .bash_history file via a script? For example, if you remove an apache log file I think you need to send a nohup / reset signal with kill to have it flush it's connection to the file. If that is the case with the .bash_history file, perhaps I could somehow use ps to check and make sure there are no connected sessions before the filtering script is run?

share|improve this question
Try ignoredups instead of erasedups for a while and see how that works for you. – jw013 Sep 20 '12 at 15:54
I don't think bash holds an open file handle to the history file - it reads/writes it when it needs to, so it should (note - should - I haven't tested) be safe to overwrite it from elsewhere. – D_Bye Sep 20 '12 at 19:00
up vote 21 down vote accepted

Sorting the history

This command works like sort|uniq, but keeps the lines in place

nl|sort -k 2|uniq -f 1|sort -n|cut -f 2

Basically, prepends to each line its number. After sort|uniq-ing, all lines are sorted back according to their original order (using the line number field) and the line number field is removed from the lines.

This solution has the flaw that it is undefined which representative of a class of equal lines will make it in the output and therefore its position in the final output is undefined. However, if the latest representative should be chosen you can sort the input by a second key:

nl|sort -k2 -k 1,1nr|uniq -f1|sort -n|cut -f2

Managing .bash_history

For re-reading and writing back the history, you can use history -a and history -w respectively.

share|improve this answer
A version of decorate-sort-undecorate, implemented with shell tools. Nice. – ire_and_curses Sep 20 '12 at 17:21
With sort, the -r switch always reverses the sorting order. But this won't yield the result you have in mind. sort regards the two occurrences of ls as identical with the result that, even when reversed, the eventual order depends on the sorting algorithm. But see my update for another idea. – artistoex Sep 20 '12 at 19:29
In case, you don't want to modify .bash_history, you could put the following in .bashrc: alias history='history | sort -k2 -k 1,1nr | uniq -f 1 | sort -n' – Nathan Jan 15 '14 at 20:35
What is nl at the beginning of each code line? Shouldn't it be history? – A.L Feb 4 '15 at 9:50
@A.L nl adds line numbers. The command as a whole solves the general problem: removing duplicates while preserving order. The input is read from stdin. – artistoex Feb 5 '15 at 21:24

Found this solution in the wild and tested:

awk '!x[$0]++'

I would add an explanation of the code, but I can't grok Awk yet. I welcome anyone who knows how this works to edit the post and add it in here.

share|improve this answer
Wow! That just worked. But it removes all but the first occurrence I guess. I'd reversed the ordering of the lines using Sublime Text before running this. Now I'll reverse it again to get a clean history with only the last occurrence of all duplicates left behind. Thank you. – trss Aug 27 '14 at 20:05
Check out my answer! – Ali Shakiba Jan 19 '15 at 10:26

These would keep the last duplicated lines:

ruby -i -e 'puts readlines.reverse.uniq.reverse' ~/.bash_history
tac ~/.bash_history | awk '!a[$0]++' | tac > t; mv t ~/.bash_history
share|improve this answer

Extending Clayton answer:

tac $HISTFILE | awk '!x[$0]++' | tac | sponge $HISTFILE

tac reverse the file, make sure you have installed moreutils so you have sponge available, otherwise use a temp file.

share|improve this answer
For those on Mac, use brew install coreutils, and notice that all the GNU utils have a g prepended to avoid confusion with the BSD built-in Mac commands (e.g. gsed is GNU whereas sed is BSD). So use gtac. – tralston Jun 11 '15 at 20:26

So I was looking for the same exact thing after being annoyed by duplicates, and found that if I edit my ~/.bash_profile (Mac) with:

export HISTCONTROL=ignoreboth:erasedups

It does exactly what you wanted, it only keeps the latest of any command. ignoreboth is actually just like doing ignorespace:ignoredups and that along with erasedups gets the job done.

At least on my Mac terminal with bash this work perfect. Found it here on askubuntu.com.

share|improve this answer
this should be correct answer – MitchBroadhead Mar 3 at 10:55

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.