60

I really enjoying using control+r to recursively search my command history. I've found a few good options I like to use with it:

# ignore duplicate commands, ignore commands starting with a space
export HISTCONTROL=erasedups:ignorespace

# keep the last 5000 entries
export HISTSIZE=5000

# append to the history instead of overwriting (good for multiple connections)
shopt -s histappend

The only problem for me is that erasedups only erases sequential duplicates - so that with this string of commands:

ls
cd ~
ls

The ls command will actually be recorded twice. I've thought about periodically running w/ cron:

cat .bash_history | sort | uniq > temp.txt
mv temp.txt .bash_history

This would achieve removing the duplicates, but unfortunately the order would not be preserved. If I don't sort the file first I don't believe uniq can work properly.

How can I remove duplicates in my .bash_history, preserving order?

Extra Credit:

Are there any problems with overwriting the .bash_history file via a script? For example, if you remove an apache log file I think you need to send a nohup / reset signal with kill to have it flush it's connection to the file. If that is the case with the .bash_history file, perhaps I could somehow use ps to check and make sure there are no connected sessions before the filtering script is run?

  • 3
    Try ignoredups instead of erasedups for a while and see how that works for you. – jw013 Sep 20 '12 at 15:54
  • 1
    I don't think bash holds an open file handle to the history file - it reads/writes it when it needs to, so it should (note - should - I haven't tested) be safe to overwrite it from elsewhere. – D_Bye Sep 20 '12 at 19:00
  • 1
    I just learned something new on the 1st sentence of your question. Good trick! – Ricardo Dec 1 '16 at 0:48
  • I'm failing to find the man page for all the options to the history command. Where should I be looking? – Jonathan Hartley 5 hours ago
  • History options are in 'man bash', search for 'shell builtin commands' section, then for 'history' below that. – Jonathan Hartley 5 hours ago
35

Sorting the history

This command works like sort|uniq, but keeps the lines in place

nl|sort -k 2|uniq -f 1|sort -n|cut -f 2

Basically, prepends to each line its number. After sort|uniq-ing, all lines are sorted back according to their original order (using the line number field) and the line number field is removed from the lines.

This solution has the flaw that it is undefined which representative of a class of equal lines will make it in the output and therefore its position in the final output is undefined. However, if the latest representative should be chosen you can sort the input by a second key:

nl|sort -k2 -k 1,1nr|uniq -f1|sort -n|cut -f2

Managing .bash_history

For re-reading and writing back the history, you can use history -a and history -w respectively.

  • 5
    A version of decorate-sort-undecorate, implemented with shell tools. Nice. – ire_and_curses Sep 20 '12 at 17:21
  • With sort, the -r switch always reverses the sorting order. But this won't yield the result you have in mind. sort regards the two occurrences of ls as identical with the result that, even when reversed, the eventual order depends on the sorting algorithm. But see my update for another idea. – artistoex Sep 20 '12 at 19:29
  • 1
    In case, you don't want to modify .bash_history, you could put the following in .bashrc: alias history='history | sort -k2 -k 1,1nr | uniq -f 1 | sort -n' – Nathan Jan 15 '14 at 20:35
  • What is nl at the beginning of each code line? Shouldn't it be history? – A.L Feb 4 '15 at 9:50
  • 1
    @A.L nl adds line numbers. The command as a whole solves the general problem: removing duplicates while preserving order. The input is read from stdin. – artistoex Feb 5 '15 at 21:24
48

So I was looking for the same exact thing after being annoyed by duplicates, and found that if I edit my ~/.bash_profile (Mac) with:

export HISTCONTROL=ignoreboth:erasedups

It does exactly what you wanted, it only keeps the latest of any command. ignoreboth is actually just like doing ignorespace:ignoredups and that along with erasedups gets the job done.

At least on my Mac terminal with bash this work perfect. Found it here on askubuntu.com.

  • 10
    this should be correct answer – MitchBroadhead Mar 3 '16 at 10:55
  • tested on Max OS X Yosemite and on Ubuntu 14_04 – Ricardo Dec 1 '16 at 1:12
  • 1
    agree with @MitchBroadhead. this solves the problem within bash itself, without external cron-job. tested it on ubuntu 17.04 and 16.04 LTS – Georg Jung Jul 29 '17 at 8:10
  • works on OpenBSD too. It only removes dups of any command it is appending to the history file, which is fine for me. It has the interesting effect of shortening the history file as I enter commands that had existed as duplicates before. Now I can make my history file max shorter. – WeakPointer Jan 19 '18 at 14:09
  • 1
    This only ignores duplicate, consecutive commands. If you alternate repeatedly between two given commands, your bash history will fill up with duplicates – Dylanthepiguy Dec 22 '18 at 0:39
15

Found this solution in the wild and tested:

awk '!x[$0]++'

The first time a specific value of a line ($0) is seen, the value of x[$0] is zero.
The value of zero is inverted with ! and becomes one.
An statement that evaluates to one causes the default action, which is print.

Therefore, the first time an specific $0 is seen, it is printed.

Every next time (the repeats) the value of x[$0] has been incrented,
its negated value is zero, and a statement that evaluates to zero doesn't print.

To keep the last repeated value, reverse the history and use the same awk:

awk '!x[$0]++' ~/.bash_history                 # keep the first value repeated.

tac ~/.bash_history | awk '!x[$0]++' | tac     # keep the last.
  • Wow! That just worked. But it removes all but the first occurrence I guess. I'd reversed the ordering of the lines using Sublime Text before running this. Now I'll reverse it again to get a clean history with only the last occurrence of all duplicates left behind. Thank you. – trss Aug 27 '14 at 20:05
  • Check out my answer! – Ali Shakiba Jan 19 '15 at 10:26
  • Nice clean and general answer (not restricted to the history use-case) without launching a bazilion sub-processes ;-) – JepZ Nov 14 '18 at 0:08
9

Extending Clayton answer:

tac $HISTFILE | awk '!x[$0]++' | tac | sponge $HISTFILE

tac reverse the file, make sure you have installed moreutils so you have sponge available, otherwise use a temp file.

  • 1
    For those on Mac, use brew install coreutils, and notice that all the GNU utils have a g prepended to avoid confusion with the BSD built-in Mac commands (e.g. gsed is GNU whereas sed is BSD). So use gtac. – tralston Jun 11 '15 at 20:26
  • I needed history -c and history -r to get it to use the history – drescherjm Oct 8 at 19:32
4

These would keep the last duplicated lines:

ruby -i -e 'puts readlines.reverse.uniq.reverse' ~/.bash_history
tac ~/.bash_history | awk '!a[$0]++' | tac > t; mv t ~/.bash_history
3

This is an old post, but a perpetual issue for users who want to have multiple terminals open, and have the history synched between windows, but not duplicated.

My solution in .bashrc:

shopt -s histappend
export HISTCONTROL=ignoreboth:erasedups
export PROMPT_COMMAND="history -n; history -w; history -c; history -r"
tac "$HISTFILE" | awk '!x[$0]++' > /tmp/tmpfile  &&
                tac /tmp/tmpfile > "$HISTFILE"
rm /tmp/tmpfile
  • histappend option adds the history of the buffer to the end of the history file ($HISTFILE)
  • ignoreboth and erasedups prevent duplicate entries from being saved in the $HISTFILE
  • The prompt command updates the history cache
    • history -n reads all lines from $HISTFILE that may have occurred in a different terminal since the last carriage return
    • history -w writes the updated buffer to $HISTFILE
    • history -c wipes the buffer so no duplication occurs
    • history -r re-reads the $HISTFILE, appending to the now blank buffer
  • the awk script stores the first occurrence of each line it encounters. tac reverses it, and then reverses it back so that it can be saved with the most recent commands still most recent in the history
  • rm the /tmp file

Every time you open a new shell, the history has all dupes wiped, and every time you hit the Enter key in a different shell/terminal window, it updates this history from the file.

  • Here is an excellent explanation to this in the comments – smilingfrog Jan 26 '18 at 8:09
  • If "ignoreboth and erasedups prevent dupes from being saved", then why do you also need to do the "awk" command to remove dupes from the file? Is it because "ignoreboth and erasedups" only prevent consecutive dupes being saved? Sorry to be pedantic, I'm just trying to understand. – Jonathan Hartley 5 hours ago
0

To uniqely record every new command is tricky. First you need to add to ~/.profile or similar:

HISTCONTROL=erasedups
PROMPT_COMMAND='history -w'

Then you need to add to ~/.bash_logout:

history -a
history -w
  • Can you help me understand why, on logout, you need to append unwritten history to the history file before then rewriting the whole history file? Can't you just write the entire file without the 'append'? – Jonathan Hartley 5 hours ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.