I have generated multiple .bash_history files by switching between snapshots on a VM. I would like to recover all these files into the single live .bash_history file in my currently running instance.

The files have timestamps in them and I would like to preserve them as much as possible. However, it looks like some lines don't have timestamps on them. For example:

#1542817796
vi ~/.bash_profile 
set +x
cd -
cd w/Edge-Agent/edge-agent-ged/
make 
set -x
man make
make -npq
make -npq .DEFAULT
make 
make 
ack npq /etc
ack npq /usr/share/
set +x
ack npq /usr/share/
sudo vi /usr/share/bash-completion/completions/make
dv
hg diff 
vi ~/.bashrc
#1542826064
ls
#1542826066
vi ~/.bashrc
#1542826074
cd ..
#1542826321

I would like to order the merged list by timestamp because I was switching back and forth between the snapshots and the files interleave with each other. If there's no timestamp available for a command we can infer it from the last timestamp before and the first timestamp after.

  • What are the existing bash_history filenames to merge? – Jeff Schaller Dec 5 at 0:42
  • edgeos_history.[012] – Stephen Rasku Dec 5 at 0:43
  • How to read that sample? Any command without a time stamp needs the previous time stamp assigned? – RudiC Dec 5 at 14:07
  • @RudiC sure. That works – Stephen Rasku Dec 5 at 14:08
up vote 0 down vote accepted

Using GNU m4 (which supports numbered diversions up to LONG_MAX) with some additional sed to prepare the input:

#!/bin/sh

{ cat <<"EOF"
m4_define(m4_chunk,`m4_divert(-1)m4_undivert($1)m4_divert($1)#$1')m4_dnl
m4_define(m4_,``m4_'')m4_dnl
m4_define(`m4_LQ',`m4_changequote([,])`m4_dnl'
m4_changequote`'')m4_dnl
m4_define(`m4_RQ',`m4_changequote([,])m4_dnl`
'm4_changequote`'')m4_dnl
EOF
sed -e "s/m4_/m4_()/g;s/\`/m4_LQ()/g;s/'/m4_RQ()/g;s/m4_/\`'m4_/g" -e 's/^#\([0-9]\+\)$/m4_chunk(\1)/'
} | m4 -P

Usage:

cat edgeos_history.* | sort_history.sh > merged_history

The m4_chunk function does most of the work, as the separate blocks are "undiverted" in ascending numerical order when all the input ends. The rest of the script is to protect any quotemarks and existing m4_ sequences in the input from being interpreted by m4.

LQ and RQ definitions based on those in Michael Breen's Notes on the M4 Macro Language.

  • Does it have to be left-justified? Just curious. – Stephen Rasku Dec 5 at 14:36
  • The lines inside the here-doc do, because m4 passes whitespace through unchanged. – JigglyNaga Dec 5 at 14:45
  • Some of the lines are in more than one file so I'm getting duplicate lines like this: #1542121355 scripts/run-system-tests-local.sh -d srasku0 -s TC263808 | ts %T #1542121355 scripts/run-system-tests-local.sh -d srasku0 -s TC263808 | ts %T . Is there a way to resolve this? If not, I could write a simple Perl script to post-process the file. – Stephen Rasku Dec 6 at 15:33
  • You could prepend m4_divert(-1)m4_undivert($1) to the definition of m4_chunk, so that any duplicate timestamp discards any lines previously seen with that same timestamp. – JigglyNaga Dec 6 at 17:00
  • Cool. That works. – Stephen Rasku Dec 6 at 17:11

Why not just do something like cat /path/to/one/.bash_history >> /path/to/live/.bash_history then you could remove duplicate enteries with awk '!seen[$0]++' .bash_history >> .bash_history2 then remove the original and rename .bash_history2 to .bash_history. However removing the duplicates like that will leave empty timestamps.

  • I would like to sort by time stamp. I’ll make that clearer on my question. – Stephen Rasku Dec 4 at 23:48
  • Yea that makes it more complicated – Michael Prokopec Dec 5 at 0:10

Would this help:

awk '/^#[0-9]*$/ {TS = $0; next} {print TS, $0}' file* | sort
#1542817796 ack npq /etc
#1542817796 ack npq /usr/share/
#1542817796 ack npq /usr/share/
#1542817796 cd -
#1542817796 cd w/Edge-Agent/edge-agent-ged/
#1542817796 dv
#1542817796 hg diff 
#1542817796 make 
#1542817796 make 
#1542817796 make 
#1542817796 make -npq
.
.
.

Make sure to list ALL history files as input stream to awk.

EDIT: Some non-satisfying, non-portable essays to retain the original order of the commands:

awk '/^#[0-9]*$/ {TS = $0; next} {print TS, NR, $0}' file* | sort  -k1,1 -k2,2n

.

awk '/^#[0-9]*$/ {TS = $0; next} {print TS,  $0}' file* | sort  -mk1,1 
  • Since this uses the same timestamp for all consecutive untimestamped entries it will sort those entries lexically instead of by the original order in the file. I think this can be improved by incrementing the inferred timestamp somehow. – Stephen Rasku Dec 5 at 14:23
  • I edited two more options in, none of which is necessarily satisfying nor portable. – RudiC Dec 5 at 14:32

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.