3

Trying to implement smart backups of my zsh terminal history (~/.zsh_history). Same goes for bash no difference just the file is called .bash_history.

So far I have decided to create a private git repo, since my history can contain secrets and such. I have created an ansible script (can be done with a bash script) which creates monthly dirs and basically copies each day the contents of the ~/.zsh_history into the monthly dir with the daily timestamp as the filename. I've made that script into daily cron job.

The issue is there is a lot of duplication. I initially thought I can just delete the contents of the history file after the backup is taken successfully, but then I realized that would mean no recursive searches. Now I am thinking about creating a functionality that only backs up the changes to the history compared to the day before. If today's history is commands (1, 2, 3) and tomorrow it is (1 , 2, 3, 4, 5) for the second day's backup we only take (4, 5) not the whole thing. Kind of like the intersection of two sets (yesterday's history file is one set and today's is another set (bigger set than contains yesterday's set!)

What's the best way to go about this? I didn't find other examples for a similar project so any help is appreciated.

Here's several lines from the history file for a reference:

: 1659932134:0;git commit -m "First initial best commit"
: 1659932139:0;git push
: 1659932147:0;git push -u origin main
: 1659932151:0;git push
: 1660008827:0;cat ~/.zshrc
: 1660008832:0;cat ~/.zsh_history

Next day for example the file would look like:

: 1659932134:0;git commit -m "First initial best commit"
: 1659932139:0;git push
: 1659932147:0;git push -u origin main
: 1659932151:0;git push
: 1660008827:0;cat ~/.zshrc
: 1660008832:0;cat ~/.zsh_history
+++++++++ ( this is just me adding to show the difference)
: 1659932113:0;git add .gitignore
: 1659932117:0;git add backups/
: 1659932119:0;git add hist_backup.yml
7
  • 1
    What you mean: "Now the issue is there is a lot of duplication?" Duplication of what: lines in zsh_history, daily files? What you commit into your repo: zsh_history or those daily files? Why do you need those monthly dirs with daily files? git is already smart enough, simply commit your zsh_history directly into repo on daily basis.
    – gapsf
    Commented Aug 9, 2022 at 4:55
  • @gapsf tbh that's so simple and great. Maybe that's what I will just do. Ofc one issue is that i will have one giant file, and won't have daily commands files. Meaning i can't be like oh let me check what i ran on the 26 th of march 2021. I can I guess if i run a script to convert epoch times to regular and then do grep for the date.
    – dobro
    Commented Aug 9, 2022 at 16:27
  • Maybe this helps cyberciti.biz/faq/unix-linux-bash-history-display-date-time
    – gapsf
    Commented Aug 9, 2022 at 17:18
  • 1
    Please do not cross post on several StackExchange sites (as here). Thanks. Commented Aug 10, 2022 at 14:22
  • 1
    @Zeitounator sorry about that. I'm knew here and wasn't familiar.
    – dobro
    Commented Aug 10, 2022 at 18:03

1 Answer 1

0

Regarding

So what's the best way to go about this

since Log all commands run (by admins on production servers) is a quite common requirement, auditd might be the right tool here.

Regarding

I didn't find other examples for a similar project

Because of timestamps, probably your requirement to be able to search, it seem that a kind of database would suit better to your use case than git. So it might be possible to go with an approach like in Logging bash history in a database.

Regarding

How to do smart backups?

you could also have a look into incremental backups via rsync.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .