Accidentally used the output redirection > instead of a pipe |

Question

A month ago I wrote a Python script to map MAC and IP addresses from stdin. And two days ago I remembered it and used to filter output of tcpdump but it went wrong because of a typo. I typed

tcpdump -ne > ./mac_ip.py

and the output is nothing. But the output should be "Unknown" if it can't parse the input, so I did cat ./mac_ip.py and found all the tcpdump data instead of the program. Then I realized that I should use

tcpdump -ne | ./mac_ip.py

Is there any way to get my program back? Anyways I can write my program again, but if it happens again with more important program I should be able to do something. OR is there any way to tell output redirection to check for the file and warn if it is an executable?

Answer 1

To prevent existing files from being overwritten by redirection > use the noclobber option in bash or any POSIX-like shell (also in (t)csh where the feature actually originated, though you do set noclobber instead of set -o noclobber/set -C there). Then, if you need to force to replace a file, use the >| redirection operator (>! in (t)csh).

Example:

$ echo abc > file
$ set -o noclobber
$ echo xyz > file
bash: file: cannot overwrite existing file
$ echo xyz >| file
$ cat file
xyz

BTW, you can check the current settings with set -o:

$ set -o
...
monitor         on
noclobber       on
noexec          off
...

Answer 2

Sadly I suspect you'll need to rewrite it. (If you have backups, this is the time to get them out. If not, I would strongly recommend you set up a backup regime for the future. Lots of options available, but off topic for this answer.)

I find that putting executables in a separate directory, and adding that directory to the PATH is helpful. This way I don't need to reference the executables by explicit path. My preferred programs directory for personal (private) scripts is "$HOME"/bin and it can be added to the program search path with PATH="$HOME/bin:$PATH". Typically this would be added to the shell startup scripts .bash_profile and/or .bashrc.

Finally, there's nothing stopping you removing write permission for yourself on all executable programs:

touch some_executable.py
chmod a+x,a-w some_executable.py    # chmod 555, if you prefer

ls -l some_executable.py
-r-xr-xr-x+ 1 roaima roaima 0 Jun 25 18:33 some_executable.py

echo "The hunting of the Snark" > ./some_executable.py
-bash: ./some_executable.py: Permission denied

Answer 3

I strongly advise to have the important scripts under a git repo, synced remotely (a fancy self-hosted platform will do), as @casey's comment says.

This way you are protected from bad human mistakes like reverting the file to the previous working state and execute it again.

Answer 4

Is the file recoverable?

Short answer: Not usually.

@Mark Plotnick points out in the comments, you can recover .py files from .pyc using Uncompyle. This should be perfect for your situation.

In general, though, this is much harder. Theoretically you can use forensics tools to undelete files. Probably the easiest I've used is testdisk (aka "PhotoRec"). It only works sometimes and it is a slow process. It's usually not worth it, so, yes, it's possible, but the real answer is "no".

Can > be changed to not overwrite executables?

No. There is no standard way to tell the shell to never redirect only for files marked executable. There is "noclobber" which will prevent redirecting into existing files, executable or not, but see my comments on that below.

What to do in the future?

1. This might sounds silly, but to prevent future mistakes, you probably need do nothing. My bet is that you've already learned this lesson.

   I've been using and teaching Unix for a very long time and while people often make this mistake once, they rarely repeat it. Why not? Likely for the same reason a person experienced with knives doesn't cut themselves: humans are good at learning. Eventually, doing the right thing becomes second nature.

2. Use a text editor that makes backups for you. For example, if you use emacs, the previous version of your program is saved in mac_ip.py~. Other editors can be configured to work similarly (e.g., "set backup" in .nanorc). For editors that don't support automatic backups, you could make a simplistic function in your .bashrc:

   myeditor() { cp -p "$1" "$1~";  editor "$1"; }
   

3. Make it easy for yourself to make copies. For example, in the directory of the project you're working on, you might have a Makefile with a target like this:

   # Use `make tar` to backup all files in this directory.
   # Tar filename will be ../<currentdirectory>-<date>.tar.gz 
   DIRNAME = $(shell basename `pwd`)
   TIMESTAMP = $(shell date +%s)
   tar:
       @echo "[Tarring up ${DIRNAME}.tar.gz]"
       (cd .. ; tar -zcvf "${DIRNAME}-${TIMESTAMP}.tar.gz" "${DIRNAME}")
   

   (Note: stackexchange is misrendering the TABs above as 4 spaces.)

4. Similarly, you could create a Makefile target that does an rsync to a remote Unix host that you have ssh access to. (Use ssh-copy-id so you won't be asked for your password repeatedly.)

5. Use git. There are many excellent tutorials on getting started. Try man gittutorial, man gittutorial-2 and man giteveryday. Setting up your own git repository isn't hard, but you can also create a remote repository at no cost at github.com

6. If the above solutions are too heavy weight, you can save small scripts to gist.github.com. While it's possible to paste or upload from a web browser, I recommend using a command line gist interface to make things super easy.

I strongly discourage using "noclobber".

Yes, if you choose, you can do set -o noclobber so you'll get error messages whenever you try to overwrite an existing file. This is a bad idea, in my opinion.^*

It makes the shell work in a non-standard way with no visible indication whether it is enabled. You have to use a different syntax for doing normal things. Worst of all, if you get used to noclobber, then some day you'll use another Unix machine without noclobber and this sort of accident might happen again.

As you probably know, the Unix shell was designed to be a sharp tool for experts. It's fast to use and won't get in your way — and it will cut you if you forget which end is pointy. But, the more you use it, the more I think you'll appreciate that that can be a good thing.

---

^* Footnote: perhaps take my opinions with a grain of salt. I'm also the kind of person who thinks bicycle training wheels are a bad idea.

Answer 5

You may have been able to recover the data after it first occurred if you had recently viewed or edited the script and it was still in the memory buffer. Otherwise, you are pretty much out of luck.

If you piped to tee to write to a file (as well as STDOUT) instead of > (or tee -a instead of >>), then you could easily replace tee with an alias, function, or symlink to a script that warns the user if the file they are about to write to is executable.

The following is by no means ideal and could be improved on a lot, but it is a starting point, just as an example of how this is possible:

wee.sh:

#!/bin/bash

if [ -n "${2}" ]; then
  if [ "$(ls -l "${2}" | awk '{print $1}' | grep x)" ]; then
    echo executable
  else
    tee -a "${2}"
  fi
elif [ "$(ls -l "${1}" | awk '{print $1}' | grep x)" ]; then
  echo executable
else
  tee "${1}"
fi

...then just echo 'alias tee="/path/to/wee.sh"' >> ~/.bashrc or something similar.

On the bright side, at least you will get more practice and the second version of your Python script will probably be much better than the first!

Answer 6

You didn't specify whether you are working on a PC or a server. If your files happen to be stored on a dedicated file server, then there are often automatic backups ("snapshots") being kept by the (OS on the) file server hardware.

Under Linux

The virtual, hidden snapshot directory exists in every directory in your file system.

Try:

cd .snapshot   
ls -l

If that directory exists, then you may be in luck. You should see a series of directories that hold backups stored automatically at certain points in time. The names indicate the relative time in the past at which the snapshot was stored. For example:

hourly.0
hourly.1
hourly.2
hourly.3
hourly.4
hourly.5
nightly.0
nightly.1
nightly.2
nightly.3
nightly.4
nightly.5
nightly.6
weekly.0
weekly.1
weekly.2

Go into any timepoint directory that is old enough (before your file-overwrite mistake). Inside the timepoint directory, you should see the state of the ../.. directory (and all subdirectories) as of that point in the past.

cd nightly.6
ls  # look around   
tee < mac_ip.py  # check for the correct content
cp mac_ip.py ~/safekeeping/mac_ip.py  # save the old file

Notes:

1. ls -a will not show the .snapshot directory; you must name it explictly. It is inserted virtually by the file server. It does not exist as a real directory in your filesystem.
2. These automatic snapshots are a rolling history. Old changes eventually fall off the end and are lost. You need to use this technique as soon as possible after you realize that you need a file back.

Under Windows

The hidden snapshot directory may be named ~snapshot and exist only at the root level of a given drive.

Advice

Snapshots are a safety net that work most of the time, but not every time. I concur with the other recommendations to use a version control system (such as git) even for trivial files.

Answer 7

It has been said before, and I will say it again. Use a revision control system.

Backups are for recovering a hardware failure. Revision control is for situations like yours (and it has many other uses). Revision control tools allow you to keep a history of a file, and to go back to any point in that history.

Examples of revision control tools include subversion (SVN) ( a bit old now, but still good-ish), mercurial (hg), and git (git) (hard to use). svn is good for office documents, and other um-mergables, git and hg have surpassed it for most other roles. hg and git allow you to work off-line, and to synchronise with a remote server, for distribution and backup.

Read up on revision control, then distributed revision control, and then try them.