21

A month ago I wrote a Python script to map MAC and IP addresses from stdin. And two days ago I remembered it and used to filter output of tcpdump but it went wrong because of a typo. I typed

tcpdump -ne > ./mac_ip.py

and the output is nothing. But the output should be "Unknown" if it can't parse the input, so I did cat ./mac_ip.py and found all the tcpdump data instead of the program. Then I realized that I should use

tcpdump -ne | ./mac_ip.py

Is there any way to get my program back? Anyways I can write my program again, but if it happens again with more important program I should be able to do something. OR is there any way to tell output redirection to check for the file and warn if it is an executable?

12
  • 19
    You can get your program back from the last backup before your overwrite, otherwise not. BTW in shell you can specify set -o noglobber and bash won't redirect into existing files anymore. See here for details: cyberciti.biz/tips/howto-keep-file-safe-from-overwriting.html
    – eckes
    Jun 24 '17 at 14:46
  • 12
    You shouldn't have write permission to important executables ... Jun 24 '17 at 15:36
  • 20
    @eckes set -o noclobber
    – GnP
    Jun 24 '17 at 16:17
  • 38
    @HagenvonEitzen I hate advice like that, as if you've set proper ownership and permissions on every one-off shell and python script you've ever wrote before running it (and, of course, back again briefly if you have to edit it). It's only marginally more meaningful than "You shouldn't type > when you mean |." Don't forget reality.
    – Jason C
    Jun 24 '17 at 16:21
  • 30
    Git repos are cheap. Commit all your code, no matter how small and meaningless and then a mistake like this is a quick and easy fix.
    – casey
    Jun 24 '17 at 20:32
22

Sadly I suspect you'll need to rewrite it. (If you have backups, this is the time to get them out. If not, I would strongly recommend you set up a backup regime for the future. Lots of options available, but off topic for this answer.)

I find that putting executables in a separate directory, and adding that directory to the PATH is helpful. This way I don't need to reference the executables by explicit path. My preferred programs directory for personal (private) scripts is "$HOME"/bin and it can be added to the program search path with PATH="$HOME/bin:$PATH". Typically this would be added to the shell startup scripts .bash_profile and/or .bashrc.

Finally, there's nothing stopping you removing write permission for yourself on all executable programs:

touch some_executable.py
chmod a+x,a-w some_executable.py    # chmod 555, if you prefer

ls -l some_executable.py
-r-xr-xr-x+ 1 roaima roaima 0 Jun 25 18:33 some_executable.py

echo "The hunting of the Snark" > ./some_executable.py
-bash: ./some_executable.py: Permission denied
8
  • 2
    /usr/local/bin is the standard location for user-created executables and scripts
    – gardenhead
    Jun 24 '17 at 18:07
  • 4
    @gardenhead It depends on how the system is set up. /usr/local is intended for host-specific things (as opposed to a directory shared across hosts via a network mount), and may or may not be writeable by non-root users.
    – chepner
    Jun 24 '17 at 20:29
  • 4
    @gardenhead it's one standard location, certainly. I use /use/local/bin for locally installed scripts and programs that are likely to be used by multiple user accounts, and $HOME/bin for things personal to a single user. There's value in both.
    – roaima
    Jun 24 '17 at 23:05
  • 1
    Note that Fedora seems to be trying to push using $HOME/.local/bin
    – Zan Lynx
    Jun 25 '17 at 6:49
  • 1
    @Zan eeeww! Seriously though, thank you. Looks like RH is trying to push everything into ~/.local as that's yet another item moved from its "traditional" place.
    – roaima
    Jun 25 '17 at 8:01
38

To prevent existing files from being overwritten by redirection > use the noclobber option in bash or any POSIX-like shell (also in (t)csh where the feature actually originated, though you do set noclobber instead of set -o noclobber/set -C there). Then, if you need to force to replace a file, use the >| redirection operator (>! in (t)csh).

Example:

$ echo abc > file
$ set -o noclobber
$ echo xyz > file
bash: file: cannot overwrite existing file
$ echo xyz >| file
$ cat file
xyz

BTW, you can check the current settings with set -o:

$ set -o
...
monitor         on
noclobber       on
noexec          off
...
2
  • While this perfectly answers the question, I wouldn't recommend it. 1. Typing >| instead of | isn't much less probable than typing >. 2. It is easy and highly advisable to make backups (an editor worth its name can save the last version; there's cron, etc.). 3. Every piece of code should be put under version control, even tiny scripts. YMMV.
    – maaartinus
    Jun 25 '17 at 17:33
  • 2
    @maaartinus come on, 1) typing two separate characters instead of one is clearly less probable. 2) Obviously backups are essential, nobody advised the OP not to make backups, this answer in no way suggests not having backups, and editor backups assume you've edited the file in an editor. 3) Again, you are only thinking of code the OP has written, as in this particular example, but the question and this answer is applicable to any file on the machine, including system executables.
    – terdon
    Jun 26 '17 at 10:09
8

I strongly advise to have the important scripts under a git repo, synced remotely (a fancy self-hosted platform will do), as @casey's comment says.

This way you are protected from bad human mistakes like reverting the file to the previous working state and execute it again.

4

Is the file recoverable?

Short answer: Not usually.

@Mark Plotnick points out in the comments, you can recover .py files from .pyc using Uncompyle. This should be perfect for your situation.

In general, though, this is much harder. Theoretically you can use forensics tools to undelete files. Probably the easiest I've used is testdisk (aka "PhotoRec"). It only works sometimes and it is a slow process. It's usually not worth it, so, yes, it's possible, but the real answer is "no".

Can > be changed to not overwrite executables?

No. There is no standard way to tell the shell to never redirect only for files marked executable. There is "noclobber" which will prevent redirecting into existing files, executable or not, but see my comments on that below.

What to do in the future?

  1. This might sounds silly, but to prevent future mistakes, you probably need do nothing. My bet is that you've already learned this lesson.

    I've been using and teaching Unix for a very long time and while people often make this mistake once, they rarely repeat it. Why not? Likely for the same reason a person experienced with knives doesn't cut themselves: humans are good at learning. Eventually, doing the right thing becomes second nature.

  2. Use a text editor that makes backups for you. For example, if you use emacs, the previous version of your program is saved in mac_ip.py~. Other editors can be configured to work similarly (e.g., "set backup" in .nanorc). For editors that don't support automatic backups, you could make a simplistic function in your .bashrc:

    myeditor() { cp -p "$1" "$1~";  editor "$1"; }
    
  3. Make it easy for yourself to make copies. For example, in the directory of the project you're working on, you might have a Makefile with a target like this:

    # Use `make tar` to backup all files in this directory.
    # Tar filename will be ../<currentdirectory>-<date>.tar.gz 
    DIRNAME = $(shell basename `pwd`)
    TIMESTAMP = $(shell date +%s)
    tar:
        @echo "[Tarring up ${DIRNAME}.tar.gz]"
        (cd .. ; tar -zcvf "${DIRNAME}-${TIMESTAMP}.tar.gz" "${DIRNAME}")
    

    (Note: stackexchange is misrendering the TABs above as 4 spaces.)

  4. Similarly, you could create a Makefile target that does an rsync to a remote Unix host that you have ssh access to. (Use ssh-copy-id so you won't be asked for your password repeatedly.)

  5. Use git. There are many excellent tutorials on getting started. Try man gittutorial, man gittutorial-2 and man giteveryday. Setting up your own git repository isn't hard, but you can also create a remote repository at no cost at github.com

  6. If the above solutions are too heavy weight, you can save small scripts to gist.github.com. While it's possible to paste or upload from a web browser, I recommend using a command line gist interface to make things super easy.

I strongly discourage using "noclobber".

Yes, if you choose, you can do set -o noclobber so you'll get error messages whenever you try to overwrite an existing file. This is a bad idea, in my opinion.*

It makes the shell work in a non-standard way with no visible indication whether it is enabled. You have to use a different syntax for doing normal things. Worst of all, if you get used to noclobber, then some day you'll use another Unix machine without noclobber and this sort of accident might happen again.

As you probably know, the Unix shell was designed to be a sharp tool for experts. It's fast to use and won't get in your way — and it will cut you if you forget which end is pointy. But, the more you use it, the more I think you'll appreciate that that can be a good thing.


* Footnote: perhaps take my opinions with a grain of salt. I'm also the kind of person who thinks bicycle training wheels are a bad idea.

2
  • I've also taught Unix for a while. Many of my students have never learned appreciate Unix's direct simplicity; I tell them they're not alone, and can at least still learn while commiserating over the Unix Hater's Handbook, which maps out some of the minefield for them. simson.net/ref/ugh.pdf
    – Jason
    Jun 25 '17 at 23:25
  • Also: I agree - training wheels on a bicycle are helpful for anyone learning how to ride a tricycle.
    – Jason
    Jun 25 '17 at 23:27
2

You may have been able to recover the data after it first occurred if you had recently viewed or edited the script and it was still in the memory buffer. Otherwise, you are pretty much out of luck.

If you piped to tee to write to a file (as well as STDOUT) instead of > (or tee -a instead of >>), then you could easily replace tee with an alias, function, or symlink to a script that warns the user if the file they are about to write to is executable.

The following is by no means ideal and could be improved on a lot, but it is a starting point, just as an example of how this is possible:

wee.sh:

#!/bin/bash

if [ -n "${2}" ]; then
  if [ "$(ls -l "${2}" | awk '{print $1}' | grep x)" ]; then
    echo executable
  else
    tee -a "${2}"
  fi
elif [ "$(ls -l "${1}" | awk '{print $1}' | grep x)" ]; then
  echo executable
else
  tee "${1}"
fi

...then just echo 'alias tee="/path/to/wee.sh"' >> ~/.bashrc or something similar.

On the bright side, at least you will get more practice and the second version of your Python script will probably be much better than the first!

1

You didn't specify whether you are working on a PC or a server. If your files happen to be stored on a dedicated file server, then there are often automatic backups ("snapshots") being kept by the (OS on the) file server hardware.

Under Linux

The virtual, hidden snapshot directory exists in every directory in your file system.

Try:

cd .snapshot   
ls -l

If that directory exists, then you may be in luck. You should see a series of directories that hold backups stored automatically at certain points in time. The names indicate the relative time in the past at which the snapshot was stored. For example:

hourly.0
hourly.1
hourly.2
hourly.3
hourly.4
hourly.5
nightly.0
nightly.1
nightly.2
nightly.3
nightly.4
nightly.5
nightly.6
weekly.0
weekly.1
weekly.2

Go into any timepoint directory that is old enough (before your file-overwrite mistake). Inside the timepoint directory, you should see the state of the ../.. directory (and all subdirectories) as of that point in the past.

cd nightly.6
ls  # look around   
tee < mac_ip.py  # check for the correct content
cp mac_ip.py ~/safekeeping/mac_ip.py  # save the old file

Notes:

  1. ls -a will not show the .snapshot directory; you must name it explictly. It is inserted virtually by the file server. It does not exist as a real directory in your filesystem.
  2. These automatic snapshots are a rolling history. Old changes eventually fall off the end and are lost. You need to use this technique as soon as possible after you realize that you need a file back.

Under Windows

The hidden snapshot directory may be named ~snapshot and exist only at the root level of a given drive.

Advice

Snapshots are a safety net that work most of the time, but not every time. I concur with the other recommendations to use a version control system (such as git) even for trivial files.

1

It has been said before, and I will say it again. Use a revision control system.

Backups are for recovering a hardware failure. Revision control is for situations like yours (and it has many other uses). Revision control tools allow you to keep a history of a file, and to go back to any point in that history.

Examples of revision control tools include subversion (SVN) ( a bit old now, but still good-ish), mercurial (hg), and git (git) (hard to use). svn is good for office documents, and other um-mergables, git and hg have surpassed it for most other roles. hg and git allow you to work off-line, and to synchronise with a remote server, for distribution and backup.

Read up on revision control, then distributed revision control, and then try them.

1
  • I agree that using revision control is best for situations like mine, but giving the right permissions to files is equally important Jul 20 '17 at 4:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.