How can I check if a file can be created or truncated/overwritten in bash?

Question

The user calls my script with a file path that will be either be created or overwritten at some point in the script, like foo.sh file.txt or foo.sh dir/file.txt.

The create-or-overwrite behavior is much like the requirements for putting the file on the right side of the > output redirect operator, or passing it as an argument to tee (in fact, passing it as an argument to tee is exactly what I'm doing).

Before I get into the guts of the script, I want to make a reasonable check if the file can be created/overwritten, but not actually create it. This check doesn't have to be perfect, and yes I realize that the situation can change between the check and the point where the file is actually written - but here I'm OK with a best effort type solution so I can bail out early in the case that the file path is invalid.

Examples of reasons the file couldn't created:

• the file contains a directory component, like dir/file.txt but the directory dir doesn't exist
• the user doens't have write permissions in the specified directory (or the CWD if no directory was specified

Yes, I realize that checking permissions "up front" is not The UNIX Way™, rather I should just try the operation and ask forgiveness later. In my particular script however, this leads to a bad user experience and I can't change the responsible component.

Answer 1

The obvious test would be:

if touch /path/to/file; then
    : it can be created
fi

But it does actually create the file if it's not already there. We could clean up after ourselves:

if touch /path/to/file; then
    rm /path/to/file
fi

But this would remove a file that already existed, which you probably don't want.

We do, however, have a way around this:

if mkdir /path/to/file; then
    rmdir /path/to/file
fi

You can't have a directory with the same name as another object in that directory. I can't think of a situation in which you'd be able to create a directory but not create a file. After this test, your script would be free to create a conventional /path/to/file and do whatever it pleases with it.

Answer 2

From what I'm gathering, you want to check that when using

tee -- "$OUT_FILE"

(note the -- or it wouldn't work for file names that start with -), tee would succeed to open the file for writing.

That is that:

• the length of the file path doesn't exceed the PATH_MAX limit
• the file exists (after symlink resolution) and is not of type directory and you have write permission to it.
• if the file doesn't exist, the dirname of the file exists (after symlink resolution) as a directory and you have write and search permission to it and the filename length doesn't exceed the NAME_MAX limit of the filesystem that directory resides in.
• or the file is a symlink that points to a file that doesn't exist and is not a symlink loop but meets the criteria just above

We'll ignore for now filesystems like vfat, ntfs or hfsplus that have limitations on what byte values file names may contain, disk quota, process limit, selinux, apparmor or other security mechanism in the way, full filesystem, no inode left, device files that can't be opened that way for a reason or another, files that are executables currently mapped in some process address space all of which could also affect the ability to open or create the file.

With zsh:

zmodload zsh/system
tee_would_likely_succeed() {
  local file=$1 ERRNO=0 LC_ALL=C
  if [ -d "$file" ]; then
    return 1 # directory
  elif [ -w "$file" ]; then
    return 0 # writable non-directory
  elif [ -e "$file" ]; then
    return 1 # exists, non-writable
  elif [ "$errnos[ERRNO]" != ENOENT ]; then
    return 1 # only ENOENT error can be recovered
  else
    local dir=$file:P:h base=$file:t
    [ -d "$dir" ] &&    # directory
      [ -w "$dir" ] &&  # writable
      [ -x "$dir" ] &&  # and searchable
      (($#base <= $(getconf -- NAME_MAX "$dir")))
    return
  fi
}

In bash or any Bourne-like shell, just replace the

zmodload zsh/system
tee_would_likely_succeed() {
  <zsh-code>
}

with:

tee_would_likely_succeed() {
  zsh -s -- "$@" << 'EOF'
  zmodload zsh/system
  <zsh-code>
EOF
}

The zsh-specific features here are $ERRNO (which exposes the error code of the last system call) and $errnos[] associative array to translate to the corresponding standard C macro names. And the $var:h (from csh) and $var:P (needs zsh 5.3 or above).

bash doesn't have equivalent features yet.

$file:h can be replaced with dir=$(dirname -- "$file"; echo .); dir=${dir%??}, or with GNU dirname: IFS= read -rd '' dir < <(dirname -z -- "$file").

For $errnos[ERRNO] == ENOENT, an approach could be to run ls -Ld on the file and check whether the error message corresponds to the ENOENT error. Doing that reliably and portably is tricky though.

One approach could be:

msg_for_ENOENT=$(LC_ALL=C ls -d -- '/no such file' 2>&1)
msg_for_ENOENT=${msg_for_ENOENT##*:}

(assuming that the error message ends with the syserror() translation of ENOENT and that that translation doesn't include a :) and then, instead of [ -e "$file" ], do:

err=$(ls -Ld -- "$file" 2>&1)

And check for a ENOENT error with

case $err in
  (*:"$msg_for_ENOENT") ...
esac

The $file:P part is the trickiest to achieve in bash, especially on FreeBSD.

FreeBSD does have a realpath command and a readlink command that accepts a -f option, but they can't be used in the cases where the file is a symlink that doesn't resolve. That's the same with perl's Cwd::realpath().

python's os.path.realpath() does appear to work similarly to zsh $file:P, so assuming that at least one version of python is installed and that there is a python command that refers to one of them (which is not a given on FreeBSD), you could do:

dir=$(python -c '
import os, sys
print(os.path.realpath(sys.argv[1]) + ".")' "$dir") || return
dir=${dir%.}

But then, you might as well do the whole thing in python.

Or you could decide not to handle all those corner cases.

Answer 3

One option you might want to consider is creating the file early on but only populating it later in your script. You can use the exec command to open the file in a file descriptor (such as 3, 4, etc.) and then later use a redirection to a file descriptor (>&3, etc.) to write contents to that file.

Something like:

#!/bin/bash

# Open the file for read/write, so it doesn't get
# truncated just yet (to preserve the contents in
# case the initial checks fail.)
exec 3<>dir/file.txt || {
    echo "Error creating dir/file.txt" >&2
    exit 1
}

# Long checks here...
check_ok || {
    echo "Failed checks" >&2
    # cleanup file before bailing out
    rm -f dir/file.txt
    exit 1
}

# We're ready to write, first truncate the file.
# Use "truncate(1)" from coreutils, and pass it
# /dev/fd/3 so the file doesn't need to be reopened.
truncate -s 0 /dev/fd/3

# Now populate the file, use a redirection to write
# to the previously opened file descriptor.
populate_contents >&3

You can also use a trap to clean up the file on error, that's a common practice.

This way, you get a real check for permissions that you'll be able to create the file, while at the same time being able to perform it early enough that if that fails you haven't spent time waiting for the long checks.

---

UPDATED: In order to avoid clobbering the file in case the checks fail, use bash's fd<>file redirection which does not truncate the file right away. (We don't care about reading from the file, this is just a workaround so we don't truncate it. Appending with >> would probably just work too, but I tend to find this one a bit more elegant, keeping the O_APPEND flag out of the picture.)

When we're ready to replace the contents, we need to truncate the file first (otherwise if we're writing fewer bytes than there were in the file before, the trailing bytes would stay there.) We can use the truncate(1) command from coreutils for that purpose, and we can pass it the open file descriptor we have (using the /dev/fd/3 pseudo-file) so it doesn't need to reopen the file. (Again, technically something simpler like : >dir/file.txt would probably work, but not having to reopen the file is a more elegant solution.)

Answer 4

I think DopeGhoti's solution is better but this should also work:

file=$1
if [[ "${file:0:1}" == '/' ]]; then
    dir=${file%/*}
elif [[ "$file" =~ .*/.* ]]; then
    dir="$(PWD)/${file%/*}"
else
    dir=$(PWD)
fi

if [[ -w "$dir" ]]; then
    echo "writable"
    #do stuff with writable file
else
    echo "not writable"
    #do stuff without writable file
fi

The first if construct checks if the argument is a full path (starts with /) and sets the dir variable to the directory path up to the last /. Otherwise if the argument does not start with a / but does contain a / (specifying a sub directory) it will set dir to the present working directory + the sub directory path. Otherwise it assumes the present working directory. It then checks if that directory is writable.

Answer 5

What about using normal test command like outlined below?

FILE=$1

DIR=$(dirname $FILE) # $DIR now contains '.' for file names only, 'foo' for 'foo/bar'

if [ -d $DIR ] ; then
  echo "base directory $DIR for file exists"
  if [ -e $FILE ] ; then
    if [ -w $FILE ] ; then
      echo "file exists, is writeable"
    else
      echo "file exists, NOT writeable"
    fi
  elif [ -w $DIR ] ; then
    echo "directory is writeable"
  else
    echo "directory is NOT writeable"
  fi
else
  echo "can NOT create file in non-existent directory $DIR "
fi

Answer 6

You mentioned user experience was driving your question. I'll answer from a UX angle, since you've got good answers on the technical side.

Rather than performing the check up-front, how about writing the results into a temporary file then at the very end, placing the results into the user's desired file? Like:

userfile=${1:?Where would you like the file written?}
tmpfile=$(mktemp)

# ... all the complicated stuff, writing into "${tmpfile}"

# fill user's file, keeping existing permissions or creating anew
# while respecting umask
cat "${tmpfile}" > "${userfile}"
if [ 0 -eq $? ]; then
    rm "${tmpfile}"
else
    echo "Couldn't write results into ${userfile}." >&2
    echo "Results available in ${tmpfile}." >&2
    exit 1
fi

The good with this approach: it produces the desired operation in the normal happy path scenario, side-steps the test-and-set atomicity issue, preserves permissions of the target file while creating if necessary, and is dead simple to implement.

Note: had we used mv, we'd be keeping the permissions of the temporary file -- we don't want that, I think: we want to keep the permissions as set on the target file.

Now, the bad: it requires twice the space (cat .. > construct), forces the user to do some manual work if the target file wasn't writable at the time it needed to be, and leaves the temporary file laying around (which might have security or maintenance issues).

Answer 7

TL;DR:

: >> "${userfile}"

Unlike <> "${userfile}" or touch "${userfile}", this will not make any spurious modifications to the file's timestamps and will also work with write-only files.

---

From the OP:

I want to make a reasonable check if the file can be created/overwritten, but not actually create it.

And from your comment to my answer from a UX perspective:

The overwhelming majority of the time this fails is because the path provided is not valid. I can't reasonably prevent some rogue user some concurrently modifying the FS to break the job in the middle of the operation, but I certainly can check the #1 failure cause of an invalid or not writable path.

The only reliable test is to open(2) the file, because only that resolves every question about the writeability: path, ownership, filesystem, network, security context, etc. Any other test will address some part of writeability, but not others. If you want a subset of tests, you'll ultimately have to choose what's important to you.

But here's another thought. From what I understand:

1. the content creation process is long-running, and
2. the target file should be left in a consistent state.

You're wanting to do this pre-check because of #1, and you don't want to fiddle with an existing file because of #2. So why don't you just ask the shell to open the file for append, but don't actually append anything?

$ tree -ps
.
├── [dr-x------        4096]  dir_r
├── [drwx------        4096]  dir_w
├── [-r--------           0]  file_r
└── [-rw-------           0]  file_w

$ for p in file_r dir_r/foo file_w dir_w/foo; do : >> $p; done
-bash: file_r: Permission denied
-bash: dir_r/foo: Permission denied

$ tree -ps
.
├── [dr-x------        4096]  dir_r
├── [drwx------        4096]  dir_w
│   └── [-rw-rw-r--           0]  foo
├── [-r--------           0]  file_r
└── [-rw-------           0]  file_w

Under the hood, this resolves the writeability question exactly as wanted:

open("dir_w/foo", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3

but without modifying the file's contents or metadata. Now, yes, this approach:

• doesn't tell you if the file is append only, which might be a problem when you go about updating it at the end of your content creation. You can detect this, to a degree, with lsattr and react accordingly.
• creates a file that didn't previously exist, if such is the case: mitigate this with a selective rm.

While I contend (in my other answer) that the most user-friendly approach is to create a temporary file the user has to move, I think this is the least user-hostile approach to fully vet their input.