Memoizing/caching command line output

Question

I have a bash script I use to run a few python and C++ programs in a sequence.

Each program takes in some input parameters which I define in the bash script. So as an example I run the program like this:

echo $param1 $param2 $param3 | python foo.py

The python program outputs some values which we use as input for later programs. The thing, like I said above, is that I do not need to run the python program if I store the values in some file and read them from there.

So my question then is. Is there some generic tool that achieves this feature? That is, is there some program called 'bar' which I could run like

bar $param1 $param2 $param3 "python foo.py"

which would check if a cache file exists, if yes it would check if the program has been run with the given parameters and if yes it would output the cached output values instead of running the program again.

Edit: The name of the log file could also be provided as input of course.

Answer 1

I just got nerd-sniped into writing a rather complete script for this; latest version is at https://gist.github.com/akorn/51ee2fe7d36fa139723c851d87e56096.

Advantages over sivann's shell implementation:

• also takes envvars into account when computing cache key;
• completely avoids race conditions using locking instead of relying on a random wait
• better performance when called in tight loop due to fewer forks
• also caches stderr
• completely transparent: prints nothing; doesn't prevent parallel execution of same command; just runs command uncached if there is a problem with the cache
• configurable via envvars and command line switches
• can prune its cache (remove all obsolete entries)

Disadvantage: written in zsh, not bash.

#!/bin/zsh
#
# Purpose: run speficied command with specified arguments and cache result. If cache is fresh enough, don't run command again but return cached output.
# Also cache exit status and stderr.
# Copyright (c) 2019-2020 András Korn; License: GPLv3

# Use silly long variable names to avoid clashing with whatever the invoked program might use
RUNCACHED_MAX_AGE=${RUNCACHED_MAX_AGE:-300}
RUNCACHED_IGNORE_ENV=${RUNCACHED_IGNORE_ENV:-0}
RUNCACHED_IGNORE_PWD=${RUNCACHED_IGNORE_PWD:-0}
[[ -n "$HOME" ]] && RUNCACHED_CACHE_DIR=${RUNCACHED_CACHE_DIR:-$HOME/.runcached}
RUNCACHED_CACHE_DIR=${RUNCACHED_CACHE_DIR:-/var/cache/runcached}

function usage() {
    echo "Usage: runcached [--ttl <max cache age>] [--cache-dir <cache directory>]"
    echo "       [--ignore-env] [--ignore-pwd] [--help] [--prune-cache]"
    echo "       [--] command [arg1 [arg2 ...]]"
    echo
    echo "Run 'command' with the specified args and cache stdout, stderr and exit"
    echo "status. If you run the same command again and the cache is fresh, cached"
    echo "data is returned and the command is not actually run."
    echo
    echo "Normally, all exported environment variables as well as the current working"
    echo "directory are included in the cache key. The --ignore options disable this."
    echo "The OLDPWD variable is always ignored."
    echo
    echo "--prune-cache deletes all cache entries older than the maximum age. There is"
    echo "no other mechanism to prevent the cache growing without bounds."
    echo
    echo "The default cache directory is ${RUNCACHED_CACHE_DIR}."
    echo "Maximum cache age defaults to ${RUNCACHED_MAX_AGE}."
    echo
    echo "CAVEATS:"
    echo
    echo "Side effects of 'command' are obviously not cached."
    echo
    echo "There is no cache invalidation logic except cache age (specified in seconds)."
    echo
    echo "If the cache can't be created, the command is run uncached."
    echo
    echo "This script is always silent; any output comes from the invoked command. You"
    echo "may thus not notice errors creating the cache and such."
    echo
    echo "stdout and stderr streams are saved separately. When both are written to a"
    echo "terminal from cache, they will almost certainly be interleaved differently"
    echo "than originally. Ordering of messages within the two streams is preserved."
    exit 0
}

while [[ -n "$1" ]]; do
    case "$1" in
        --ttl)      RUNCACHED_MAX_AGE="$2"; shift 2;;
        --cache-dir)    RUNCACHED_CACHE_DIR="$2"; shift 2;;
        --ignore-env)   RUNCACHED_IGNORE_ENV=1; shift;;
        --ignore-pwd)   RUNCACHED_IGNORE_PWD=1; shift;;
        --prune-cache)  RUNCACHED_PRUNE=1; shift;;
        --help)     usage;;
        --)     shift; break;;
        *)      break;;
    esac
done

zmodload zsh/datetime
zmodload zsh/stat
zmodload zsh/system
zmodload zsh/files

# the built-in mv doesn't fall back to copy if renaming fails due to EXDEV;
# since the cache directory is likely on a different fs than the tmp
# directory, this is an important limitation, so we use /bin/mv instead
disable mv  

mkdir -p "$RUNCACHED_CACHE_DIR" >/dev/null 2>/dev/null

((RUNCACHED_PRUNE)) && find "$RUNCACHED_CACHE_DIR/." -maxdepth 1 -type f \! -newermt @$[EPOCHSECONDS-RUNCACHED_MAX_AGE] -delete 2>/dev/null

[[ -n "$@" ]] || exit 0 # if no command specified, exit silently

(
    # Almost(?) nothing uses OLDPWD, but taking it into account potentially reduces cache efficency.
    # Thus, we ignore it for the purpose of coming up with a cache key.
    unset OLDPWD
    ((RUNCACHED_IGNORE_PWD)) && unset PWD
    ((RUNCACHED_IGNORE_ENV)) || env
    echo -E "$@"
) | md5sum | read RUNCACHED_CACHE_KEY RUNCACHED__crap__
# make the cache dir hashed unless a cache file already exists (created by a previous version that didn't use hashed dirs)
if ! [[ -f $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.exitstatus ]]; then
    RUNCACHED_CACHE_KEY=$RUNCACHED_CACHE_KEY[1,2]/$RUNCACHED_CACHE_KEY[3,4]/$RUNCACHED_CACHE_KEY[5,$]
    mkdir -p "$RUNCACHED_CACHE_DIR/${RUNCACHED_CACHE_KEY:h}" >/dev/null 2>/dev/null
fi

# If we can't obtain a lock, we want to run uncached; otherwise
# 'runcached' wouldn't be transparent because it would prevent
# parallel execution of several instances of the same command.
# Locking is necessary to avoid races between the mv(1) command
# below replacing stderr with a newer version and another instance
# of runcached using a newer stdout with the older stderr.
: >>$RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.lock 2>/dev/null
if zsystem flock -t 0 $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.lock 2>/dev/null; then
    if [[ -f $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.stdout ]]; then
        if [[ $[EPOCHSECONDS-$(zstat +mtime $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.stdout)] -le $RUNCACHED_MAX_AGE ]]; then
            cat $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.stdout &
            cat $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.stderr >&2 &
            wait
            exit $(<$RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.exitstatus)
        else
            rm -f $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.{stdout,stderr,exitstatus} 2>/dev/null
        fi
    fi

    # only reached if cache didn't exist or was too old
    if [[ -d $RUNCACHED_CACHE_DIR/. ]]; then
        RUNCACHED_tempdir=$(mktemp -d 2>/dev/null)
        if [[ -d $RUNCACHED_tempdir/. ]]; then
            $@ >&1 >$RUNCACHED_tempdir/${RUNCACHED_CACHE_KEY:t}.stdout 2>&2 2>$RUNCACHED_tempdir/${RUNCACHED_CACHE_KEY:t}.stderr
            RUNCACHED_ret=$?
            echo $RUNCACHED_ret >$RUNCACHED_tempdir/${RUNCACHED_CACHE_KEY:t}.exitstatus 2>/dev/null
            mv $RUNCACHED_tempdir/${RUNCACHED_CACHE_KEY:t}.{stdout,stderr,exitstatus} $RUNCACHED_CACHE_DIR/${RUNCACHED_CACHE_KEY:h} 2>/dev/null
            rmdir $RUNCACHED_tempdir 2>/dev/null
            exit $RUNCACHED_ret
        fi
    fi
fi

# only reached if cache not created successfully or lock couldn't be obtained
exec $@

Answer 2

An implementation exists here: https://bitbucket.org/sivann/runcached/src Caches executable path, output, exit code, remembers arguments. Configurable expiration. Implemented in bash, C, python, choose whatever suits you. The "bash" version is somewhat limited.

Answer 3

For Bash users I created bash-cache, which provides a similar feature set to András Korn's zsh approach. It supports env vars, avoids races, caches stderr and exit codes, invalidates stale data, support asynchronous warming and mutex locks, and more.

For example I think you can implement what you're describing like so:

foo() {
  python foo.py <<<"$*" # this writes all arguments to the python stdin
} && bc::cache foo

Which you'd then invoke as:

foo param1 param2 param3

And foo will cache the results and reuse them in subsequent calls.

See my earlier answer for more details.

Answer 4

Simple python solution based on existing packages (joblib.Memory for memoizing):

cmdcache.py:

import sys, os
import joblib
import subprocess

mem = joblib.Memory('.', verbose=False)
@mem.cache
def run_cmd(args, env):
    process = subprocess.run(args, capture_output=True, text=True)
    return process.stdout, process.stderr, process.returncode

stdout, stderr, returncode = run_cmd(sys.argv[1:], dict(os.environ))
sys.stdout.write(stdout)
sys.stdout.write(stderr)
sys.exit(returncode)

Usage example:

python cmdcache.py ls