2

Hope this is the right place to ask.

Is there a unix tool that works similarly to this?

# invokes echo, saves to cache (using command with arguments as key), returns "hi"
cache 10 echo hi 

# wait 2 seconds
sleep 2

# doesn't invoke echo, from cache returns "hi"
cache 10 echo hi 

# wait 10 seconds
sleep 10

# with cache expired, invokes echo, returns "hi"
cache 10 echo hi 

Obviously echo wouldn't be the real use-case.

Basically cache STDOUT, STDERR and status for a given command+arguments, so the next call to the same process doesn't have to re-run it.

I could write a script that does it but I wondered if there was one in the unix toolset that I don't know about.

  • 1
    Not that I know of, and I have a hard time figuring out a real use-case for this. Do you have something specific in mind? – Mat Jan 3 '17 at 14:12
  • Usually if people are doing something that requires performance to be considered they'll use something other than shell scripting. Shell scripting is mainly for simple stuff where you just need to run particular commands in a way that's too involved to manually type it out each time. You need a real scripting language. – Bratchley Jan 3 '17 at 16:33
  • Looking online there are examples of people using memcached via netcat you could also use /dev/shm and flat files but again that's more of a novelty thing than something you should really be doing. – Bratchley Jan 3 '17 at 16:36
  • There's not much call for such a tool because there aren't many commands whose output depends solely on their arguments (as opposed to file contents, data received from the network, user input, etc.). For things that depend only on file contents, there's make. – Gilles Jan 3 '17 at 23:28
1

You could put the result in a file and then read it back from that file...

tmpDir=/tmp/$$
rm -rf "$tmpDir"
mkdir "$tmpDir"

echo cmd1 > "$tmpDir"/cmd1_stdout 2> "$tmpDir"/cmd1_stderr
echo $? > "$tmpDir"/cmd1_exitcode

# Retrieving output of cmd1:
( cat "$tmpDir"/cmd1_stdout ; cat "$tmpDir"/cmd1_stderr 1>&2; exit $(cat "$tmpDir"/cmd1_exitcode) )

From this we could define a "cache" function. This version need a character that we will never use as argument. For example the comma ",". You can change it at the line "IFS=,"

tmpDir=/tmp/$$
rm -rf "$tmpDir"
mkdir "$tmpDir"

cache() {

 IFS=, cmd="$*"
 if [ -f "$tmpDir/$cmd"_exitcode ]; then 
   cat "$tmpDir/$cmd"_stdout
   cat "$tmpDir/$cmd"_stderr 1>&2
   return $(cat "$tmpDir"/cmd1_exitcode)
 fi

   # This line is bash-only:
 "$@" 2> >(tee "$tmpDir/$cmd"_stderr 1>&2) > >(tee "$tmpDir/$cmd"_stdout)
 local e=$?
 echo $e > "$tmpDir/$cmd"_exitcode

 return $e
}

The timeout could be implemented with "date +%s" and "stat -c %Y" :

tmpDir=/tmp/$$
rm -rf "$tmpDir"
mkdir "$tmpDir"

cache() {

 local timeout=$1
 shift

 IFS=, cmd="$*"
 if [ -f "$tmpDir/$cmd"_exitcode ]; then 

   local now=$(date +%s)
   local fdate=$(stat -c %Y "$tmpDir/$cmd"_exitcode)

   if [ $((now-fdate)) -le $timeout ]; then 
     cat "$tmpDir/$cmd"_stdout
     cat "$tmpDir/$cmd"_stderr 1>&2
     return $(cat "$tmpDir/$cmd"_exitcode)
   fi

 fi

   # This line is bash-only:
 "$@" 2> >(tee "$tmpDir/$cmd"_stderr 1>&2) > >(tee "$tmpDir/$cmd"_stdout)
 local e=$?
 echo $e > "$tmpDir/$cmd"_exitcode

 return $e
}

The "bash only" line could be replaced by :

  "$@" 2> "$tmpDir/$cmd"_stderr > "$tmpDir/$cmd"_stdout
  local e=$?
  cat "$tmpDir/$cmd"_stdout
  cat "$tmpDir/$cmd"_stderr 1>&2
  • /tmp isn't necessarily going to be tmpfs and so the pages could be evicted from the filesystem cache or create an undesirable dependency (booting from network, etc). If you're going to use flat files it's probably better to use /dev/shm or something that's always stored in RAM. – Bratchley Jan 3 '17 at 16:41
  • Also, minor quibbles with your specific implementation: 1) $$ by itself probably isn't unique enough since some PID's can be pretty small and you're rm -rf-ing in a global directory after all. Probably better to have a static prefix for this script like tmpDir=/dev/shm/bashCache.$$ or something – Bratchley Jan 3 '17 at 16:48
  • 1
    2) You really should have eviction happen parallel to script execution and not on each save/retrieval of individual items otherwise your cache could become huge and unlike the filesystem cache the memory could just permanently stay used even if there's another process that needs it and the cache entry has expired – Bratchley Jan 3 '17 at 16:48
  • @Bratchley on macOS I get ls: /dev/shm: No such file or directory – Dan2552 Jan 3 '17 at 18:37
  • @Dan2552 then use a ramdisk if you're on OS X. Point being that you need to ensure the pages don't get swapped out to disk and stay in memory. I can't edit the answer to make it work since that'd be a substantive change and kind of Vouze's call on whether he wants to include it. – Bratchley Jan 3 '17 at 19:33
0

To my knowledge, there is no standard or traditional tool which works this way. However, searching for "cache stdout" brought me to https://github.com/Valiev/cache, which appears to be a tool which does what you want.

  • This requires to install Python !! Everything could be done in shell, without any additional tool. – Vouze Jan 3 '17 at 16:14
  • 1
    Sure. But if you have Python installed, there you go. – mattdm Jan 3 '17 at 16:18
0

The most common mechanism for caching command output in shell scripts is to assign it to a variable. This is easily done with a subshell. It doesn't expire in the way a traditional cache does, but those who write shell scripts most often find it acceptable. Here's your above script using a subshell and variable

HI=$(echo hi)
echo $HI
sleep 2
echo $HI
sleep 10
echo $HI

Another alternative is to create a cache function in shell script. Something like ...

cache() {
expiry=$1
cmd=$2-
cache=/tmp/{$2-}_cache

if test "`find -not -newermt '-30 seconds' -delete ${cache}`"; then
$cmd |tee "$cache"
else
cat "$cache"
fi
}
  • If the variable is not an environment variable (not exported), then it should be lower-case. This is a useful tradition, to avoid bugs. – ctrl-alt-delor Dec 11 '18 at 7:52
0

I cant post a comment so I'me just going to crate a new post.

The Problem analyzes

As far as i know there is no shell comand (part of the corr of bash) that is used to cash data... how ever that dose not mean that there are no means to implement it...

How ever how you want to actually implement it is entirely dependent on you're distributions enviroment and what file structure it is is build on. Filesystem Hierarchy Standard (wiki)

Go to the section about FHS compliance, and what it will basically tell you is not all Linux distribution are created equally (case and point different shell version across different distribution aka bourn shell vs kornshell)... Somme may include the /dev/shm Somme may not... Thats why you got the message:

ls: /dev/shm: No such file or directory

you're distrubution apparently is one with out the file /dev/shm !!!


Although most:

Most Linux distributions follow the Filesystem Hierarchy Standard and declare it their own policy to maintain FHS compliance.

Conclusion ?!

So Now my question what is a good way to deal with this conundrum? well i would say its rather simple...

Create a list off all potential directory which store in memory instead of on the hdd and create a if elif conditional where you check if a file exists or not.Should one of the files exist (which is more then likley, unless they changed the file structure) You can then just simlpy declare a variable called temp and fill it with the path of the tmpfs (temporary filesystem)

To give you a gist of it

#!/bin/bash                                                          

#/run directory as a temporary filesystem (tmpfs) 
#which stores volatile runtime data, following the FHS version 3.0.
if [ -e /run ]
    then
        temp='/run'
# with FHS version 2.3, such data were stored in /var/run
#for compatibility there might be a simlink
elif [ -e /var/run ]
    then
        temp='/var/run'
elif [ -e /dev/shm/]
    then
        temp='/var/shm'
elif [ -e /var/tmp ]
    then
        #this should not occur with the current fhs standerts
        #switching to /var/tmp just in case ;)
        temp='/var/tmp'
fi
echo $temp

This is just to give you a basic idea (I'm still working on it my self(for my final project ;)

Additionally but i believe this could be helpful for others. What you could now do is just create a file in the directory with the name of the process followed by a time stamp to create a cronjob to automatically clean up you're temporary fills in regular intervals or just by calling the rm command when you no longer require the file.

PS If you find any further tmpfs pleas just go a head and edit this post Thanks.

0

I'm not aware of a generic tool for doing this sort of caching, but I'm aware of a specific tool for caching the result of compiling C or C++ files, the ccache compiler cache.

This stores the preprocessed sources and compiled object code files, and uses hashes based on both the source code and the compiler signature to figure out whether to use the cache or to recompile a file.

A generic cache for the shell would have to similarly hash the script, the signature of each external utility used, etc., and also take into account that the output of a shell script may depend on the time of day (if it uses date) and many many other factors.

Creating a cache for a single command would not be beneficial, especially not for echo since this is built into most shells and the extra processing of it for caching would only slow it down (unless what was echoed also called heavy external utilities through command substitutions etc.)

I don't think it would be worthwhile in the end, but possibly only useful in a very specific context, such as the ccache utility for caching compilations, which is a problem slightly more limited in its scope.

Also note that although a utility may produce some output, it may also have side-effects (like writing to files etc.) that are not easily cached.

  • ccache only works with gcc and only knows how to cache the compilation of a single C/C++/Objective-C/Objective-C++ file – Vouze Jan 3 '17 at 16:17
0

I just wrote a rather complete script for this; latest version is at https://gist.github.com/akorn/51ee2fe7d36fa139723c851d87e56096.

#!/bin/zsh
#
# Purpose: run speficied command with specified arguments and cache result. If cache is fresh enough, don't run command again but return cached output.
# Also cache exit status and stderr.
# License: GPLv3

# Use silly long variable names to avoid clashing with whatever the invoked program might use
RUNCACHED_MAX_AGE=${RUNCACHED_MAX_AGE:-300}
RUNCACHED_IGNORE_ENV=${RUNCACHED_IGNORE_ENV:-0}
RUNCACHED_IGNORE_PWD=${RUNCACHED_IGNORE_PWD:-0}
[[ -n "$HOME" ]] && RUNCACHED_CACHE_DIR=${RUNCACHED_CACHE_DIR:-$HOME/.runcached}
RUNCACHED_CACHE_DIR=${RUNCACHED_CACHE_DIR:-/var/cache/runcached}

function usage() {
    echo "Usage: runcached [--ttl <max cache age>] [--cache-dir <cache directory>]"
    echo "       [--ignore-env] [--ignore-pwd] [--help] [--prune-cache]"
    echo "       [--] command [arg1 [arg2 ...]]"
    echo
    echo "Run 'command' with the specified args and cache stdout, stderr and exit"
    echo "status. If you run the same command again and the cache is fresh, cached"
    echo "data is returned and the command is not actually run."
    echo
    echo "Normally, all exported environment variables as well as the current working"
    echo "directory are included in the cache key. The --ignore options disable this."
    echo "The OLDPWD variable is always ignored."
    echo
    echo "--prune-cache deletes all cache entries older than the maximum age. There is"
    echo "no other mechanism to prevent the cache growing without bounds."
    echo
    echo "The default cache directory is ${RUNCACHED_CACHE_DIR}."
    echo "Maximum cache age defaults to ${RUNCACHED_MAX_AGE}."
    echo
    echo "CAVEATS:"
    echo
    echo "Side effects of 'command' are obviously not cached."
    echo
    echo "There is no cache invalidation logic except cache age (specified in seconds)."
    echo
    echo "If the cache can't be created, the command is run uncached."
    echo
    echo "This script is always silent; any output comes from the invoked command. You"
    echo "may thus not notice errors creating the cache and such."
    echo
    echo "stdout and stderr streams are saved separately. When both are written to a"
    echo "terminal from cache, they will almost certainly be interleaved differently"
    echo "than originally. Ordering of messages within the two streams is preserved."
    exit 0
}

while [[ -n "$1" ]]; do
    case "$1" in
        --ttl)      RUNCACHED_MAX_AGE="$2"; shift 2;;
        --cache-dir)    RUNCACHED_CACHE_DIR="$2"; shift 2;;
        --ignore-env)   RUNCACHED_IGNORE_ENV=1; shift;;
        --ignore-pwd)   RUNCACHED_IGNORE_PWD=1; shift;;
        --prune-cache)  RUNCACHED_PRUNE=1; shift;;
        --help)     usage;;
        --)     shift; break;;
        *)      break;;
    esac
done

zmodload zsh/datetime
zmodload zsh/stat

# This is racy, but the race is harmless; at worst, the program is run uncached 
# because the cache is unusable. Testing for directory existence saves an
# mkdir(1) execution in the common case, improving performance infinitesimally;
# it could matter if runcached is run from inside a tight loop.
# Hide errors so that runcached itself is transparent (doesn't mix new messages 
# into whatever the called program outputs).
[[ -d "$RUNCACHED_CACHE_DIR/." ]] || mkdir -p "$RUNCACHED_CACHE_DIR" >/dev/null 2>/dev/null

((RUNCACHED_PRUNE)) && find "$RUNCACHED_CACHE_DIR/." -maxdepth 1 -type f \! -newermt @$[EPOCHSECONDS-RUNCACHED_MAX_AGE] -delete 2>/dev/null

[[ -n "$@" ]] || exit 0 # if no command specified, exit silently

(
    # Almost(?) nothing uses OLDPWD, but taking it into account potentially reduces cache efficency.
    # Thus, we ignore it for the purpose of coming up with a cache key.
    unset OLDPWD
    ((RUNCACHED_IGNORE_PWD)) && unset PWD
    ((RUNCACHED_IGNORE_ENV)) || env
    echo -E "$@"
) | md5sum | read RUNCACHED_CACHE_KEY RUNCACHED__crap__

# Unfortunately, I couldn't find a less convoluted way of getting rid of an error message when trying to open a logfile in a nonexistent cache directory.
exec {RUNCACHED_temp_stderr} >&2
exec 2>/dev/null
exec {RUNCACHED_LOCK_FD}>>$RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.lock
exec 2>&$RUNCACHED_temp_stderr
exec {RUNCACHED_temp_stderr}>&-

# If we can't obtain a lock, we want to run uncached; otherwise
# 'runcached' wouldn't be transparent because it would prevent
# parallel execution of several instances of the same command.
# Locking is necessary to avoid races between the mv(1) command
# below replacing stderr with a newer version and another instance
# of runcached using a newer stdout with the older stderr.
if flock -n $RUNCACHED_LOCK_FD 2>/dev/null; then
    if [[ -f $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.stdout ]]; then
        if [[ $[EPOCHSECONDS-$(zstat +mtime $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.stdout)] -le $RUNCACHED_MAX_AGE ]]; then
            cat $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.stdout &
            cat $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.stderr >&2 &
            wait
            exit $(<$RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.exitstatus)
        else
            rm -f $RUNCACHED_CACHE_DIR/$RUNCACHED_CACHE_KEY.{stdout,stderr,exitstatus} 2>/dev/null
        fi
    fi

    # only reached if cache didn't exist or was too old
    if [[ -d $RUNCACHED_CACHE_DIR/. ]]; then
        RUNCACHED_tempdir=$(mktemp -d 2>/dev/null)
        if [[ -d $RUNCACHED_tempdir/. ]]; then
            $@ >&1 >$RUNCACHED_tempdir/$RUNCACHED_CACHE_KEY.stdout 2>&2 2>$RUNCACHED_tempdir/$RUNCACHED_CACHE_KEY.stderr
            RUNCACHED_ret=$?
            echo $RUNCACHED_ret >$RUNCACHED_tempdir/$RUNCACHED_CACHE_KEY.exitstatus 2>/dev/null
            mv $RUNCACHED_tempdir/$RUNCACHED_CACHE_KEY.{stdout,stderr,exitstatus} $RUNCACHED_CACHE_DIR/ 2>/dev/null
            rmdir $RUNCACHED_tempdir 2>/dev/null
            exit $RUNCACHED_ret
        fi
    fi
fi

# only reached if cache not created successfully or lock couldn't be obtained
exec $@
0

The mainframe world call{ed,s} this checkpoint / restart. Not for performance issues, but rather to avoid losing the computer time investment in long-running jobs. These utilities usually also capture file positions, etc., for restarting jobs.

A search for linux checkpoint restart returns some 400+K hits. One of the interesting ones is https://en.wikipedia.org/wiki/CRIU and a companion https://criu.org/Comparison_to_other_CR_projects

Best wishes ... cheers, drl

-1

use this cmd https://github.com/jingminglang/cmd_cache

git clone https://github.com/jingminglang/cmd_cache

go build main.go

./main -c [your command] -t [cache time]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.