Does bash have a built in caching command available (sort of like mktemp or sponge)?

Question

I'm using the amazon ec2 command line tools, and the ec2-describe-instances tool is a little painful because of the 2-5 seconds it requires to make the request and display the output.

I'm looking into using the fec2din tool described in this question to format the output of ec2-describe-instances and was wondering what the best way would be to cache the output from the call.

fec2din uses mktemp to create a temporary file and then uses awk to format the output.

Does is there some tool I could use with a TTL paramater that would only run ec2-describe-instances if the timestamp on the cache file was older than a certain time?

It would be neat if there was a little utility to do this (like sponge does for helping with stdout).

Answer 1

While this does not address your question as such, I can highly recommend using Amazon EC2 via the excellent boto instead, which is a Python package that provides interfaces to Amazon Web Services.

It pretty much covers the same ground as the Amazon EC2 API Tools, but doesn't suffer from the painful delays due to relying on the modern and fast AWS REST APIs, while the EC2 API Tools are written in Java and used to use the old and slow SOAP APIs (don't know whether they might have changed gears in this regard already, but your experience as well as the still required AWS X.509 Certificates seem to suggest otherwise).

In addition, you don't need to use these AWS X.509 Certificates anymore, rather can use the nowadays more common and flexible approach via an AWS Access Key ID and an AWS Secret Access Key, which might as well (and usually should be) provided via AWS Identity and Access Management (IAM) in order to avoid exposing your main AWS account credentials.

On top of that, boto it is an obvious candidate for orchestrating your everyday AWS usage via Python scripts - this can as well be done with bash of course, but you get the idea ;)

Documentation

You can find documentation and examples in boto: A Python interface to Amazon Web Services, which provides decent (i.e. more or less complete) API References (e.g. for EC2) as well as dedicated introductory articles explaining the basic usage for several services (but not all yet), e.g. An Introduction to boto’s EC2 interface covers the use case at hand.

In addition you may want to read Boto Config for setting up your environment (credentials etc.).

Usage

You can explore boto via the Python Read–eval–print loop (REPL), i.e. by starting python.

Once you are satisfied with your fragments you can convert them into a Python script for standalone usage.

Example

Here is a sample approximately addressing your use case (it assumes you have setup the credentials in your environment, as explained in Boto Config):

$ python
Python 2.7.2 (default, Jun 12 2011, 14:24:46)
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto
>>> ec2 = boto.connect_ec2()
>>> instances = ec2.get_all_instances()
>>> instances
[Reservation:r-916d01f2, Reservation:r-3f7e055c, Reservation:r-c37209a0]

Okay, get_all_instances() actually returned a list of boto.ec2.instance.Reservation, so here is an annoying indirection in place (stemming from the EC2 API), which you won't see elsewhere usually - the docs are conclusive already, but let's see how to find that out by introspection:

>>> dir(instances[0])
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', 
'__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', 
'__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 
'connection', 'endElement', 'groups', 'id', 'instances', 'item', 'owner_id', 
'region', 'startElement', 'stop_all']
>>> insts = instances[0].instances
>>> insts
[Instance:i-5d9a593a]

That's more like it, so finally you want to see the attribute values of i-5d9a593a (most attributes omitted for brevity and privacy):

>>> vars(insts[0])
{'kernel': u'aki-825ea7eb', 'private_dns_name': '', 'id': u'i-5d9a593a', 
'monitored': False, 'state': u'stopped', 'architecture': u'x86_64',  
 'public_dns_name': '', 'ip_address': None, 'placement': u
'us-east-1a', 'ami_launch_index': u'0', 'dns_name': '', 'region': RegionInfo:us-east-1
# ...
}

Not quite, but Python's Data pretty printer (pprint) to the rescue:

>>> import pprint
>>> pp = pprint.PrettyPrinter(indent=4)
>>> pp.pprint(vars(insts[0])) {   
    '_in_monitoring_element': False,
    'ami_launch_index': u'0',
    'architecture': u'x86_64',
    'dns_name': '',
    'hypervisor': u'xen',
    'id': u'i-5d9a593a',
    'instance_class': None,
    'instance_type': u'm1.medium',
    'ip_address': None,
    'kernel': u'aki-825ea7eb'
    # ...
    }

Answer 2

A makeshift sponge utility can be made by reading lines into an array and then outputting it, for example:

sponge() {
    local line lines
    while IFS= read -r line; do
        lines+=( "$line" )
    done
    printf '%s\n' "${lines[@]}"
}

Then run it as command1 | sponge | command2.

Answer 3

Here's a very crude caching facility for bash/ksh/zsh.

typeset -A output_cache
cache () {
  local IFS='
  ' ret=0
  if [[ -z ${output_cache["$*"]} ]]; then
    output_cache["$*"]=$(unset IFS; "$@")
    ret=$?
  fi
  echo "${output_cache["$*"]}"
  return $ret
}
uncache () {
  local IFS='
  '
  unset output_cache["$*"]
}

Example:

$ cache mycommand --options        # takes a while
…
$ cache mycommand --options        # instantaneous
…
$ uncache mycommand --options      # remove a cache entry

Limitations:

• This only makes sense if running the command twice produces the same output each time.
• The cache is not shared between shell instances.
• This only works for simple commands, not pipelines, loops, etc. (unless you use cache sh -c …).
• Cache entries that only differ by word separation (e.g. mycommand "foo bar" vs. mycommand "foo" "bar") are not distinguished. This shouldn't be a real problem in practice.
• The first invocation of the command returns its return code. Cached invocations return 0. If you want to return the original return code, add an associative array to store return codes. Another possible behavior would be to store the command output in the cache only if the command returns a nonzero status.
• Only the standard output is saved, output on the standard error or other file descriptors is not.
• Empty output is not considered cached.
• The number of newlines at the end of the output is normalized to 1. (This is easy to fix but probably better this way in practice.)
• You can't use an alias after cache. This cloud has a silver lining: if you find you always want to cache a command, you can alias it to cache mycommand.
• There's no expiry. There are many potentially useful expiry mechanisms (try to detect relevance, keep a maximum number of entries, give entries a time to live…), I won't implement them as part of this answer.

Answer 4

I'm guessing, given that we're talking about "2-5 seconds" being painful, that you're trying to display this information in your shell prompt. I developed bash-cache to speed up my prompt by caching the results of long-running commands I wanted to display.

To cache a call like this (whether or not its for your prompt) simply create a shell function that invokes the command(s) you need, and then call bc::cache with the function's name to enable caching:

# Memoized wrapper around the ec2-describe-instances command
ec2-describe-instances() {
  # Could also use "$(which ec2-describe-instances)" but for a one-off it's
  # probably just easier to hard-code the path.
  /usr/bin/ec2-describe-instances "$@"
} && bc::cache ec2-describe-instances

Then call ec2-describe-instances as normal, and watch as most invocations return near instantly!

See my earlier answer for more details.

Answer 5

#!/bin/sh
PROG="$(basename $0)"
DIR="${HOME}/.cache/${PROG}"
mkdir -p "${DIR}"
EXPIRY=600 # default to 10 minutes
[ "$1" -eq "$1" ] 2>/dev/null && EXPIRY=$1 && shift
CMD="$@"
HASH=$(echo "$CMD" | md5sum | awk '{print $1}')
CACHE="$DIR/$HASH"
test -f "${CACHE}" && [ $(expr $(date +%s) - $(date -r "$CACHE" +%s)) -le $EXPIRY ] || "$CMD" > "${CACHE}"
cat "${CACHE}"