Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm using the amazon ec2 command line tools, and the ec2-describe-instances tool is a little painful because of the 2-5 seconds it requires to make the request and display the output.

I'm looking into using the fec2din tool described in this question to format the output of ec2-describe-instances and was wondering what the best way would be to cache the output from the call.

fec2din uses mktemp to create a temporary file and then uses awk to format the output.

Does is there some tool I could use with a TTL paramater that would only run ec2-describe-instances if the timestamp on the cache file was older than a certain time?

It would be neat if there was a little utility to do this (like sponge does for helping with stdout).

share|improve this question
add comment

3 Answers

up vote 2 down vote accepted

While this does not address your question as such, I can highly recommend using Amazon EC2 via the excellent boto instead, which is a Python package that provides interfaces to Amazon Web Services.

It pretty much covers the same ground as the Amazon EC2 API Tools, but doesn't suffer from the painful delays due to relying on the modern and fast AWS REST APIs, while the EC2 API Tools are written in Java and used to use the old and slow SOAP APIs (don't know whether they might have changed gears in this regard already, but your experience as well as the still required AWS X.509 Certificates seem to suggest otherwise).

In addition, you don't need to use these AWS X.509 Certificates anymore, rather can use the nowadays more common and flexible approach via an AWS Access Key ID and an AWS Secret Access Key, which might as well (and usually should be) provided via AWS Identity and Access Management (IAM) in order to avoid exposing your main AWS account credentials.

On top of that, boto it is an obvious candidate for orchestrating your everyday AWS usage via Python scripts - this can as well be done with bash of course, but you get the idea ;)

Documentation

You can find documentation and examples in boto: A Python interface to Amazon Web Services, which provides decent (i.e. more or less complete) API References (e.g. for EC2) as well as dedicated introductory articles explaining the basic usage for several services (but not all yet), e.g. An Introduction to boto’s EC2 interface covers the use case at hand.

In addition you may want to read Boto Config for setting up your environment (credentials etc.).

Usage

You can explore boto via the Python Read–eval–print loop (REPL), i.e. by starting python.

Once you are satisfied with your fragments you can convert them into a Python script for standalone usage.

Example

Here is a sample approximately addressing your use case (it assumes you have setup the credentials in your environment, as explained in Boto Config):

$ python
Python 2.7.2 (default, Jun 12 2011, 14:24:46)
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto
>>> ec2 = boto.connect_ec2()
>>> instances = ec2.get_all_instances()
>>> instances
[Reservation:r-916d01f2, Reservation:r-3f7e055c, Reservation:r-c37209a0]

Okay, get_all_instances() actually returned a list of boto.ec2.instance.Reservation, so here is an annoying indirection in place (stemming from the EC2 API), which you won't see elsewhere usually - the docs are conclusive already, but let's see how to find that out by introspection:

>>> dir(instances[0])
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', 
'__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', 
'__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 
'connection', 'endElement', 'groups', 'id', 'instances', 'item', 'owner_id', 
'region', 'startElement', 'stop_all']
>>> insts = instances[0].instances
>>> insts
[Instance:i-5d9a593a]

That's more like it, so finally you want to see the attribute values of i-5d9a593a (most attributes omitted for brevity and privacy):

>>> vars(insts[0])
{'kernel': u'aki-825ea7eb', 'private_dns_name': '', 'id': u'i-5d9a593a', 
'monitored': False, 'state': u'stopped', 'architecture': u'x86_64',  
 'public_dns_name': '', 'ip_address': None, 'placement': u
'us-east-1a', 'ami_launch_index': u'0', 'dns_name': '', 'region': RegionInfo:us-east-1
# ...
}

Not quite, but Python's Data pretty printer (pprint) to the rescue:

>>> import pprint
>>> pp = pprint.PrettyPrinter(indent=4)
>>> pp.pprint(vars(insts[0])) {   
    '_in_monitoring_element': False,
    'ami_launch_index': u'0',
    'architecture': u'x86_64',
    'dns_name': '',
    'hypervisor': u'xen',
    'id': u'i-5d9a593a',
    'instance_class': None,
    'instance_type': u'm1.medium',
    'ip_address': None,
    'kernel': u'aki-825ea7eb'
    # ...
    }
share|improve this answer
    
Thanks, this looks good. How do you use it - by writing your own python scripts that use the sort of framework it provides, or directly via the command line? Saw the README but it didn't have any examples –  cwd May 7 '12 at 16:49
    
@cwd: I've updated the answer with brief hints in this regard, hope you can continue from there. Eventually I'll be able to add a sample addressing the functionality of ec2-describe-instances later on as well, but am lacking the time right now, sorry. –  Steffen Opel May 7 '12 at 17:07
    
@cwd: I've added an example approximately addressing your use case; please don't let the verbosity put you off here, the particular EC2 API is unusually complex and I've added the noise and tips for clarity - you might want to try something simpler like ec2.get_all_regions() first eventually. –  Steffen Opel May 7 '12 at 18:45
    
+1 thanks so much! –  cwd May 8 '12 at 18:37
add comment

A makeshift sponge utility can be made by reading lines into an array and then outputting it, for example:

sponge() {
    local line lines
    while IFS= read -r line; do
        lines+=( "$line" )
    done
    printf '%s\n' "${lines[@]}"
}

Then run it as command1 | sponge | command2.

share|improve this answer
    
I guess that's good if you can't do apt-get install moreutils. Thanks for sharing. –  cwd May 8 '12 at 18:38
add comment

Here's a very crude caching facility for bash/ksh/zsh.

typeset -A output_cache
cache () {
  local IFS='
  ' ret=0
  if [[ -z ${output_cache["$*"]} ]]; then
    output_cache["$*"]=$(unset IFS; "$@")
    ret=$?
  fi
  echo "${output_cache["$*"]}"
  return $ret
}
uncache () {
  local IFS='
  '
  unset output_cache["$*"]
}

Example:

$ cache mycommand --options        # takes a while
…
$ cache mycommand --options        # instantaneous
…
$ uncache mycommand --options      # remove a cache entry

Limitations:

  • This only makes sense if running the command twice produces the same output each time.
  • The cache is not shared between shell instances.
  • This only works for simple commands, not pipelines, loops, etc. (unless you use cache sh -c …).
  • Cache entries that only differ by word separation (e.g. mycommand "foo bar" vs. mycommand "foo" "bar") are not distinguished. This shouldn't be a real problem in practice.
  • The first invocation of the command returns its return code. Cached invocations return 0. If you want to return the original return code, add an associative array to store return codes. Another possible behavior would be to store the command output in the cache only if the command returns a nonzero status.
  • Only the standard output is saved, output on the standard error or other file descriptors is not.
  • Empty output is not considered cached.
  • The number of newlines at the end of the output is normalized to 1. (This is easy to fix but probably better this way in practice.)
  • You can't use an alias after cache. This cloud has a silver lining: if you find you always want to cache a command, you can alias it to cache mycommand.
  • There's no expiry. There are many potentially useful expiry mechanisms (try to detect relevance, keep a maximum number of entries, give entries a time to live…), I won't implement them as part of this answer.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.