Garr – Lessons from Bamboo

I have been a fan of Garr Reynolds, since I discovered his first book Presentation Zen. This is Garr’s recent talk from TEDxTokyo which is well worth watching.

These are the slides for the talk, which are done in Garr’s trademark style.

The ascendancy of JSON

I’ve long been in despair over the popularity of XML as an information interchange format.  My main complaint is that is so verbose that it is very easy to end up with the XML document structure taking up more memory than the actual data it encodes.  This phenomenon is so common it even has a name: the ‘Angle Bracket Tax‘ and can be very painful on memory or bandwidth limited embedded systems.

JSON is based on a subset of the JavaScript scripting language and this is one of the big drivers of its adoption is that JSON is trivial to work with in JavaScript applications.  Mainstream adoption is taking place with languages like Python and Ruby and frameworks like Microsoft’s .Net offering JSON support.

Karsten Januszewski has an interesting post on ‘The Rise of JSON‘ that is well worth checking out.

Hacking Work Manifesto

This is an interesting video manifesto from the guys at Hacking Work: working around cooperate processes and systems to achieve higher productivity.

What stands out the most to me in this video is the following statistic: ‘Workers receive 325 pages of information a day but only uses about 5 pages’.  As I’ve recently had to modify my email filtering scheme to mark certain categories of email as read automatically to prevent information overload.

Extracting image EXIF data with Python

Most digital cameras and smartphones embed EXIF (EXchangeable Image Format) data into the photographs they capture.  This can include: camera make & model, date and time, camera settings like orientation, aperture, ISO, shutter speed, focal, length and even GPS location.

After a bit of experimentation I have found the following method of using the undocumented ExifTags module in the Python Image Library (PIL) to be the simplest way to extract EXIF tags from images using Python.  There are other EXIF modules available for Python however currently PIL is the simplest to install on Mac OS X.

from PIL import Image
from PIL.ExifTags import TAGS

def get_exif_data(fname):
    """Get embedded EXIF data from image file."""
    ret = {}
    try:
        img = Image.open(fname)
        if hasattr( img, '_getexif' ):
            exifinfo = img._getexif()
            if exifinfo != None:
                for tag, value in exifinfo.items():
                    decoded = TAGS.get(tag, tag)
                    ret[decoded] = value
    except IOError:
        print 'IOERROR ' + fname
    return ret

The above code was based on the code snippet in Paolo’s answer to this StackOverflow question. I have added basic exception handling and a check for the existence of the _getexif attribute prior to accessing it.

Graphing real data with MatPlotLib

In a previous post I covered the basics of graphing in Python with the MatPlotLib module.  In this post I am going to demostrate how to use MatPlotLib with some real world data retrieved from a web service and then processed into a format usable by MatPLotLib.

The example script performs the following steps:

  1. Takes a specified stock’s ticker symbol and column to plot over time (from Open, High, Low, Close, Volume, Adj Close) as input.
  2. Fetches the corresponding stock data from Yahoo! Finance and saves it into a CSV file using the urllib module.
  3. Processes the data in the CSV file into a suitable format for matplotlib using the csv, datetime and matplotlib.dates modules.
  4. Plots a graph of the data plotted over time using MatPlotLib and a saves a copy as PNG format image.

Note: To keep the example concise I am not performing any error handling.

"""Fetches specified stock data from Yahoo and graph it with MatPlotLib."""

from urllib import urlretrieve
from csv import DictReader
from matplotlib import pyplot
from matplotlib.dates import date2num
from datetime import datetime

def fetchstockdata( stockticker, filename ):
    """Fetch specified stock data and store it in named file."""
    url = 'http://ichart.finance.yahoo.com/table.csv?s=%s' % stockticker
    urlretrieve( url, filename )

def importstockdata( filename ):
    """Import CSV data into dict of lists, converting dates into timestamps."""
    results = {}
    for row in DictReader( open( filename,'rb' ) ):
        for col in row.keys():
            if col == 'Date':
                coldata = date2num( datetime.strptime( row[col], '%Y-%m-%d') )
            else:
                coldata = row[col]
            results.setdefault( col, [] ).append( coldata )
    return results

def plotstockdata( stockdata, stockticker, dates, col ):
    """Use MatPlotLib to graph speciifed stock data."""
    pyplot.plot_date( stockdata[dates], stockdata[col], '-', xdate=True )
    pyplot.title( '%s - %s / %s' % (stockticker, col, dates) )
    pyplot.xlabel( dates )
    pyplot.ylabel( col )
    pyplot.savefig( '%s.png' % stockticker )
    pyplot.show()

if __name__ == '__main__':
    from sys import argv
    # Use second argument as ticker and third argument as column.
    TICKER = argv[1].upper()
    COL = argv[2]
    # Grab the stock data from Yahoo!
    FILENAME = '%s.csv' % TICKER
    fetchstockdata( TICKER, FILENAME )
    # Import the data.
    DATA = importstockdata( FILENAME )
    # Plot the graph with Date as X-Axis and User selected column as Y-Axis.
    plotstockdata( DATA, TICKER, 'Date', COL )

Running this script with using the command line “python StockChart.py goog ‘Adj Close’” will produce a chart like the following.

This is a good example of why I like Python’s batteries included philosophy so much: it means I spend more of my time writing interesting bits of code as the utility functionality I need has already been implemented or is only an easy_install away.

Small steps to big goals

Scott McIntyre recently wrote a post for Zen Habits on ‘The Small-Scale Approach to Achieving Great Things’, I found his post particularly interesting as I’ve observed a similar pattern over the years.

For me, the stumbling blocks to success were that I allowed myself to be overcome by the size of the task and how long it would take to get there. What really helped was to break down the overall journey into smaller steps and to stop looking as far into the future.

To successfully complete a big goal requires breaking the goal down into small actionable steps.  This doesn’t need to be an intricately detailed master plan, knowing as little as the next actionable step is sufficient to make progress.  Without knowing at least the next actionable step it is very easy to become discouraged by the scale of the goal.

It is also easy to underestimate how much the positive feeling of ‘getting stuff done’ when completing these small steps can encourage you to continue pursuing your goal to completion.  Without the feeling of making progress towards a goal it is very easy to become discouraged and give up.