So far my favorite additions and changes in Python 2.7.1 since upgrading from the default Python 2.6.1 installation in Mac OS X Snow Leopard are the following:
- Dictionary and Set Comprehensions.
List comprehensions are one of my favorite language features in Python, they are incredibly useful for processing and building lists. So I am very excited to see dictionary and set comprehensions back ported from Python 3 to Python 2.7.1.
- The ArgParse Module.
As a C/C++ programmer I original did command line argument processing in Python manually using sys.argv, then I discovered the C-style getopt module. I always found myself wondering if there was a more concise Pythonic way to handle command line parameters. The argparse module is the solution, it replaces the optparse module. I particularly like how argparse (and optparse) will generate the command line help for you!
- csv.DictWriter.writeheader method.
While this is a very minor change (in Python 2.7 to be precise), I am a big fan of the csv module’s DictWriter class as a way to easily dump lists of dictionaries to a file for easy analysis and debugging with Excel. The addition of the DictWriter class of an new writeheader method makes this class even easier to use.
You can find the full release notes for Python 2.7.1 here, there are so many more changes than I’ve covered here so its well worth checking out the release notes. What are your favorite changes in Python 2.7.1?
I wrote this script to find and optionally delete duplicate files in a directory tree. The script uses MD5 hashes of each file’s content to detect duplicate files. This script is based on zalew’s answer on stackoverflow. So far I have found this script sufficient for accurately finding and removing duplicate files in my photograph collection.
"""Find duplicate files inside a directory tree."""
from os import walk, remove, stat
from os.path import join as joinpath
from md5 import md5
def find_duplicates( rootdir ):
"""Find duplicate files in directory tree."""
filesizes = {}
# Build up dict with key as filesize and value is list of filenames.
for path, dirs, files in walk( rootdir ):
for filename in files:
filepath = joinpath( path, filename )
filesize = stat( filepath ).st_size
filesizes.setdefault( filesize, [] ).append( filepath )
unique = set()
duplicates = []
# We are only interested in lists with more than one entry.
for files in [ flist for flist in filesizes.values() if len(flist)>1 ]:
for filepath in files:
with open( filepath ) as openfile:
filehash = md5( openfile.read() ).hexdigest()
if filehash not in unique:
unique.add( filehash )
else:
duplicates.append( filepath )
return duplicates
if __name__ == '__main__':
from argparse import ArgumentParser
PARSER = ArgumentParser( description='Finds duplicate files.' )
PARSER.add_argument( 'root', metavar='R', help='Dir to search.' )
PARSER.add_argument( '-remove', action='store_true',
help='Delete duplicate files.' )
ARGS = PARSER.parse_args()
DUPS = find_duplicates( ARGS.root )
print '%d Duplicate files found.' % len(DUPS)
for f in sorted(DUPS):
if ARGS.remove == True:
remove( f )
print '\tDeleted '+ f
else:
print '\t'+ f
I discovered the argparse module (added in Python 2.7) in the standard library this week and it makes command line parameter handling nice and concise.
UPDATE: Changed uniques array into a set and added first pass using file sizes as performance improvement, allot faster now.
UPDATE: You can now find this script on github at github.com/dpbrown/Duplicate-Files.
I’ve long been in despair over the popularity of XML as an information interchange format. My main complaint is that is so verbose that it is very easy to end up with the XML document structure taking up more memory than the actual data it encodes. This phenomenon is so common it even has a name: the ‘Angle Bracket Tax‘ and can be very painful on memory or bandwidth limited embedded systems.
JSON is based on a subset of the JavaScript scripting language and this is one of the big drivers of its adoption is that JSON is trivial to work with in JavaScript applications. Mainstream adoption is taking place with languages like Python and Ruby and frameworks like Microsoft’s .Net offering JSON support.
Karsten Januszewski has an interesting post on ‘The Rise of JSON‘ that is well worth checking out.
Most digital cameras and smartphones embed EXIF (EXchangeable Image Format) data into the photographs they capture. This can include: camera make & model, date and time, camera settings like orientation, aperture, ISO, shutter speed, focal, length and even GPS location.
After a bit of experimentation I have found the following method of using the undocumented ExifTags module in the Python Image Library (PIL) to be the simplest way to extract EXIF tags from images using Python. There are other EXIF modules available for Python however currently PIL is the simplest to install on Mac OS X.
from PIL import Image
from PIL.ExifTags import TAGS
def get_exif_data(fname):
"""Get embedded EXIF data from image file."""
ret = {}
try:
img = Image.open(fname)
if hasattr( img, '_getexif' ):
exifinfo = img._getexif()
if exifinfo != None:
for tag, value in exifinfo.items():
decoded = TAGS.get(tag, tag)
ret[decoded] = value
except IOError:
print 'IOERROR ' + fname
return ret
The above code was based on the code snippet in Paolo’s answer to this StackOverflow question. I have added basic exception handling and a check for the existence of the _getexif attribute prior to accessing it.
In a previous post I covered the basics of graphing in Python with the MatPlotLib module. In this post I am going to demostrate how to use MatPlotLib with some real world data retrieved from a web service and then processed into a format usable by MatPLotLib.
The example script performs the following steps:
- Takes a specified stock’s ticker symbol and column to plot over time (from Open, High, Low, Close, Volume, Adj Close) as input.
- Fetches the corresponding stock data from Yahoo! Finance and saves it into a CSV file using the urllib module.
- Processes the data in the CSV file into a suitable format for matplotlib using the csv, datetime and matplotlib.dates modules.
- Plots a graph of the data plotted over time using MatPlotLib and a saves a copy as PNG format image.
Note: To keep the example concise I am not performing any error handling.
"""Fetches specified stock data from Yahoo and graph it with MatPlotLib."""
from urllib import urlretrieve
from csv import DictReader
from matplotlib import pyplot
from matplotlib.dates import date2num
from datetime import datetime
def fetchstockdata( stockticker, filename ):
"""Fetch specified stock data and store it in named file."""
url = 'http://ichart.finance.yahoo.com/table.csv?s=%s' % stockticker
urlretrieve( url, filename )
def importstockdata( filename ):
"""Import CSV data into dict of lists, converting dates into timestamps."""
results = {}
for row in DictReader( open( filename,'rb' ) ):
for col in row.keys():
if col == 'Date':
coldata = date2num( datetime.strptime( row[col], '%Y-%m-%d') )
else:
coldata = row[col]
results.setdefault( col, [] ).append( coldata )
return results
def plotstockdata( stockdata, stockticker, dates, col ):
"""Use MatPlotLib to graph speciifed stock data."""
pyplot.plot_date( stockdata[dates], stockdata[col], '-', xdate=True )
pyplot.title( '%s - %s / %s' % (stockticker, col, dates) )
pyplot.xlabel( dates )
pyplot.ylabel( col )
pyplot.savefig( '%s.png' % stockticker )
pyplot.show()
if __name__ == '__main__':
from sys import argv
# Use second argument as ticker and third argument as column.
TICKER = argv[1].upper()
COL = argv[2]
# Grab the stock data from Yahoo!
FILENAME = '%s.csv' % TICKER
fetchstockdata( TICKER, FILENAME )
# Import the data.
DATA = importstockdata( FILENAME )
# Plot the graph with Date as X-Axis and User selected column as Y-Axis.
plotstockdata( DATA, TICKER, 'Date', COL )
Running this script with using the command line “python StockChart.py goog ‘Adj Close’” will produce a chart like the following.
This is a good example of why I like Python’s batteries included philosophy so much: it means I spend more of my time writing interesting bits of code as the utility functionality I need has already been implemented or is only an easy_install away.
One of the Python modules that has most interested me recently is MatPlotLib which is a sophisticated graphing module which can be used to create journal grade graphs of almost anything. The official gallery for MatPlotLib is worth checking out to get an idea of the sheer range of graph types it can be used to create.
It is simple enough to get started using MatPlotLib for example to create a line graph of x*x and save it as a PNG file requires only the following:
"""Simple demonstration of MatPlotLib plotting."""
from matplotlib import pyplot
X = range(0,100)
Y = [ i*i for i in X ]
pyplot.plot( X, Y, '-' )
pyplot.title( 'Plotting x*x' )
pyplot.xlabel( 'X Axis' )
pyplot.ylabel( 'Y Axis' )
pyplot.savefig( 'Simple.png' )
pyplot.show()
The above script will produce the following graph:
To plot data over a time period the simplest solution is to convert date/time units to timestamps using MatPlotLibs date2num function and then to plot using the plot_date method as follows:
"""Simple demonstration of MatPlotLib Date plotting."""
from matplotlib import pyplot
from matplotlib.dates import date2num
from datetime import datetime, timedelta
# Generate a series of timestamps from today to today + 100 years.
X = [date2num(datetime.today()+timedelta(days=365*x)) for x in range(0,100)]
Y = [i*i for i in range(0,100)]
pyplot.plot_date( X, Y, '-', xdate=True )
pyplot.title( 'Plotting x*x' )
pyplot.xlabel( 'X Axis' )
pyplot.ylabel( 'Y Axis' )
pyplot.savefig( 'SimpleDates.png' )
pyplot.show()
Which will generate a chart like the following:
As you can see it is fairly simple to graph data using MatPlotLib. This makes Python and MatPlotLib a compelling solution for data analysis when combined with the many available modules for dealing with common data storage formats like text (using RegEx), CSV, XML and JSON files and SQL databases.
Recent Comments