Questions starting at 31:20 are pretty interesting.

This is a very convenient pairing as SQLite is an excellent lightweight SQL implementation that I find very useful for a variety of tasks e.g. data mining.  Or any task involving manipulating complex data sets where I’d otherwise end up resorting to using a full blown SQL server like MySQL.

Here is a simple example of using SQLite in Python using it’s built-in sqlite3 module:

import sqlite3

# craete a connection
con = sqlite3.connect('test.db')

# create a cursor
cur = con.cursor()

# create a test table
cur.execute( "CREATE TABLE testTable (myKey INT, myValue INT)" )

# insert some data
for i in range(0,10):
 cur.execute( "INSERT INTO testTable VALUES ( %d, %d )"%(i,i*i) )

# select the data
for row in cur.execute( "SELECT * FROM testTable" ):
 print row

# destroy (drop) our test table
cur.execute( "DROP TABLE testTable" )

# close the connection
con.close()

As you can see Python makes handling SQLite (a C language library) much easier, less error prone, and the resulting code much more compact than SQLite’s native C.

The awareness of what I don’t know helps keep me humble, humility makes working as part of a team easier:  as there is no pressure to have to know everything or not make mistakes.  In fact I tend to expect to make mistakes more now than when I first started programming.  Perhaps it is the years I have spent shipping games that finally proved to me that I too write software that contains bugs (the horror!).

I remember a professor at university telling me that the main difference between a professor and a first year student working on a programming task is that the student will start working immediately and also start making mistakes immediately, the professor will think for a while then start work and start making mistakes as well.  I have found this observation to have a surprising amount of truth in it,:whether it is at university, work or even sports.

Mistakes are healthy: without mistakes we would have not reason to every really think about what we are doing e.g. why didn’t that work?  By continually pushing (or stretching) ourselves to failure we discover our boundaries, once we know where are our boundaries are we can then start to work on pushing them further.  However if we always play it safe and never push ourselves (which can be scary) we will never discover our boundaries which makes improvement much harder and also makes approaching those boundaries harder due to fear (typically of loss of control).

An excellent example of this is people learning to ice skate: young children will tend to fling themselves around the rink with wild abandon falling all over the place (as failure is expected but irrelevant due to lack of social stigma), yet adult beginners typically skate in a much more conservative fashion taking less risks (as there is social stigma against falling as an adult: falling is seen as failure).  Interestingly when observing ice hockey players, it is noticeable that those with the best skating technique are typically falling more than the other the non-beginner players on the ice, as they do not fear falling.

As most of us are not engaged in high risk activities on a daily basis we can easily begin to revise how we think about failure and to learn to embrace it as a powerful tool for self improvement.

It is not that I have anything against Perl as a programming language: it is a fine language.  I find that Perl’s motto of `There’s more than one way to do it.’ means that it is a programming language with (you guessed it) more than one way to implement most operations.  This flexibility is a double edged sword: it allows a lot of creativity in program implementation, yet it also means that a dozen Perl coders will produce a dozen radically different scripts for the same problem.  This makes Perl quite hard on beginners or infrequent maintainers of Perl scripts: as to understand any given Perl script requires an understanding of much more of the Perl programming language than it would for most comparable programming languages.

I digress.

I’ve known for years that it is possible to write awful obscure code in low level languages like C and C++ but it really hadn’t occurred to me that a high level language like C# could be abused to the same degree.  Thinking about this some more, it is not the fault of the programming language itself but of the programmer that is using it.

So, brittle obtuse code is the result of a bad programmer not a bad programming language: well except maybe Brain Fuck (but thats kinda the point).

This may seem harmless but consider that the future use cases you are imagining are not something you fully understand yet: as most of us will admit we are pretty bad at anticipating the future.  So how can we hope to implement future functionality successfully?  The most likely outcome is that whatever you implement now not be sufficient when you actually try to use it in the future.  This will mean more time will have to be invested in the future to refactor or replace the functionality with what is actually required.

Unless you also plan to implement a full set of tests for this new functionality then you are adding extra code to your program that will not be correct and will not stay correct.  This can also lead to really weird run time behavior if the flow of program execution goes into your new ‘future proof’ functionality unexpectedly: which can be very hard to debug.  Adding functionality for future use without corresponding tests also means that as the rest of your program evolves the future functionality does not evolve with it, this means by the time you actually try to use it that it is most likely hopelessly out of sync with the rest of the code base.

The future functionality is also expanding the size of the source code of your program which leads to unnecessary code bloat.  In some compiled languages (e.g. C++) code bloat will lead to increased memory usage and decreased program performance which is highly undesirable.

Another thing to consider from a business stand point is that you are spending time and money implementing future functionality when you are not being paid for it.  This is a strong indication to me that implementing future functionality makes little business sense.  As it means spending money now paying developers to implement functionality that you can’t sell right now.

There is a handy acronym for this: it is YAGNI which stands for ‘You Aren’t Gonna Need It’.  The essence of this concept is to only implement the functionality you need right now and to implement in as simple and robust a fashion as possible.  As simple robust code (preferable with acompaning unit tests) is code that can be easily refactored in the future to meet future needs and that is cost effcient!

Assume the task that has been set is complex and contains some degree of uncertainty which means that the first few attempts will probably fail.  Which programmer would you want working on the task?  If the task is actually impossible then programmer A will be the cheapest from a business standpoint as they fail earliest followed by programmer B and finally programmer C.  Also, programmer A will have five attempts at a solution compared to programmer B’s two and a half attempts and programmer C’s single attempt.  As iteration leads to improved quality (Boyd’s Law) this means that programmer A is also likely to hit on a more superior solution than programmer B or programmer C in the same time period.

The interesting thing here is that in my experience new programmers tend to work like programmer C: they will attempt a single solution working diligently to overcome any hurdles they encounter, no matter how severe the hurdle.  This sounds harmless but remember that few things are impossible in programming, so the more a programmer hammers away at a problem the more likely they are to fudge a solution and that solution (in my experience) is not often of a high quality.  The counter point to this is that more productive programmers will rapidly discard attempts early in each attempt when it becomes clear that the solution though possible is not actually desirable e.g. the dog is hungry and solutions include: A) feed the dog B) kill the dog.

This can also be observed by monitoring how long a programmer will remain stuck for before asking for assistance.  The most productive programmers will get help within minutes of becoming completly stuck.  However the least productive (and often least experienced) programmers will often struggle with the problem for days before actually asking for assistance and this delay is expensive in terms of money, deadlines and product quality.

For example, it faster on the CPU you are targeting to compare an integer to zero than to compare an integer to another integer.  Therefore you have decided to write all your loops so they count down to zero instead of the normal approach of counting up from zero.  This will save you a few cycles per iteration of your loop, but have you considered:

• Counting down and not up in a loop is not conventional in most programming languages and may result in you making math errors which can result in hard to track down bugs in your loop control logic or code inside your loop.
• Pointer arithmetic in languages like C/C++ is hard enough when counting up (e.g, pObj++) but counting down is even harder to understand as it is so incredibly rarely done and requires extra pointer math to set up.
• The programmer that comes after you to develop or maintain the source code may not realise you are counting down and not up in your loop and then implement functionality that relies on your loop counting up which would again cause bugs and confusion.
• That despite your loop being invoked for thousands or millions of iterations, your loop’s control logic is no where near as expensive in terms of CPU operations as function call X that comes right in the middle of your loop which costs millions of cycles by itself every iteration!
• That it is almost unheard of for loop control logic to be an issue performance wise even when it is in tightest inner loops.
• That most optimising compilers are perfectly able to make this optimisation for you during compilation on your target platform with none of the above programmer confusion?

My advice here is never optimise without measuring first: as at best you will save a few CPU cycles where it really does not matter at the cost of making your source more complex and harder to understand and at worse you will add subtle bugs to your program and make the source code more complex and harder to understand or maintain!  I have lost count now of how many times I’ve tracked subtle bugs in programs down to programmers making preemptive optimisations without first measuring to see if any optimisation is required and if the extra level of complexity introduced by the optimisation is a desirable trade off.

Occasionally there will be an exception to this rule but in general if you are going to optimise something should always measure first and then optimise only if it is nessessary.

This diagram emphasises the importance of measurement in the optimisation process: you cannot begin the process, evaluate your optimisations, even discard your optimistations or honestly finish the process without first measuring the program performance versus your target performance.  As I have mentioned before the initial optimisations to a program tend to yeild larger returns and then returns drop off until much more time is invested. These initial easy optimisations are not usually the sort of changes new programmers expect: they expect sexy l33t code like inline assembler for big performance wins, not tweaking compiler flags or removing calls to a pure virtual method on a base object in an inner loop.

I agree teaching a higher level language with built in memory management as a first programming language is the more humane option as far as first time students are concerned.  As learning your first programming language and all the associated concepts is hard enough without all the nasty memory related gotchas in a language like C or C++.  Although the nice thing about learning something like C or php (as Jani suggests) is that teaching object orientation can be avoided initially, as that particular concept does seem to be something that some students struggle with a lot the first time they encounter it.

However I do think a low level language that has manual memory management should be at least experienced by every programmer, as it is a fundamental concept of programming effectively.  And I believe that having minimal experience with some form of manual memory management would help most programmers write higher performance programs.  But perhaps I was being a bit over zealous recommending that everyone learns a language like C/C++ as a first language, although I think at least a short course featuring a language with manual memory management would be invaluable to all programmers.  The course would not even need to cover object orientation as you can teach that in higher level languages that are easier to manipulate than C++: the key point of the course would be to teach memory management and its implications for writing fast high quality software.

As much as I go on about knowing assembler, I admit its not something I write very often at all, however it is something that is incredibly handy to be able to read and comprehend. Even the most basic level of understanding of the instructions for loading data from or to memory and registers, branching and basic math operations would allow you to check the compiler has actually generated the assembly code you expected. This is especially useful for debugging unexpectedly slow code in compiler optimised builds.  You also don’t need to know about all the fancy vendor specific assembly instructions, as I’ve mentioned the basics are usually sufficient to be able to understand roughly what a slow piece of code is doing to then rewrite the higher level source code in a way that prompts compiler to generate faster assembly code.  Actually writing assembly code should always be the last resort and only done by experts after all other higher level refactorings are attempted, as higher level optimisation or refactoring work is usually more effective and easier to understand.  Also, assembler is not usually easily portable to other hardware platforms and most programmers find it harder to debug assembler than normal C/C++ source code.  Plus as Jani mentions it is scary to find a block of inline assembler in the C++ program you are working as it is much harder to decipher than regular source code unless it is very well documented.

This ability to check what is going on ‘under the hood’ of your language is essential for those hard to track down bugs and when optimising your application for performance, especially where the compiler has reordered the program flow or generated unexpectidally slow code. This can also be applied to high level languages as well as C/C++: checking that the IL byte code generated by your C# compiler or the Java byte code generated by the JVM is doing what you expected can be very useful in understanding your program’s execution and performance.

I know I don’t want to use the following languages mostly for reasons of personal taste:

• Perl as I can never remember it’s syntax and I can’t seem to read any script I’ve written in Perl after a week or so.
• php as I already use it for web development (and like it a lot) but it does not really seam suitable for scripting or quick application development that is not running on a web server.
• Java I’ve never gotten on with, but that was probably more to do with falling out with my Java lecturer at university.  Although I should probably revisit Java at some point as it seems very popular at the enterprise level.
• I also want something not Microsoft specific, as much as I love C# I want something I can use easily on my Mac or Linux boxes at home without compiling.

So far I’ve narrowed it down to either Python or Ruby.  Originally I was going to learn Python as I like its use of white space for scoping, its wide range of libraries and maturity as a language.  However there seems to be a lot of buzz about Ruby these days, especially its Rails framework and all the buzz has piqued my interest.  Further investigation into Ruby has me now leaning towards learning it mostly for the trivial reason (as the languages are equally matched in my mind) that Ruby has the similar concept and syntax of public, protected and private for methods which is familiar to me as a C++ and C# programmer.

In fact I am more then leaning towards Ruby as I bought a copy of ‘The Ruby Programming Language‘ at lunchtime, mostly as it was getting good reviews online and because the original developer of Ruby was a co-author.  I always enjoyed Bjarne Stroustrup‘s ‘The C++ Programming Language‘ book, getting the language creators view on their creation is always fascinating and provides valuable insights.

So expect to see more posts about Ruby as I work my way through the book and experiement.  I can even post code snippets with syntax highlighting thanks to the Google Syntax Highlighter in the form of a handy plug-in for WordPress!