Programming Software Development: Architecture Hardware Programming Languages
by Daniel
2 comments
Teaching students memory managment
Yesterday morning I spent a fascinating hour or so in a meeting listening to a very senior engineer give and in depth presentation about the performance characteristics of the low level memory systems on a console and the content of the presentation got me thinking about how software engineering is taught. Specifically are students being taught enough about low level languages and the hardware, so that they have some experience of what is happening under the hood of these fancy modern memory managed languages like Java, C#, php, python or ruby? If students are not being taught assembler or languages where you need to think about managing memory and the underlying hardware e.g. C or C++ how can they then be expected to program effectively in a higher level language? As systems with finite resources like consoles have memory systems that are one of the biggest potential bottlenecks on the platform and also one of the most subtle to understand.
It used to be the case that students would be first taught assembler then a language like C before moving on to a higher level object orientated language like C++ or Java. Although this was even starting to change when I went to university: the mainstream computer science students were being taught in Java from the start. Although the course I was on started in C with some assembler plus hardware theory and then onto C++ but this was the exception. Plus I’d already encountered assembler in high school too, which I’d imagine is no longer the case in high school computing classes today. Talking to the students we have had on placement at work recently, it sounds like C# and Java are now the languages that are mainly being taught. This means only the theory of memory management and hardware concepts are being taught with no practical experience to reinforce that knowledge.
Why is this a big deal? Well if a programmer does not understand how the system his high level language is running upon works, how can they then be expected to program that high level language in a way that takes advantage of the available hardware and system architecture? Or at a minimum avoid the bottlenecks inherent on the particular hardware configuration they are working on? For example how could they appreciate the potential performance implications of accessing a variable length array vs a fixed length array of structures in a key inner loop if they do not understand how these data types have been implemented in the underlying language libraries or how their underlying hardware system will process it?
Maybe this is something to do with our high performance requirements and that our platforms are closed systems with more finite resources compared to a common PC but we see this sort of thing a lot especially with new software engineers straight out of university. The lack of understanding of the implications of the performance characteristics of our programming languages and our platform(s) hardware, leads these new software engineers to write code that is not only not as sound as a more senior engineer (which is to be expected) but also with much poorer performance characteristics.
This is not a problem with our students however, it is a problem with how they are taught. I think part of the reason is that it is harder to teach someone a language like assembler or C or even C++ than a managed language like Java or C#. As thinking about low level concepts like memory and cache management is not easy and concepts like pointers, memory alignment or functions pointers are not intuitive for many students, so I’d imagine it can be frustrating to teach those classes. And it is easier to teach object orientation in a language like C# or Java than C++ but that mean that we should skip straight to the memory manged high level languages?
So without fundamental knowledge of the basics that experience with lower level languages supply, it is very hard for that student to later write high performance applications without having to go back and learn the basics in their own time. And I believe writing high performance software is something most software engineers will be required to do even in this Web 2.0 generation. As anyone who has had to make a website scale to thousands of concurrent users can no doubt atest to the fact that performance is not just for those of us that program the more traditional high performance systems…
Getting work done: the empty office effect.
You may have noticed the lack of updates this week, I was in the office all weekend implementing something for a customer, so that they could then evaluate it this week which they were not expecting. I do not mind doing the occasional brief bit of overtime like this, especially when it is my idea. And it also gets me ahead of schedule plus it allows me to surprise my customers with an early delivery! It did however mean that I did not have time to write posts, for that I apologise!
Working this last weekend and observing the results (on our scrum’s burn down chart) did get me thinking about just how much more productive I can be over the course of a day at the weekend compared to a normal weekday: I believe I did 3-4 weekdays of work over two days this weekend. Yet I did not work any more hours than I would during the week but I achieved more in terms on implementation and testing than I would normally expect. I also experienced that familiar loss of a sense of time (losing track of hours while working) which I’ve come to associate with really productive periods of work: some people call it ‘the zone’ which sounds a bit too dramatic (and cheesy) to me.
As to the causes for this bout of hyper productivity I think part of it was that I went into the weekend knowing exactly what I wanted to get done in terms of goals and also having a fair idea of how to achieve those goals. I believe that helped me a lot, as a large portion of development time can be swallowed up with experimentation, thinking and research. So knowing what needed done helped me focus my efforts but I don’t believe it was the sole reason for the apparent (brief) doubling of my productivity over the weekend.
Perhaps another beneficial factor was the lack of people in the building over the weekend (sorry folks). The area where I work was very very quite this weekend, I was alone for most of the time with only one co-worker popping in and out for a few hours. This meant there was no external distractions like background conversations, people (or dogs) moving about or folk asking me questions to distract me from my task at hand. The building being mostly empty also meant there was no external electronic distractions e.g, email or instant messaging icons competing for my attention.
The lack of meetings was also great as this meant I had large unbroken periods of time to work in rather than an hour or so between meetings, as most experiments have shown that when you are in ‘the zone’ any disruption costs you about 15-20 minutes of time before you can get back to your ‘zone’ or train of thought. As I have mentioned before in previous posts I make a practice of booking myself in my office calender system for meetings that I call work blocks during the week. These blocks are usually about half a day in size and help prevent my day from getting horribly fragmented by meetings. Working this weekend has just reinforced to me just how disruptive meetings can be to actually getting work done and how important it is to have uninterrupted periods of time for getting work done in.
Yet for all that an empty office helped me get stuff done this weekend, I would I be as productive over the long term working in such a solo fashion? I’m not sure as these days software like games are suck large undertaking and require large teams of developers which means that communication and synchronisation becomes such key factors in development sucess but also key bottlenecks as well.
Ruby or Python: Round Two!
Yesterday in my post about web development frameworks, I mentioned I would be revisiting my previous post on whether to learn Python or Ruby. So this post is a continuation of that earlier discussion. I’m bringing up the topic of Python or Ruby again as I have been thinking about web frameworks since watching Cal Henderson’s ‘Why I hate Django’ talk from DjangoCon 2008 at the weekend and then the ‘Snakes and Rubies’ presentation.
The availability of MVC oriented web frameworks for Python (Django) and Ruby (Ruby on Rails) was not something I had considered when making my earlier decision about which of these two languages to put time into learning. However now that the idea of writing a new version of this site in one of these new fangled MVC/ORM frameworks has occurred to me, I’d like to reconsider my earlier decision. I’ve done some Google searches on ‘Django or Ruby on Rails’ and have been reading through the results. One of the more interesting blog posts I found on Django, Rails and php (here) has lead me on to the videos of the ‘Snakes and Rubies’ presentations which are very informative and well worth watching (they can be found here).
So far after watching the Django presentation I must admit I’m loving a lot of what was demonstrated: automatic administration page generation, built in caching, URL pattern matching and templates which separate content and logic/code sounds ideal. The automatic generation of an administration interface sounds especially nice as writing an decent administration interface is very time consuming to do correctly. And these interfaces are not something the end users will generally ever see so it can fell like a lot of effort for a feature that is never appreciated by the general population only by the administrators. The template system sounds excellent as well as it means graphic/web designers can handle the design of the site without requiring programming knowledge or support.
I’ve now watched the Ruby presentation too and I have to admit it did not grab me as much as the Django presentation did. Beautiful code, which seemed to be the main selling point of the presentation is a wonderful thing. However features like administration interface generation, caching and templates are more attractive to me than being able to use fewer lines of code to do things.
To digress for a moment, I think that my lack of excitement at the concept of beautiful code as it was presented in the video, could be in part to me being mainly a C/C++ programmer: so beautiful code in the sense of code that does a lot of work in a few lines of source code is not something you see very often in C/C++. In fact in a lot of cases that particular brand of beautiful code is something you would be down right suspicious of in C/C++ but that is perhaps more due to the nonrestrictive nature of C/C++: its lack of hand holding or built in safety as a language. This lack of safety perhaps drives C/C++ programmers to be more obsessive about understanding what exactly their source code does under the hood to avoid unpleasant surprises later. This encourages a preference for easy to read and comprehend source code and perhaps promotes suspicion of ‘voodoo’ code that does allot of work in a few lines of usually harder to read code. In short beautiful code is a worthy goal but something I would personally be suspicious of but that is probably more due to my personal C/C++ baggage than any fault in Ruby.
Now back to Ruby on Rails or Django. Another thing which I had not really considered in my earlier post was which language had the best range of high quality libraries available. I had assumed that both languages would have a similar range of libraries available but so far my investigations seem to be pointing to Python as having more. Ruby does not seem to have as many available libraries, unless of course I’m using the wrong keywords when searching the ruby gems system and if that is actually the definitive library index for Ruby. This lack of libraries is quite a big deal to me, as I am a big fan of not reinventing the wheel unless it is absolutely necessary, so I like having lots of high quality libraries I can use. That also speeds up prototyping efforts as it is easier to hook up existing libraries than write the same functionality from scratch again yourself.
In summary when I consider learning Python or Ruby for scripting, rapid prototyping and with added dimension of writing a website with an MVC framework. I think I would now reverse my decision and learn Python rather than Ruby. If I was still just thinking about scripting and prototyping with these languages I would consider the two languages evenally matched. With Python having an advantage with the apparent range and quality of its libraries and Ruby having an advantage with its support of Public/Private class members which I am more comfortable with.
Web Development Frameworks
When I created my first website back in 1998 it was purely static HTML that I created with Adobe’s DreamWeaver application. Next I discovered dynamic websites built using php and MySQL databases, so my next few sites were all written in php and accessing a MySQL database for the content. I have to confess that these initial dynamic sites were written in the worst style possible: inline php in the HTML and spaghetti php, all things that invoke horror in other programmers.
It was not until I started helping out with RockClimbing.com for a brief period which exposed me to a large scale php project supporting many thousands of users. With its advanced language usage, separation of code, content and layout with smarty templates, advanced caching and high performance database queries (one of the developers ended up working for MySQL iirc) it was beyond anything I had seen before. All this opened my eyes to how to architect a site in php correctly and that it could be a truly elegant language to work with and I am not alone in this feeling. Then for a period of several years my personal site fell into disuse as I was distracted by work and life from doing any active web development.
This year I decided to start this blog on a new domain name and start writing about software engineering and programming. My choice of blog engine? WordPress: which is a php based blogging engine which runs with MySQL as its database. I found myself tweaking my WordPress installation almost immediately and I found it easy enough to do as php was already familiar to me. In fact I’ve been tweaking so much that I’ve now set-up a Perforce server at home so I can track, audit and keep a log of my changes. As I’ve explored the WordPress code base over the months, I’ve noticed that it is not the prettiest of code in place and again I’m not alone in noticing this.
Please don’t misunderstand me, I like WordPress a lot, it has made setting up and running this blog trivial, especially when compared with writing my own blog engine from scratch. I’m simply commenting on my perception of the quality of the underlying source code, something I think all programmers do especially with things they’ve written (although I did not write WordPress, I just modified it). It would be nice to see a code tidy up and the introduction of templates and caching to the core engine, but that is a big job to undertake and I don’t think I have the required amount of spare time to achieve it myself.
All this has got me thinking about the recent(ish) upsurge in the number of web development frameworks that are embracing things like Object Relational Mapping (ORM) and development models like the Model-View-Controller (MVC) pattern. The two programming languages that seem to be leading this charge are Python with DJango and Ruby with its Rails framework, no doubt there are other frameworks in other languages too (e.g. ASP MVC, php’s symfony framework). The increase in popularity of these frameworks with built in template engines and abstraction of content from layout and logic via MVC are making me consider setting out on the path of writing my own personal blog engine again.
Mostly writing my own blog engine is a way to learn and embrace these new technologies and concepts and also a chance to see what all the fuss is about. This would mean leaving the cosy comfort of my WordPress armchair, so most likely this experiment will become something I initially develop in a test sandbox at home until I have something that is worthy of publishing online. My decision now is what language and framework to use. Do I stick to php with Smarty and MySQL or do I try something new like Python’s Django or Ruby on Rails? Right now I’m tempted by either Python or Ruby for reasons I will talk about in a post tomorrow.
Adventures with Twitter
I decided to try out Twitter at the end of last week, mostly as I’d heard so much about it but not really looked into it seriously more than a quick glance over the demo page. I don’t know what made me suddenly sign up for the service I think I was wanting to more quality information. This is a common occurrence for me as I read very quickly and even with search engines as good as they are it is still hard to find quality sties to read on any given subject. So I decided to jump into Twitter with both feet and see what it is like…
You can find my Twitterings on this page, I should probably add a link on the side bar at some point.
So the first thing I noticed after signing up and getting my account activated was that the ‘Find People’ search seems to be disabled, which makes find other people to listen to hard. Fortunately for me I knew Jeff Atwood of Coding Horror and StackOverflow was an keen twitter user so I managed to find him and Joel Spolsky of Joel on Software via some Google searching. I also found some more interesting people to follow by looking through who they were following. But the lack of a working search feature had me stunned for a while that a service could be so popular and yet be missing such a fundamental feature.
Needless to say I eventually discovered search.twitter.com. Although I can’t remember how I found it, I think it may have been a link I followed after searching through some blog posts regarding finding people on twitter. I find it bizarre that the link to their main search page is not more prominent than it is: I had to search through the html source of the homepage to find it (its in the footer on the right side). Unless the developers are actively trying to lower the hits on the search pages for performance reasons: maybe their server was getting swamped?. Why else would they place the link to their search page (a key feature) in such an obscure location?
Anyway enough about the search page being not obvious to find. So far I have found twitter fairly useful in terms of content and information: I would not have found Cal Henderson’s excellant ‘Why I hate Django’ presentation from DjangoCon 2008 if I hadn’t seen a link to it on twitter. I also like reading some of the updates from bloggers I respect allot like Jeff, Joel and Rands as well as watching the flow of information from people talking to those guys. I’ve also found some more interesting people to follow about web development, php, django and a few other topics via the search page now.
The last thing I’m going to mention is the apparent use of some sort of informal keyword/tag system that seems to be being employed by people, it is as simple as putting a hash (#) in front of the word you want to have as a keyword or tag. Judging from the amount of times I’ve seen tags like #webdev, #php, #django this seems to be getting used allot by the more technical members of the twitter community. It would be interesting to see if there is anyway of filtering these posts by their keywords into channels for each keyword, that would make very interesting reading indeed. And I’m sure someone has probably already done it, I’ve just not found the link yet.
Configuring Perforce command line client P4 on Mac OS X
I wrote a post yesterday about setting up the Perforce server P4D on Mac OS X once you’ve done that all that is left is to configure P4V the visual client and P4 the command line client. Fortunately P4V has a helpful set-up wizard to help you configure it for the first time: connect to the server, create your new user account, new client spec and sync up for the first time.
The command line utility P4 takes a little bit more configuration via environment variables which is not quite as simple as setting up P4V, so I thought I would document it. The first thing I did was run P4V and configured it, this has the side effect of working out the perforce port, creating a user account and associated client spec. Take a note of the username, clientspec and perforce port as you setup P4V, you’ll use those values to configure P4.
Note for this to work you need to have P4 already set-up in your system’s path variable. If you don’t have P4 in your path when you type ‘p4‘ and hit enter in a terminal window you will get a response along the lines of ‘-bash: p4: command not found‘. If you don’t have this already set-up and you are using the default bash terminal on Mac OS X, then you can add P4 to your path by doing the following:
- Open a Terminal window, (its in the utilities folder in the application folder).
- Type vi .bash_profile this will open the hidden file that defines your bash profile file in the vi text editor.
- Edit the line (press i to start editing and esc to stop) that starts with path=, insert the path to where you put p4 (in red) followed by a : right after path= e.g. PATH=”/Developer/Perforce/:…:${PATH}”.
- Save the file (with :s then enter) and then exit vi (with :q then enter).
- Quit Terminal.
- Reopen Terminal.
- Type p4 and hit enter, you should now get a response from the p4 application, although this will most likely get an error response unless you’ve already set-up your Perforce environment variables.
Now we can configure the three environment variables that the P4 application requires to work, these are $P4PORT, $P4USER and $P4CLIENT. These variables tell the P4 application how to connect to the server and the user to connect as and clientspec to connect with for that user. As I mentioned above the values of these three variables are the same as the perforce port, username and clientspec you used in P4V. To setup these environment variables we are going to add them to a hidden system file which is run every time you login to set-up the environment variables for your user profile (from this guide).
- Open a Terminal window.
- Type mkdir .MacOSX and hit enter to create the hidden directory the file resides in, the dot at the start means it is a hidden file.
- Type echo “” > .MacOSX/environment.plist and hit enter to create an empty file called environment.plist in your user directory directory, which is hidden.
- Type open .MacOSX/environment.plist and hit enter to open the file we just created in the plist editor.
- Add a child value with name P4PORT and the value from P4V e.g. localhost:1666.
- Add a sibling value with name P4USER with the value of your user from P4V e.g. daniel.
- Add a sibling value with name P4CLIENT with the value of the clientspec for your user in P4V e.g. daniels_clientspec.
- Save the file and exit the editor.
- Quit terminal.
- Logout and log back in.
- Open terminal and type P4 and hit enter, you should now get a non-error response from P4.
It is worth configuring the P4 command line application as it is the main way to administer a perforce server, so if you want add, delete or manage users, depots etc then you need to have P4 configured.
