Category Archives: Development

Sublime Text: Always run a single file in a project no matter what is focused

When I program, I often try to split up my code into modules. This requires that I use different files. However, I also want to run my code from a “main” or a “master” file as well. It’s pretty annoying in Sublime Text, or any text editor, to have to switch to your main file to run your project.

I’ve talked about how to do this in TextMate but how can you pull this off in Sublime Text? Let me show you how.

  1. Create a Sublime Text project for your program. You can do this by using the menu Project > Save Project As…”.
  2. Open the settings for your project using the menu Project > Edit Project.
  3. You’ll get a mostly empty text window. Modify this window to include a build definition, such as the following one for Python:
    • "build_systems": [
          "name": "PyProj",
          "cmd": ["python", "-u", "$file"],
          "file_regex": "^[ ]*File \"(...*?)\", line ([0-9]*)",
          "selector": "source.python"
  4. Modify the line with “cmd” and change “$file” to the master file you always want to run. For example, if your main file is called, then change it to
Add the "build_systems" JSON field in your project settings file, accessible through the Project menu > Edit project.

Add the “build_systems” JSON field in your project settings file, accessible through the Project menu > Edit project.

That’s it! While you’re editing your project settings, you can make other customizations as well as you see fit.

Obviously, this isn’t limited to just setting a file to be run no matter what file you’re focused on. You could use this to add command-line arguments to a single project, to run project-specific scripts for building and testing, or to otherwise customize the way the “Control-B” (Command-B on Mac OS X) works when you are running the project. You can essentially define your own build system on a per-project basis in a project file’s “build_systems” section and use any information in Sublime Text’s build system documentation to do so. Unfortunately, the build system documentation doesn’t really describe this because it’s focused more toward Package development.

For more information about how project settings work, look at Sublime Text’s documentation for build systems and for projects!


A Brief Ludum Dare 26 Update: Results

The final results are in for Ludum Dare 26, a game competition in which participants have 48 hours to program a game from scratch. As I previously wrote, I participated in Ludum Dare for the first time last month and learned an immense amount in an extremely short amount of time. Here are the ratings I received for Painter’s Cat. The ratings were out of 5, and there were 1610 games submitted to the Compo category (which I participated in).


Coolness 60%
#47 Audio 3.85
#176 Innovation 3.64
#187 Humor 2.96
#219 Mood 3.33
#265 Theme 3.87
#457 Overall 3.26
#569 Graphics 3.00
#700 Fun 2.82

(For those who are wondering, Coolness is a measure of how involved you are in the community, especially with playing, rating, and leaving comments on other people’s games. The more you play and comment and rate other games, the higher your coolness is).

First of all… I’m in the Top 100 for Audio at #47? Holy crap. That’s amazing. I’m pretty blown away by this – all of that time doing One Hour Compos has really paid off! I couldn’t find where my overall ranking was, but it looks like I managed to place in the top 40%, which isn’t bad for a first game.

Ludum Dare was an amazing positive experience for me overall. I’m pretty excited for next time. There’s still a lot for me to learn and this is one way to help me not only learn about software development and game development, but to also have fun doing it.

TextMate and Running a Master File

Many people find my blog searching about how to make their multi-file LaTeX projects build in TextMate.

It’s worth nothing that the TM_MASTER_FILE works for almost any programming project, not just LaTeX. So, if you’re working on a Python or a Ruby project and have a main file that you want to run, set the TM_PROJECT_MASTER to point to that and save yourself some effort. Here’s a reminder on how to do that.

  1. Open your project file (or create one if you don’t have one).
  2. De-select any selected files in the TextMate drawer by clicking on empty space.
  3. Click on the “i” in the bottom-right corner of the drawer.
  4. Add a variable named TM_PROJECT_MASTER.
  5. Set the value to the name of your main file (in my Ludum Dare game, it was

Now your “run” ⌘r shortcut will always run your main file with minimal hassle!

Inheritance in Javascript: Getting it Done from a Newcomer’s Perspective

The Idea Garden is made up of a number of suggestions that each follow a certain structure. The person who wants to use the Idea Garden in a new environment has to write templates and a bit of boilerplate code. As part of the Idea Garden meets Gidget project, I’ve been doing some work porting the Idea Garden architecture from a Firefox plugin, CoScripter.

The purpose of this post is to discuss how I used a particular style of Javascript prototype inheritance that I discovered on Stack Overflow. In some respects, it’s trying to communicate to new developers from the perspective of a new Javascript developer and explain some gotchas on the way. This isn’t meant to be a really in-depth guide to Javascript prototypes or the Javascript object model – there’s some good reading on the Internet already that discusses the intricacies of the language.

The Prototype-based Programming Model

Javascript’s object-oriented model is prototype-based rather than class-based: what this means, practically, is that unlike most classical object-oriented languages (Java, Smalltalk, C++, Python, Ruby, etc.), there is no distinction between a class definition and an object. In other words, as soon as you “declare” a “class” in Javascript, it exists in the world immediately!

This has led to some oddities in Javascript, such as there being at least two ways to declare a new class, with each of these ways looking a little different from each other. In some respects, trying to think in terms of normal object-oriented programming in Javascript only gets you mired in some unnatural programming practices.

Because Javascript isn’t classicly object-oriented, trying to comprehend its inheritance model is a bit tricky. For this reason, along with reasons outlined in the Gang of Four book), and others, I’d actually recommend composition over inheritance.

But, in this particular case, I was asked to implement an architecture to use inheritance because we have a number of Idea Garden suggestions, and Idea Garden suggestions behave similarly and borrow a number of common functions, but might have some small specialized differences in their content.

Which Inheritance Style, Exactly?

There is no shortage of Javascript inheritance styles. Because there’s really no accepted “way” of doing it (or of NOT doing it), anyone who wants to do it has to really work at studying what they want, exactly.

In the end, I emulated a style following this Stack Overflow post by Sean McMillan because it (1) used the Javascript Module pattern, which was pervasive throughout this code and (2) used Object.create, which basically does the inheritance for me (Object.create is ECMAScript 5 and from what I understand, basically replaces new). However, the answer, while providing a pretty good code example, didn’t explain all of the concepts behind it that I needed to know.

The Return of the Prototype

As I mentioned above, Javascript is a protoype-based language, and as such doesn’t have class definitions. The module pattern probably comes the closest to emulating a “class definition”, but is still not exactly a class definition. If I create a module:

IdeaGarden = (function() {
	var exports = {},
		environmentName = "Gidget";

	exports.hostInfoToIdeagarden = function() {
		// Do stuff here

	return exports;

it exists as soon as the code runs, and I can modify the object’s state, use its functions, and so forth, without doing an Object.create or new call.

An important part of inheritance is to separate the definition from the instantiation. In general, you want to inherit from a certain state of class, rather than whatever you happened to have used and extended most recently.

To do this, we define a prototype in our module that we will inherit from when we do an Object.create.

Vehicle = {};

Vehicle.AbstractVehicle = (function() {
	var exports = {};
	exports.prototype = {};
	exports.prototype.init = function() {
		this.identifier = "Abstract";
		this.passengers = 0;
		this.speed = 0;
		this.running = false;

	exports.prototype.getName = function() {
		return this.identifier;

	exports.prototype.accelerate = function() {
		if (this.running) {
			this.speed = this.speed + 10 - this.passengers;

	exports.create = function(identifier) {
		var ret = Object.create(exports.prototype);
		return ret;

	return exports;


This code, which pretty much conceptually matches to Sean’s code in his post, illustrates a few important points.

First, we have a callable “exports.create”, which creates a prototype of the base class, runs its “init” function, and returns the prototype for us to use. The fact that it returns a prototype is important! Unlike in the normal module pattern, where you would declare additional private members somewhere in the body of Vehicle, in the inherited class from the prototype, only members declared using this in the init function will be visible to instantiated classes and their descendants.

The difference between “what belongs to the module” and “what belongs to the new created class” isn’t exactly difficult, but it tripped me up as a beginner because of the close association between the module class having its own private members and the members appearing in the init function.

Because the create function returns a prototype, everything needs to exist in the scope of the prototype, rather than in the module!

The corrilary to this is that public members need to exist in the init function and that public methods need to be part of the prototype, rather than the module. Here’s an example method:

exports.prototype.accelerate = function() {
		if (this.running) {
			this.speed = this.speed + 10 - this.passengers;

Inheriting from the Base class

Once you realise the importance of returning the prototype, the rest of it pretty much falls into place. Since you’re using Object.create to create a new version of the prototype, you can create multiple versions of the base class and modify them freely without worrying about changing the definitions(that’s something that you would have to worry about otherwise). Here’s some code to inherit from the base class.

Vehicle.Car = (function() {
	var exports = {};
	exports.prototype = Object.create(Vehicle.AbstractVehicle.prototype);
	exports.prototype.init = function(identifier) {
		Vehicle.AbstractVehicle.prototype.init.apply(this, [identifier]);
		this.identifier = identifier;
		this.passengers = 4;

	exports.prototype.cruiseControl = function(newSpeed) {
		while (this.speed < newSpeed) {
			this.speed += 1;

	exports.prototype.startVehicle = function() {
		console.log("Starting the I4");
		this.running = true;

	exports.create = function(identifier) {
		var ret = Object.create(exports.prototype);
		return ret;

	return exports;

Let’s go through some of this code as well.

First, you’ll notice that exports.prototype in this version does an Object.create(Vehicle.AbstractVehicle.prototype). This basically matches the prototype to the base class. In the exports.prototype.init function, we call the init from the base class as well – it’s essentially equivalent to our “super” call from other languages. Note that we pass in our arguments (identifier in this case) in an array – that’s just how apply works. Our variables from the base class will be initialized at that time, along with any additional variables we want here.

As mentioned above, because we return the prototype, every method that we want to extend the base class with needs to be preceeded with prototype. That’s why we have exports.prototype.cruiseControl and exports.prototype.startVehicle. Note that we can use any of the variables and methods we declared in the base class within these methods as long as we precede them with this.

Finally, we have our create function, which is NOT in the prototype space because we want to be able to call it from our code. This is pretty much boiler-plate code from the base class but we need it.

Now, we can use this just as we might expect (with a little help from JQuery assigning an identifier to outputDOM):

var workCar = Vehicle.Car.create("Work");
workCar.startVehicle(); // Outputs "Starting the I4 of Work" to the console.
outputDOM.append(workCar.getName() + ". Speed: " + workCar.speed + "
"); // Outputs "Work. Speed: 0". workCar.accelerate(); outputDOM.append(workCar.getName() + ". Speed: " + workCar.speed + "
"); // Outputs "Work. Speed: 6"

If we create another car using Vehicle.Car.create it’ll hold its own set of variables.

In Closing

I find Javascript is a little bit of an adventure because it’s a language that appears to have evolved to do things that no one envisioned it to do fifteen years ago. It used to be written directly in HTML as part of “onclick” attributes, and now it’s manipulating the DOM, applying CSS elements, and animating on-screen. Everything’s evolved heavily in the back-end as well. Douglas Crockford attempted to apply a classical inheritance model to Javascript but then gave up, eventually saying that “it was a mistake”. However, a lot of this information is still on the web, and it’s difficult to identify what the state-of-the-practice is for Javascript these days.

I do want to, however, keep up to date and be sure that if I’m writing Javascript now, it’ll be recognizable to people who are reading my code two, three, five, ten years from now. I like trying to keep code reasonably simple and clean – especially in a situation like the one I’m in where others are going to be working off of the code that I started.

If anyone notices that I’ve made a mistake in my understanding, or would like to recommend other ways to accomplish what I’ve outlined here, I’d be glad to hear about it and update this post accordingly – I’m not a Javascript expery by any means, but by writing and self-reflecting on what I’ve done, I would like to keep improving my skills and my understanding. So, if you have any suggestions or improvements please feel free to drop me a line or leave a comment.

The Challenges and Rewards of Non-Competitive “Compos”

As I mentioned in my previous posts, I participated in Ludum Dare, a game development “compo”, where you build a video game by yourself, from scratch, in 48 hours (a variation gives 72 hours and a team, but I did the 48-hour version).

A “compo” is a “composition competition”, but I’ve yet to participate in a compo where the competition aspect is what actually gets people energized. If anything, the compo is more community-oriented than it is competition-oriented. I participate weekly at One Hour Compo, which is a music compo in which you create a song from scratch in one hour. The competition aspect supposedly is because people vote on their favourites at the end, but in the end I don’t think the votes are what anyone’s really fighting for.

In any case, I feel that compo has probably been one of the top ways for me to improve my music making skills and that doing the Ludum Dare compo was an excellent way for me to simply program for the sheer joy of it.

But, I think one of the greatest benefits of doing these compos is simply being able to succeed, and to feel happy and proud doing it. It’s a real self-esteem booster and it also helps you beef up your skills and the ability for you to work under huge time pressure.

Below I’m going to present the post that I wrote for Ludum Dare 26. Most people there tend to write technical post-mortems, but I thought that the emotional barrier was actually a bigger barrier to cross than the technical ones!

An earlier version of this post below first appeared on my Ludum Dare 26 blog.

I think I’m over my Ludum Dare Adrenaline Rush now.

In my first initial post, I said that I was going to fake Ludum Dare. To my surprise, a few commenters actually wrote in and encouraged me to just enter anyway. I did. And I came out with a game that had some interesting ideas, along with a number of problems.

I’m actually happy I participated and would like to thank the handful of people who encouraged me (some strangers, some friends) to enter anyway. It was a great experience and I want to be around next time, if I have time for it.

But, what led me to go down a road of, “I don’t think I can do it?”

I’ve never designed a game before

Well, technically I did. In my first ever C programming class when I was 15, the final project was a video game. Mine compiled but didn’t work – we had Macs at school, and when I realised that my game wasn’t getting close to finished, I brought my game home, wrote almost all of my code on my PC, and hand-checked it to see if it would work, in theory, when I brought into school the next day.

With some work I made it compile, but it didn’t really work as I wanted it to.

In some respect though, it doesn’t take much to be a game. I’ve seen a lot of things that people recognize as games. Top-down shooters, side-scrollers, role-playing games, adventure games, text-based games, board games, and so forth. But I’ve also seen a lot of creative work as well. A game where the main idea was to walk in a city. A game where  you woke up, experienced a main character’s morning, and looked at all of the objects in his or her room. A game where you planted seeds in a garden and watched them grow. They’re barely games by the standard definition – but they are all welcome in Ludum Dare.

Even if you don’t have a strong idea of what you want to do, build a game archetype anyway and then see what comes out. In some respect, one of the thrills of doing a game in 48 hours is that the first few ideas you get, you have to stick with because there’s no time to really make it better. So you get all of these raw, unrefined concepts that are the pure essence of creativity. And it’s great to see so many of these concepts work.

My Programming Chops

I don’t program a lot. I have a computer science degree so I know how to program, but my work is primarily focused on research activities that don’t require any development. I find a lot of programming boring – especially mathematics riddles. “Compute the least-common denominator of two numbers?” Snooze. “Write an algorithm that will identify is a string is a palindrome?” Ugh.

But then I start building a game and suddenly, everything is fun, even when I groan at thinking about the geometry and trigonometry. It’s because those things suddenly aren’t just mathematics. They’re situated in my game as a core concept now. I don’t need to understand them for the sake of knowing them – I’m understanding them because I know that they’re useful to me, now.

I learned a lot on the fly, and suddenly I realized that programming isn’t just about what I know – but about how fast I can learn what I need to know. I didn’t know PyGame existed until the morning I decided to do Ludum Dare. I didn’t know how to blit a sprite to a screen before reading about it on my lunch break. I had never thought about sprite rectangles, mouse movements and controls, or drawing tiled backgrounds until the hour the competition started – so I learned those things with a lot of help from the Internet. I can’t say that I know all of those things well, even now… but I’ve done them before now and I can only improve from here.

So even if you’re not a hot-shot programmer, it’s not just about how well you program – it’s about how well you can get what you need done by learning what you need to learn.

Ludum Dare and self-esteem

I think a lot of people who post on this site have a lot of confidence – you have to to enter something that is billed (albiet weakly) as a competition. The games that get all of the press are the ones that have the shiniest graphics, the best lighting engines, the cutest artwork, the most thrilling sound. In the end, history remembers the winners, and all of the winners kind of blob up together into this gigantic “super-winner” amoeba where it feels like one guy participated in 30 Ludum Dares and came up with a hundred amazing games along with a procedural level generator and a memory-management allocation system in the span of a week. The secret though is that this mythical superhuman game programmer doesn’t exist. That programmer is really hundreds of individuals

I think one big lesson to learn from this is that very few of us are superhuman, and more importantly, the majority of people who participate in a Ludum Dare are normal people. They’re not all rock-star programmers, hotshot artists, or amazing musicians. They’re regular people and normal people. Just like you, just like me.

The second point to remember is that Ludum Dare isn’t really a competition. Sure, it’s about getting votes and comments and people get into the top list at the end, but in the end there’s no reward and there’s very few bragging rights. This really isn’t a competition.

I think a big outcome of these two points is that Ludum Dare is a showcase. It’s not just “how good your game is”. It’s the fact that you’ve managed to produce a game at all. No where else would I have been able to build almost any program in 48 hours and then, in the span of under a week (so far!) convince 50 people to play and download my game and leave constructive comments.

In that sense, everyone’s a winner. And even though I said above that almost everyone in Ludum Dare is a “normal person”, the fact that we’ve programmed something from absolutely nothing to a working, deliverable product in 48 hours (or in a team in 72 hours) makes every one of us extraordinary.

I hadn’t been as excited about something before as I had right after Ludum Dare – and I think that’s because that, as soon as I had finished, I realized that I had done something extraordinary. A few thousand of us, together, had each accomplished something to be proud of.

Using PyInstaller to make EXEs from Python scripts (and a 48-hour game design compo)

How to Create a Single Windows Executable from a Python and PyGame Project (Summary)

Here’s how you use PyInstaller and PyGame to create a single-file executable from a project that has a data directory that contains resources like images, fonts, and music.

  1. Get PyInstaller.
    • On Windows, you might also need pywin32 (and possibly MinGW if you don’t have Visual Studio).
    • On Mac OS X, you will need XCode’s command line tools. To install the Command Line tools, first install XCode from the App Store, then go to Preferences – Downloads and there is an option to download them there.
  2. Modify your code so that whenever you refer to your data directory, you wrap it using the following function:
    def resource_path(relative):
        if hasattr(sys, "_MEIPASS"):
            return os.path.join(sys._MEIPASS, relative)
        return os.path.join(relative)

    An example of usage would be

    filename = 'freesansbold.ttf'
    myfontfile = resource_path(os.path.join(data_dir, filename)

    This is mostly for convenience – it allows you to access your resources while developing, but then it’ll add the right prefix when it’s in the deployment environment.

  3. Specify exactly where your fonts are (and include them in the data directory). In other words, don’t use font = Font(None, 26). Instead, use something like font = Font(resource_path(os.path.join('data', 'freesansbold.ttf')), 14).
  4. Generate the .spec file.
    • Windows: (You want a single EXE file with your data in it, hence --onefile).
      python --onefile
    • Mac OS X: (You want an App bundle with windowed output, hence --windowed).
      python --windowed
  5. Modify the .spec file so that you add your data directory (note that these paths are relative paths to your main directory.
    • Windows: Modify the section where it says exe EXE = (pyz, and add on the next line:
      Tree('data', prefix='data'),
    • Mac OS X: Modify the section where it says app = BUNDLE(coll, and add on the next line:
      Tree('data', prefix='data'),
  6. Rebuild your package.
    python your_main_file.spec
  7. Look for your .exe or your .app bundle in the dist directory.

Phew! That took me a long time – the better part of a few hours to figure out. This post on the PyInstaller list really helped.

So why was I trying to package a Python executable file anyway? Read on…

Ludum Dare 26: 48-hour Game Design Compo

This weekend, I decided to participate in a 48-hour game design “competition”. Ludum Dare is a compo that asks you to create a video game from scratch in a 48-hour time period – you have to write your code and create all of your assets in that time period.

This means no reusing graphics, pictures, music, or sound from other projects, for example. You’re also not supposed to reuse code either. I decided to participate on the Thursday the day before. Most people use the previous weekend as a “warmup weekend” to test their tools, get some practice, and so forth. (My entry is located here, by the way).

I’ll do a more detailed compo writeup later, but I just want to concentrate on one thing that kept me up for hours after the competition: getting a Windows executable created from a Python project that uses PyGame and a data directory.

Python, Distribution, and You

I rather enjoy Python as a programming language. The syntax is reasonably concise, the language does a lot of things for you, and it’s well-laid out. There’s also a lot of good support in the form of third-party libraries. I’ve been using Python for various things for the past few years (usually small scripts for data extraction and analysis in research).

One thing I had never thought about before was distributing a Python project as an executable package, and while it was on my mind throughout the entire compo, I didn’t actually learn the process of creating the package until the last hour of the comp before submission. After you submit your primary platform, Ludum Dare allows you around 48 hours to compile for Windows, since the majority of reviewers use Windows.

The ideal submission is a single binary file (an .exe file for Windows) that doesn’t have to extract a lot of data, so that it’s easy for people to download and run your game.

PyInstaller vs. Py2exe vs. Py2app

I went on a wild goose chase trying to find out how to make a single executable file out of a Python project that would include all of my data assets. I first tried py2exe and py2app. py2app mostly worked all right, but py2exe was a pretty big mess.

The end story is that PyInstaller is newer and shinier than py2exe, and that you need to secret sauce code that someone out there on the Internet found before I did. PyInstaller basically runs EXE files by extracting the assets into a temporary data file that has a path _MEIPASS in it ((technical details here). Be sure that you check that every file is loaded in through that wrapper. The Tree() TOC syntax was also confusing, but basically, it’s the relative path of your data files and it will automatically load all of the files in that directory. Make sure it exists in the EXE portion (Windows) or the APP portion (Mac).

There’s a Make/Build cycle in PyInstaller to generate the spec file and build it in a single step as well – I find it easier to do that to generate the spec file and do an initial binary run, then to modify the spec and run PyInstaller again with the spec file as the argument. PyInstaller is pretty smart about rebuilding, and you save a lot of time.

I think in the long run, if you compare py2exe, py2app, and PyInstaller, PyInstaller is the program worth learning. It did have a pretty sharp curve for me – it didn’t help that I was trying to do this late at night after a challenging weekend!

If you do wish to use py2app to build your Mac OS X application bundle, then do keep in mind that you need to have a import pygame._view because of some kind of obscure issue.

Anyway, that’s all there is to this post for now.


Here’s the I used for py2app.

from setuptools import setup

APP = ['']
DATA_FILES = ['data']

    "argv_emulation": False,
    "compressed" : True,
    # "iconfile":'data/game.icns',        

    options={'py2app': OPTIONS},

The Whats and Hows of Programmers’ Foraging Diets: What Types of Information are Programmers Looking for?

Information seeking is one of the most important activities in human-computer interaction! One of the most influential theories in understanding, modelling, and predicting information seeking is information foraging theory. In our research, we want to understand what kinds of diets – that is, the types of information goals programmers seek while debugging. By investigating the information diets of professional programmers from an information foraging theory perspective, our work aims to help bridge the gap between results from software engineering research and Information Foraging Theory foundations as well as results from human-computer interaction research.

A pork chop taken by johnnystilletto on Flickr

Is this tasty?

A head of broccoli by Jim Mead

Is this tasty?

My co-author, David Piorkowski, is travelling soon to Paris to present our latest work: “The Whats and Hows of Programmers’ Foraging Diets”. It’s a great time to expand on this paper. Here’s the PDF Preprint!

Our Method

We had two coders examine video of nine professional programmers to identify what exactly they were looking for when trying to fix a bug in an unfamiliar open-source program. We tried to identify their overall diet by identifying if they asked questions (and received answers) belonging to one of four categories: (1) finding a place to start in code, (2) expanding on that initial starting point, (3) understanding a group of code, or (4) understanding groups of groups of code.

What is a programmer’s diet while debugging?

Overall, we found that programmers spend 50% of their debugging time foraging for information.

Surprisingly, even though all participants were pursuing the same overall goal (the bug), they sought highly diverse diets. For example, Participant 2 asked mostly about groups of groups, Participant 3 asked about finding a place to start, Participant 5 didn’t really ask about anything at all, and Participant 6 also looked for a place to start. This suggests a need for debugging tools to support “long tail” demand curves of programmer information.

How did a programmer consume these diets?

How exactly did programmers go about finding what they wanted to consume?

Again, participants used a diverse mix of strategies. Participants spent only 24% of their time following between-patch foraging strategies (such as code inspection or simply reading the package explorer straight up-and-down), but between-patch foraging (such as doing data flow or control flow) has received most of the research attention.

Surprisingly, search was not a very popular strategy, accounting for less than 15% of participants’ information foraging – and not used at all by 4 of our 9 participants—suggesting that tool support is still critical for non-search strategies in debugging!

Whats Meets Hows

Participants stubbornly pursued particular information in the face of high costs and meager returns. Some participants followed a single pattern over and over again, using the same strategy. For example, in the cases that involved a programmer looking for Type 1-initial goals, participants used code search and spatial strategies extensively but not particularly fruitfully. This emphasizes a key difference between software development and other foraging domains: the highly selective nature of programmers’ dietary needs!


Thus, we considered what programmers want in their diets and how they forage to fulfill each of their dietary needs. Our results suggest that the diet perspective can help reveal when programming tools help to reduce this net demand—and when they do not—during the 50% of debugging time programmers spend foraging.

References and Links

Are you going to be at CHI 2013? Where and when is David’s talk?  It’s on Thursday, May 2, at 11:00 in Room Blue… be there!

D. Piorkowski, S. D. Fleming, I. Kwan, M. Burnett, C. Scaffidi, R. Bellamy, J. Jordhal. The Whats and Hows of Programmers’ Foraging Diets, to appear in ACM Conference on Human-Computer Interaction (CHI), Paris, France, 2013. PDF Preprint

Our paper on the CHI 2013 web site

And… in case you haven’t seen it yet, the video preview!

Picture of tasty pork chop by Johnny Stilleto. Picture of tasty broccoli by Jim Mead.

Automating the Web with Selenium: Complete Tasks Automatically and Write Test Cases!

While teaching Software Engineering I during the Winter 2013 term, I learned of a web testing suite named Selenium.

I was on the lookout for a good unit testing suite for Javascript. I had previously been introduced to the YUI Testing Framework, which provides a console and enables you to easily write and run tests from a browser window, but one limitation of YUI is that, out of the box, it doesn’t support interaction with the site itself. So, while the basic usage is good for verifying libraries and similar, I wanted something that was able to interact with page elements in a different way.

Enter Selenium. Now, here’s a way to not only automate web interactions, but to make them into automated unit tests for my research projects. This post is going to serve as a brief introduction to Selenium and how to start using it to interact with websites.


Selenium is a web automation framework that enables a user to essentially script a web site. In this post we’ll talk about using Selenium version 2, which is the main, up-to-date version of the software.

There are two main ways for you to interact with a website.

The Selenium IDE: Recording actions by Demonstration

First, you can use the Selenium IDE, which is a Firefox plugin that allows you to demonstrate interactions with a web site. You essentially record actions and then as you click around the page, type in form elements, and push buttons, the IDE records the actions for you in a pseudo-markup language. (If you happen to follow my other research, it is actually somewhat similar to CoScripter, another program-by-demonstration tool for the web, but with better web interaction and fewer features. For example, Selenium cannot store temporary data into tables.)

Thus, even with minimal web development or programming experience, you could create a script for Selenium that plays back, for example, a series of clicks on a page, completes a survey, or similar.

There are some limitations with the IDE. The main one that led me to using the WebDriver (below) is that it doesn’t interact very well with “contentEditable” div tags and other HTML5 elements.

Note that the Selenium IDE is version 1.x, which does NOT correspond to Selenium itself (which is version 2.x).

The Selenium WebDriver: Programming actions in code

The second way to interact with a website is using the Selenium Webdriver, which is a driver that essentially launches a website and then enables you to look through that website’s DOM to interact with elements on the page. Thus, you can use your own browser, like Firefox or Internet Explorer to load and navigate a web site.

I wanted Selenium to be able to work with a highly interactive web app: Gidget. Gidget is a programming game for kids and teenagers that I am working on in collaboration with Andy Ko and Michael Lee. Gidget runs with a lot of HTML5, JQuery, and Javascript and is extremely visual – something that seems perfect for Selenium.

Unfortunately, since Gidget isn’t available publicly yet, I can’t actually put tests on the blog that’ll directly run Gidget so instead I’ll use Google.

Building a Selenium Script

Selenium has a number of bindings in Java, Ruby, Python, and Javascript. I personally chose to use Python – I like its concise syntax and the fact that it has pretty good library support. I’ll focus exclusively on Python in this particular post. Most of the Java bindings can be derived from the Python commands if you remove the underscores and instead use CamelCase – for example, “element.is_enabled()” in Python would be “element.isEnabled()” in Java.

To install Selenium, you generally need to only type

pip install selenium

assuming that Python exists already on your system.

The first place I’m going to point you at is the Selenium Documentation Example of WebDriver, where they already include an example of making a query on Google. Here’s a reproduction of the code.

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from import WebDriverWait
from import expected_conditions as EC

# Create a new instance of the Firefox driver
driver = webdriver.Firefox()

# go to the google home page

# find the element that's name attribute is q (the google search box)
inputElement = driver.find_element_by_name("q")

# type in the search

# submit the form (although google automatically searches now without submitting)

# the page is ajaxy so the title is originally this:
print driver.title

    # we have to wait for the page to refresh, the last thing that seems to be updated is the title
    WebDriverWait(driver, 10).until(EC.title_contains("cheese!"))

    # You should see "cheese! - Google Search"
    print driver.title


It works exactly as advertised and opens up a web browser, goes to Google, and then searches for “cheese!”. However, there are a number of features that are essential to actually testing web pages.

Waiting for Pages

The unfortunate reality of web pages nowadays is that you have to wait a lot. Whether it’s a form submit where you have to wait for the POST request to complete, or some really slow JQuery fading box, not everything you want to interact with is available. To get around this, you have to use the “wait” commands in Selenium.

The Selenium documents do provide a few examples but I had to do a lot of searching and testing to get things working so I’ll just provide my use cases here directly.

To wait for an element to appear in Selenium, you need to provide an explicit wait along with a condition. It basically waits until either the identified element loads, or until the timeout passes (at which point it will throw an exception). There is an example of that in the Selenium example above, but if you have Google Instant turned on, you’ll realise notice that Google now returns search results to you as soon as you start typing. How can you interact with page elements if you don’t even know what’s going to pop up, when?

In this case, we’re going to wait until the “Search Results” text pops up.

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from import WebDriverWait
from import Select
from import By
from import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException

if __name__ == "__main__":
    driver = webdriver.Firefox()
    wait = WebDriverWait(driver, 100)
    inputElement = driver.find_element_by_name("q")
    inputElement.send_keys("Irwin Kwan")
    wait.until(EC.element_to_be_clickable((By.XPATH, "//a[@href='']")))
    blog = driver.find_element_by_xpath("//a[@href='']")

Basically, the driver loads, we set up a “wait” that waits up to 100ms, and then we do a search on Google for my name. Note that we don’t actually submit the form – instead, we wait for the link to my blog to appear in the Google search results, find the element, then click on it. And yes, this is a cheap way to rack up those hits for my blog, so everyone run this code. 😉

There are a lot of expected conditions available in Selenium and this is probably the most important thing to be aware of when first starting. You can’t interact with page elements unless they’re available, after all. Here are some of the useful ones:

  • EC.element_to_be_clickable: Wait for the element to be clickable. Good for elements that aren’t always visible or enabled. I use this a lot in the game.
  • EC.visibility_of_element_located: Wait for the element to be visible. A lot of pages load all of their content, but hide it from the user. Here’s how you ensure that what’s being interacted with is actually visible.
  • EC.presence_of_element_located: The element exists somewhere on the page.

There are a lot of these expected conditions. A full list is available on the API document page.

The second key aspect here is the interaction. To do this, you have to “find” the element. So far, I’ve used two main ways of finding an element: ID, and XPath.

Finding elements by ID

Finding an element by ID is pretty much what it sounds like: searching for an element using its ID tag. ID tags are unique in the DOM and are therefore ideal for searching and testing. You can use it like this:


Finding elements by XPath

XPath is a query language designed to navigate XML. I won’t go through all of the details of XPath here, but I’ll present a few basic use cases. You basically can search on a number of conditions that you specify in the language so you can see if that element exists in the DOM.

  • If you want to search if a specific tag exists:
  • If you want to search for nested tags:
  • If you want to search that text within the tags matches:
    driver.find_element_by_xpath("//h1[text()='Heading 1']")
  • If you want to search for text in attributes
    driver.find_element_by_xpath("//img[@title='Irwin Kwan']")

With these two find_element commands, you can find most of what you need in your web pages. Selenium supports a number of other ways to search, including searching by CSS, but I haven’t needed to use it yet.

Dynamic Interactions with a Page

Another issue I encountered with the Gidget game is clicking through introduction text when I didn’t know how many pages were present. Essentially, the problem is that there’s a button that I have to push on the page, and if I push it a certain number of times, it’ll be disabled. However, I don’t know how many times I have to push it because it might be different each time.

I managed to get around this with a little fragment of code below:

wait.until(EC.element_to_be_clickable((By.ID, 'main_buttonMissionTextNext')))
while EC.element_to_be_clickable((By.ID,'main_buttonMissionTextNext')):
    if not driver.find_element_by_id("main_buttonMissionTextNext").click().is_enabled():
    wait.until(EC.element_to_be_clickable((By.ID, 'main_buttonMissionTextNext')))

There were two gotchas here: First, I wasn’t aware of it at the time, but the button was actually regenerated whenever you clicked it, so I had to search for it again when I clicked it. Second, I didn’t realise that there was an is_enabled() function you could use to test if an element is enabled or not. But now, you know!

I posted a StackOverflow question about this (which I ended up answering myself).

Using the Selenium IDE and the Selenium WebDriver Together To Save Time

While writing code is nice and fun, HTML pages are very large and are covered with tags with various IDs. It becomes tedious very quickly, even with good web development debugging tools, to search through the DOM elements to identify what you need to interact with, then writing the code to search and click on it.

So work smarter, not harder and use the Selenium IDE. If you record your actions with a page and save it in the Selenium IDE, you can use “Export Test Case As… > Python 2 / unittest / WebDriver”. Now you have Python code for that series of actions that you just performed and can integrate it into your other tests.

In my case, I used the Selenium IDE to automatically complete an exit survey in Gidget, then exported it and used the code in my other test suites. It’s a great way to save time writing code.

Making your code into a Test Suite

In Python, the unit testing is built in. You simply have to import unittest, create a class that extends it, and then write your setup and teardown functions, along with a method beginning with “test”. Here’s the previous code for interacting with Google converted into a Python test suite.

import unittest

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from import WebDriverWait
from import Select
from import By
from import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException

class WebTester(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()
        self.wait = WebDriverWait(self.driver, 100)
    def test_load_blog(self):
        inputElement = self.driver.find_element_by_name("q")
        inputElement.send_keys("Irwin Kwan")
        self.wait.until(EC.element_to_be_clickable((By.XPATH, "//a[@href='']")))
        blog = self.driver.find_element_by_xpath("//a[@href='']")
        self.wait.until(EC.visibility_of_element_located((By.XPATH, "//h2[text()='Irwin Kwan']")))
        self.assertTrue(self.driver.find_element_by_xpath("//h2[text()='Irwin Kwan']"))
    def tearDown(self):
if __name__ == "__main__":

If you place multiple test suite classes in the file, unittest will run them all. Selenium will launch a new, fresh browser instance for each one as well.

Conclusion: Web Automation is Pretty Cool

I’m really just a new user. Selenium has a number of features that I haven’t needed and therefore don’t know much about. There’s a “remote control” mode that allows you to use a separate server to run tests for you. There are ways to store session variables, load specific Firefox profiles with add-ons, and there is a “Selenium server” mode as well. If you need these features, chances are you’ll be able to find information about them on the Selenium site documents.

I feel that this information should set most people up with enough information to get started with Selenium and making it work for them in a useful way. I hope that this post is useful to you guys!

Happy automating!

Listing the contents on a web server without listing directories

I seem to require this snippet of code quite often.

This PHP code will list the contents of a web server’s directory. It will list only files, and not directories (so it’ll skip the parent directory link which shows up by default on most Apache Index files).

if ($handle = opendir('.')) {
	while (false !== ($file = readdir($handle))) {
		if ($file != "index.php" && strncmp($file, ".", 1) != 0) {
			echo "<li><a>" . $file . "</a></li>\n";