In the last part I covered the use of CruiseControl.net in the CI process, which is pretty much one of the most important parts of the whole system. Out of the box, CC.net is pretty useful, but it can be easily extended and this part is going to cover how we use CC.net and what we have done to get a bit more out of it.
The biggest problem with Cruise Control is the time it can take to report a new (or more importantly broken) build. At the start of a project this can be a quick turnaround, but later, when we have masses of code, more platforms and more build configurations, it can quite easily take hours to complete a full build. Since no-one should be going home on a non-green build, this either means they stay in the office until late, or don’t check in after 2pm. None of these are suitable solutions.
The quickest thing we do is reduce the number of configurations we build. The ‘profile’ configuration, which is rarely used by anyone, can often be removed, along with full debug builds (since these builds actually become unplayable near the end of a project anyway). We can also cut down on the number of platforms. A game might be destined for PS3, but it will probably start life on a PC. Once everyone has finally moved onto the target platform, a whole platform plus its configurations can be removed.
Finally, we never do a ‘rebuild’ on Cruise Control. While a ‘build’ sometimes needs a kick from behind due to linker errors or similar cropping up that full builds generally fix, this generates a massive saving of time, and the full build is always done at night anyway.
One technique that has been used in other teams at Blitz is to have a separate CC machine, which simply builds the release build of the title on the target platform. Not much use for the programmers, but an excellent time saver for designers and artists, who don’t care if the debug PS3 build is broken.
In one case we reduced the build time for a project from around 2 hours on average to about 20 minutes. A massive boost to productivity.
Extending Cruise Control Reporting
The Cruise Control Tray tool is useful, but it’s actually quite easy to miss the fail message and people do not always check their service tray to see if the CC is red. The easiest way we have extended this is for the CC machine to automatically send e-mails to all programmers on a team when the build is broken. This is apparently very easy to do with NANT, but our extensions are written in Ruby, simply because I had a lot of previously written Ruby libraries that were suitable for the job.
This can easily be extended to send a message to our Yammer service (which we use for inter-team communication already) and that is something that I hope to do in the future.
Tracking Build Statistics
It’s very easy to say “The build is always broken” but it’s also very useful to be able to have hard evidence for the % of broken versus fixed builds. Again, Cruise Control gives you the ability to track this by storing the build status as local Environment Variables which can be used to track the state. Along with easy access to the SVN log, we know who broke the build, when and how (was it a check-in or a forced build?). By hooking these into a simple php graph generation tool, statistics on the broken vs. working ratio and broken builds count.
I would never make the ‘who breaks the build the most’ information public, but it is an excellent tool for finding problems in the development process or if anyone’s way of working simply isn’t… working.
Obviously Cruise Control can have its limitations. Multiple projects, dependencies and configurations can soon add up and no amount of tweaks and omissions can stop this. Once the feedback loop is redundant due to it simply taking too long, Cruise Control can lose its impact and other methods need to be used to bring it back on track.
For example, at its worst case some projects can take over 4 hours to complete a set of dependant builds. If you are right at the end (which unfortunately I am!), the amount of information being give to you is dramatically limited. But we have made changes which mean this delay still takes place, but other information from the CC machine is fed back much quicker meaning it becomes useful once again.
So that’s generally how we use Cruise Control. If anyone reading this does anything differently then let me know as I’m always looking for ways to improve the tools use and make it more beneficial to the team.
In the next part I’ll look at the process of nightly builds and how that can help the rest of the game, since nothing so far has covered the fact that games are more than just code!