Continuous Integration, build processes and testing are all very important parts of the development process, whether it’s a AAA game, small arcade title or a set of middleware. It’s only in the last couple of years that this has really gone mainstream in the industry (it’s sort of swung in with the whole agile thing which has unfortunately led some people to think it can only be part of an agile process – which is thankfully not true) but it’s been used for decades in various forums throughout the IT industry.
For the next few posts I’m going to detail the CI process we use on all of the titles developed in Blitz Arcade. I’m going to keep each post short so I can actually get around to writing them without having to write it all in one go.
The best place to start is to describe the situation we have before we even start looking at a CI process so we at least know that we have a problem to solve…
So imagine we have a game in development, let’s call it Super Space Fighter VII, that is being developed for LIVE Arcade, PSN and will be released on PC via Steam. It is being created by 5 programmers, 3 artists and 2 designers and is going to be a simultaneous release on all platforms, so each milestone requires builds on all three platforms, with no scope to leave a build behind for later. Development kits are expensive and so to keep costs down we have 2 PS3 dev kits (both for the lead PS3 programmer) 3 Xbox 360 kits and the final programmer works solely on the PC.
This means this if one of the X360 programmers changes some code, they have no way of testing (or compiling if time is an issue) the code on the PS3, so the lead PS3 programmer is going to spend a large amount of time making other peoples code work on his platform, especially as the different compliers all have their own quirks. This is also the case for the PC developer and any changes that are not tested on the 360.
For testing purposes each programmer will have custom code on their machine (usually wrapped in a #ifdef/end block) that if not removed could be checked in and if a build is done on their machine this code could find it’s way into the finished product.
So come milestone day – usually a Friday – it is someone’s responsibility to bring together everyone’s work for that month, starting by simply making it all compile then testing the game so that it doesn’t contain any serious crash or gameplay bugs. Since this can be a long process, people have avoided updating their local copies for the past month because they don’t want to be stopped in their tracks, so this is made even more difficult as code will simply be out of step with each other. So to make it easier, we’ll start the build on Wednesday so it isn’t such a rush (but that means we do lose almost 3 solid days of development). Some people will try to ease this process by doing weekly build rather than monthly builds, but you still have the same problems, just spread a bit thinner.
This is fine for the programmers but what about the designers or the artists? They need constantly updated builds so they can test their new features or see how their new assets sit in the game. But because people don’t want to stop working to create a new build the artists will simply be given whatever build is on the nearest programmer’s machine, or if they need a specific persons changes, they have to pester them for another build so they can continue work. So the designers and artists work is also never tested on a ‘real’ build, which also means it might have to be manually brought together when a new build is produced.
QA are also at a disadvantage. What is the point of stress-testing a game when it’s not actually the complete product? Which means testing can only kick off at the end of the milestone month and that build will be out of date before they even start.
So what we end up with here is a process where the ‘real’ game is never really seen until it is really needed, and if there are serious problems in there they are simply not seen until it is too late. Since everyone is literally using a custom build of the game every day, no-one has a full picture of where the title is, how stable it is and most importantly how fun it is to play.
And this is where Continuous Integration comes into its own…