Continuous Integration

Continuous Integration – Self Testing Builds

So far we have a process which guarantees that the game builds and that the assets can be generated.  We have processes for doing this automatically and for making sure people are as up to date as possible.  But there is still one question which none of this answers…

When I run the game, will it actually work?

Games are a mass of systems, all interacting together to produce something that is fun and interesting to play.  It’s QA’s job (on the whole but not exclusively) to make sure that when it plays it plays well and there are no glaring bugs, but if we are automating so much, we can also automate some of the testing too?

Unit Testing

A Self Testing Build in this case is really just a fancy term for Unit Testing, something a lot of people have heard of but just as many havn’t.

The basic premise behind this is that at the end of every single compile small pieces of code are run that test various systems in the game.  These are run as post build steps so any changes to the code are instantly tested against the tests that already exist.  If some code has changed that makes one of the tests fail then the whole build fails.  This means Cruise Control fails, the nightly build fails, but most importantly everyone knows it’s broken and they should avoid updating until it is fixed.

The following are a couple of very simple tests, that are run every time the code changes to make sure the vector’s behave the same and as expected.  This might look simple (one of the main reasons people don’t unit test is because it will ‘obviously work’).  But with vector code being platform centric, a change on one platform can effects all the others.  The tests are written using UnitTest++ which is the library we use to write all our tests.

Feed back to the developers tells them something has broken, the test results are displayed in Cruise Control and the person responsible instantly sets about fixing the build.

Game Based Unit Testing

It’s often difficult to think about games from the point of Unit Tests but it often results in code being written in a more encapsulated and less coupled way, which in itself results in code being easier to write and maintain.  But as an example, the following could easily be tested in a small suite of Unit Tests…

  • Does and AI character ‘see’ the player when they are in front of them
  • Is the player ignored if they are outside the enemies cone of vision?
  • What state do they enter when they hear a noise?

By testing the under-lying logic of the system we can be confident that when something breaks we will have a better idea of where the problem lies that having to look at every system at every level.

Running Tests On Different Platforms

Unit Tests are only useful if they are run often and on the platforms the code is destined to be used on.  This can cause problems when trying to run Unit Tests on development kits as the code usually needs to be loaded onto the machine before it can be run and the results fed back to the host.  While this might only take 10 seconds, doing it after every build would be a massive drain on a programmers time.  While this isn’t a problem on the PC platform, we generally expect the console tests to be run as part of the nightly build when it has all the time it needs to run and feed back the results.

But, even if it isn’t being run on every platform all of the time, we still have a system running on the PC and some testing is still better than no testing at all.

So this is how the Blitz Arcade system adds a level of testing to the Continuous Integration process.  It’s one of the hardest parts of the process because Unit Testing does take a while to get into, and to understand how code can be tested when at first it looks like it is a series of black box objects.

In the final part, I’ll take a look at what I would like the process to do in the future, and the kind of things that would be cool to have in a fully integrated Continuous Integration process.

2 thoughts on “Continuous Integration – Self Testing Builds”

  1. UnitTest++ was written by Noel Llopis and Charles Nicholson, so it has a pretty good set of authors behind it who understand the needs of game developers rather than anyone else.

    But most importantly it’s remarkably easy to get it up and running on the Xbox360 and PS3 (if I remember correctly it was an additional include file and two or three platform specific lines of code). Not done it yet, but I expect the same to be said of the Wii.

    I’ve been very tempted to used the Google testing libs, but tbh if it’s not broken…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s