Our current Continuous Integration process works quite nicely and has greatly benefited our team, but these systems can always be extended and improved upon. The following post will simply cover some of the ideas I have for the future and how it might improve the whole process.
Visual Unit Testing
Visual Unit Testing is the process of creating small, scripted tests that run the game and allow you to test that certain states have been reached. For example, having a test which runs the game, scripts the input so one character shoots another and then tests if the enemy is dead or the score has been increased would be a very basic test. It could also be used to perform long soak tests with more intelligent input rather than the kind of tests that are used at the moment.
Allowing the process to take screen shots and post them to a web server with the test results would allow a huge number of systems to be tested in the environment they will actually be used in. Extending this to allow QA Technicians to create rafts of tests would also remove the need to continually test the more mundane aspects of games, such as does the menu flow behave as it should, do the leaderboards work as intended etc.
Obviously running this process would require a large amount of time, especially if run on multiple platforms. But it could easily be hooked into the nightly build process meaning hours of testing would be carried out when no-one was in the office.
Some Arcade titles originally supported simple scripting using Lua to perform automatic controller input, but this wasn’t taken far enough, and the BlitzTech engine now has its own state machine system which would most likely be used instead.
Automated Code Reviews
Teams at Blitz carry out code reviews at different times and to different levels, but one thing that is usually done is checking that the code conforms to a given Blitz coding standards (with people moving around teams as they do, a consistent standard is pretty much essential). But again, checking something that is based against a set of rules is something that an automated system is much better at that our human co-workers.
Adding a code metrics step to the Cruise Control build step, meaning the code is run through after every check-in could catch if the code is not standards-compliant below a certain threshold. If this was the case, the build would simply fail (and this step could be removed if we had a designer/artist build machine as discussed in a previous post) and the programmers would have to re-examine the code they posted.
While this could seem quite draconian, it would free up code reviews to concentrate on the bigger picture, rather than getting stuck in the rut of commenting on the use of ‘m_’ or class name pre-fixes.
Extending The Nightly Build
As I mentioned in Extending Cruise Control we collect information on how often the build is broken and the ratio of who breaks the build the most. We also generate a massive amount of information by using Subversion, such as check-in rates and lines changed per check-in etc.
The nightly build could easily be extended to incorporate all this information and, along with links to the Visual Unit Testing web service, could produce reports on the activity and stability of the game for that day. This would be a very useful tool for the programming managers, especially as some people already use this information, but it is usually done on an ad-hoc basis when the tools are manually run and the information manually collated.
Automatic CI Support
A large portion (at the moment pretty much all) of my current work is involved with extending the Arcade technology I developed and integrating it into the BlitzTech engine. It has the ability to automatically generates the required configuration files to set up a Cruise Control server, but the current plug-in system (like adding automated e-mail generation, stats collection etc.) is pretty much set up to do only what it needs to do, without much flexibility.
I fully intend to make all these plug-in’s come as standard, allowing developers to cherry pick the elements they want and hook them into their CI process with very little work. While people may start to ask why I do not use NANT since they are all hooked into CruiseControl.net, I can simply do more, and do it faster, with Ruby.
So I have a few ideas on where I want to go with our Continuous Integration process in the next few months (years?) but there is nothing like lofty goals to keep you going. Visual Unit Testing is obviously the biggest, and will require the largest amount of time to developer, but will generate the biggest payback once it’s being used to its fullest potential.
So what kind of things would you like to see in a CI process? Or is there something your process is doing that you think is pretty special?