Programming, Unit Testing

Testing Objects with the Same Behaviour

One of the most difficult things when it comes to unit testing is cutting down on the number of duplicated tests. In the past I’ve often developed objects which have very different implementations but externally produce very similar results (for example when developing a vector compared to a fixed vector, or implementing different types of iterators), and one thing I don’t want to do is have a set of tests which are almost identical to another set of tests I wrote last week.

One method I’ve used is to write a common set of tests independent of a specific type and had the type defined in another file, along with other flags which are used to indicate which tests should or shouldn’t be run.

So for example, when developing a ring buffer style iterator, it was useful to know that the const and non-const versions produced the same results in a lot of different situations without having to have comparison tests or duplicating a large amount of test code.

The following examples are written using UnitTest++ but will work with a number of different unit testing Frameworks

// In the file RingBufferIterator_Tests.inl

// Tests are defined in a separate file assuming certain
// settings have been defined
TEST(Blah)
{
RINGBUFFER_ITERATOR_TYPE myItor;

//...  Doing tests with this type
}

Now we simply need to define the type of iterator and indicate if some tests should or shouldn’t be run

// In the file RingBufferIterator_TestTypes.cpp

SUITE(RingBufferItor)
{
// Define the type of iterator we want to test
#define RINGBUFFER_ITERATOR_TYPE ftl::itor::ringbuffer

// Define a couple of settings which the tests file uses to make
// sure some type specific tests are run or excluded

// This one simply indicates that the tests checking the ability
// to alter the content of the iterator will be run
#define RINGERBUFFER_ITERATOR_NONCONST_CONTENT

// Now we simply need to include the file used to
// implement all the unit tests for this kind of iterator
#include "RingBufferIterator_Tests.inl"
}

We do the same for the const version of the iterator we are testing

// In the file RingBufferConstIterator_TestTypes.cpp

SUITE(RingBufferConstItor)
{
// Define the type of iterator we want to test
#define RINGBUFFER_ITERATOR_TYPE ftl::itor::ringbuffer_const

// In this case we don't have any additional settings so we
// simply include the test files and let the tests run
#include "RingBufferIterator_Tests.inl"
}

You can take this as far as you want but there is obviously going to be a point where the number of flags you’re defining starts to make the tests hard to read or unmanageable. Limiting it to about two, where you can easily group the tests within these defines generally works quite well.

You don’t need to limit it to a single file. For example, some types might be similar enough to share 50% of the tests but not the rest. Splitting up the common tests is easy enough.

// In the file VectorContainer_TestTypes.cpp

SUITE(Vector)
{
// Define our type
#define VECTOR_TYPE ftl::vector

// Include our common tests
#include "Vector_CommonTests.inl"

// Include only the tests for this type
#include "Vector_DynamicTests.inl"
}

When tests are not written like this there is a a big gap in the ability to test if comparable types behave the same (fixed and non-fixed containers for example) especially when simple things like human error can get in the way of creating duplicate behaviour tests. By changing over to this style of testing we can automatically test the behaviour of similar types without creating additional work.

It does have it’s draw backs, the main one being that you can get multiple test failures (if a test is used by >1 type and they both fail at the same time) or worse you get one fail and you’re not sure which type caused the problem. An easy solution to this would be to allow the failure message to have a option component, allowing you to identify the type in the message, but this isn’t supported by any test framework that I know of.

Occasionally my compilers dependancy checker doesn’t cope very well, compiling only one of the type files rather than all of them if a test changes, but this has been remarkably rare and these draw backs don’t overshadow the ability to confirm that your types are functionally the same even if their implementation is vastly different.

8 thoughts on “Testing Objects with the Same Behaviour”

  1. Hey – apologies as this isn’t about the unit testing so much as something I noted in the first paragraph when you mentioned implementing your own container objects.

    I wanted to ask your opinion on whether the STL is appropriate in commercial game development. I’ve often read in game dev books that it is pointless to write your own containers thinking that they will be more efficient – STL is already there for you and has been rigorously tested. While there are some quriks, poor performance is often the cause of misuse of object semantics or the use of the wrong container type: http://archive.gdconf.com/gdc_2004/isensee_pete.ppt

    Then again, I’ve seen some books that jump in and implement their own versions of containers such as vectors without much of a word said about it… usually in a bid to cut down on bloat. It is a general consensus that homegrown versions of these containers are often slower than the original STL version.

    So, I was just wondering what your opinion of this is – I guess it goes without saying that hobby developers and students should just work with STL unless they’re trying to prove something specific.

  2. I take it either RTTI is not enabled for the tests or they generate unrecognisable names for the defines, otherwise you could stuff the types name into the test macro output message. (typeid()name())

    To me it sounds like it is not test driven code instead tests after the fact, but you do make the point of saying whilst under development.

    “So for example, when developing a ring buffer style iterator, it was useful to know that the const and non-const versions produced the same results in a lot of different situations without having to have comparison tests or duplicating a large amount of test code.”

    Unit testing is _normally_ an iterative process with one step (also a continuing one) being the refactoring of the test code, is what you are suggesting an early refactor or is this a result of the being templated code? I would propose that the duplicating code is what could be abstracted out, which in essence is what you are doing by using the defines and inline files, leaving simple unit test which call into this code.

    As you mention that there is a upper bound which makes this implementation hard to read and I have a feeling that including the files in this manner may make the tests harder to read, yet obliviously you think this it is fine within reason.

  3. @Liam

    You’re right in that RTTI is not enabled. We are generally used to living without RTTI in game development, especially in runtime code, so not having it enabled is the default setting. We could use it, in that we also usually don’t have exceptions enabled except in test code, but it’s something I’m simply used to not working with.

    This is still test driven development, though it certainly isn’t pure TDD as you mention (I don’t write a failing test, make it work, write another one etc.). Implementation and testing are written at the same time for one of the objects, sometimes test first but usually a feature of the object and then the related tests which feedback into the implementation.

    When another object is being developed tests can be introduced in the same manner, usually by removing them from a compiled out block of code, so the same process is followed, just with tests that happen to already be there.

    It’s also interesting in that one of the benefits of TDD is that you refactoring towards a solid, usable interface, which when developing against a fixed standard like the STL, is already done for you. But every process needs to be flexible to the people using it.

    Quite often the related classes (for example vector and fixed vector) are totally separate entities, with no shared based class (either functionally or as a pure interface). This is by design to avoid the overhead of inheritance and virtual functions so in this case the ability to extract common code is very limited, and even if it was, so much of the objects take into account how they are implemented that even simple functions like empty() and size() are unique to the type of object being developed.

    As you mention, it is using the unit testing process to enforce a common interface and common behaviour and as a benefit places all the work and overhead of enforcing that into a project that is only used during development.

  4. @James

    Hi James, don’t worry about it not being about testing, any discussion is worth having no matter where it’s done 🙂

    Unfortunately the question you ask is not an easy one. Ask 10 game developers and you’ll probably get 13 different answers so all I can do is give you my opinion and see where it fits with what you already know.

    The idea of the STL is great for game development, especially in companies with shared code bases or technology. Having 4 projects being developed using 4 different types of list just isn’t sustainable, and I’ve always stressed that your version of a link list is understood by you and a couple of others but std::list is understood by thousands (or millions) of developers out there.

    But you have a risk, especially in cross platform development, of using an external implementation. If you were to stick with the platform vendors version, you’re on a road to nowhere. In the linked article, it’s mentioned that in the VS implementation, clear() frees memory. That’s fine, we can deal with that, until you run on another platform which doesn’t free the memory. Such fundamental differences make it a nightmare to develop like this as you have to be aware of every platforms quirks when it really should be standard across them all.

    Some companies I know have taken an implementation (STLPort for example) and made it their own, added it to their repositories and tweaked and developed it when needed. This avoids the cross-platform issues but it still comes with a lot of bloat. There are some fantastic containers in the STL, but sometimes it’s best to not use them for all the reasons mentioned, but when the opportunity is there, it’s easy to ‘just use it this once’. But this is certainly the fastest approach that avoids a lot of the problems you can have with external libraries.

    Speed is often used as an excuse to not develop your own containers, and I would often agree – if you don’t have the time to fully optimise (or test) it will probably be slower (or broken) – but if you do, there is nothing stopping it being as fast or faster. We’ve benchmarked our internal STL implementation quite extensively, and it is slower in some cases but in the majority (and in areas we care about) it’s as fast or faster on all platforms, which is something we could specifically aim for.

    If I was doing something on my own at home, or if I was at a smaller company without shared tech or code, I would be very tempted to use an external STL implementation (a cross platform one – not a vendor implemented one) as the overhead of a custom implementation either wouldn’t be worth it or would be too costly.

    Like I said, this is all my own opinion, and (if Twitter is anything to go by) you’d get some very opposite comments on the topic, but it’s what fits your needs at the time.

    I’ve written a couple of posts in the past related to our implementation based on the STL (which I hope to continue writing about soon) so that might give you a bit more information too.

  5. Lee – thanks for the useful reply. I’d not heard of STLport, infact I hadn’t even really considered that cross-platform issues would arise with vanilla STL. I wonder, is the unique behaviour of STL on other platforms something that can be improved in future C++ standards? I’m going to check out EASTL, I was pleased to discover they had open-sourced it.

    I would also guess that ‘Not Invented Here’ syndrome applies in many situations 🙂

  6. @James There is one caveat of using STL – it have different goals in mind.
    For example we use (exclusively) containers that do NOT call constructors. For most cases you should use STL’s allocators to get what you need (proper align, budgeting, etc) and it’s not that straightforward, plus you can’t control how much memory is allocated etc.
    As a sidenote we’re currently using 6 arrays (not one stl’s vector) ;-). Hashtab instead of map (I know it will be in 0x). So that’s the question not only of portability/performance but also of really custom tasks

  7. @James

    I don’t believe any of the inconsistencies between platforms are really being resolved in the next release of the C++ standard – I just don’t think it’s at the top of anyones list nor would it be possible. Even if some of the points were revised to make the behaviour standard across all implementations, it still wouldn’t be perfect. There would always be areas of ‘interpretation’ and simple human error would introduce differences across vendor implementations.

    The thing I would like to see is clarification, simplification and standardisation of allocator models in the STL.

    The EASTL document is a fantastic read and I’d recommend anyone to read it, even if they were going with an implementation that was already out there.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s