As I mentioned in a previous post, we’ve ben trying to fix a whole slew of tests that flicker, and we’ve spent some time fixing a number of integration tests that leave side effects.
This is the first time I’ve encountered the testing pattern (not recommended!) where SpringJUnit4ClassRunner loads up a spring context, a bean is pulled out, and then mocks used to stub out services. It’s a really evil pattern.
For one thing, mocks are about interaction based testing, not really about stubbing. I typically use them to drive out class-based roles. However, in this case, they were used to take out a portion of the application.
Using the above JUnit Runner means that there is one spring context per test run, effectively, a global pool of objects. Of course, when you pull one out and start modifying the objects, it means you have plenty of side effects across other tests. Unfortunately the don’t always manifest themselves in obvious manners.
Our solution to fix this was to use reflection to look at all beans in the spring context, and fail the test if it found any that had any mocked instances. I’d still recommend you avoid the testing pattern altogether, but if you are forced down this route, you now have a solution to detect side-effects across tests.
