Spies are not reset after test execution when using @SpyBean

[feliksik]

I experience the same problem as in #7374: The @SpyBean on the class is not reset between test cases.

I create a small (minimal?) version to reproduce, as suggested in the mentioned issue.

See the git repo on https://github.com/feliksik/bug-reports/tree/master/spybean-spring.

For convenience, the test inline:

@DataJpaTest
@ContextConfiguration(classes = SpyBeanJpa.MyConfig.class)
@SpyBean(classes = EntityJpaRepo.class)
class SpyBeanJpa {

    @Configuration("TestService")
    @EnableJpaRepositories({"nl.feliksik.bugs.spybean.jpa"})
    @EntityScan({"nl.feliksik.bugs.spybean.jpa"})
    @ComponentScan({
            "nl.feliksik.bugs.spybean",
    })
    public static class MyConfig {
    }

    @Autowired
    private MyService systemUnderTest;

    @Autowired
    private EntityJpaRepo jpaRepo;

    @BeforeEach
    void setUp() {
        // workaround for https://github.com/spring-projects/spring-boot/issues/33830
        //Mockito.reset(jpaRepo);
    }

    @Test
    void testA_break() {
        RuntimeException runtimeException = new RuntimeException("We simulate a failure");

        // Note that if you use the syntax that starts with when(), the real method
        // is called with null param. This is an unrelated issue I don't understand.
        doThrow(runtimeException).when(jpaRepo).saveAndFlush(any());

        try {
            systemUnderTest.upsert("someId");

            // should not reach this
            assertThat(true).isFalse();
        } catch (RuntimeException e) {
            // ok
            assertThat(e).isEqualTo(runtimeException);
        }
    }


    @Test
    void testB_ok() {
        // should be ok -- but it FAILS!
        systemUnderTest.upsert("someId");
    }
}

The first test mocks an exception (the test expects it, and succeeds), the second test suffers from this: it fails, as it does not expect an exception. (If I run them separately or change the alphabetic order of cases, all is fine).

The workaround I discovered is the reset() in the setup method.

Note that I could not reproduce this without the Jpa repo; but I want to simulate a database failure.

1. Possibly there is some of the Jpa/Mocking magic that conflicts here?
2. Is this a bug?
3. If someone can point out a better way to test for database failures, that would be appreciated.

[wilkinsona]

Thanks for the report and sample. I've reproduced the problem. The spy isn't being reset due to #7271 and the problem that we anticipated to some extent in this comment:

Changing from getBean(name) to getSingleton(name) (as proposed in #7271) avoids the problem as we end up working with the FactoryBean rather than the bean that it will produce. That generally feels ok to me, but there is an edge case where it'll break: if someone has a FactoryBean in some test configuration that produces a mock, the mock will no longer be reset.

This is what's happening in your sample (although the FactoryBean isn't declared in test configuration) and we end up checking if a JpaRepositoryFactoryBean instance is a mock (it isn't) rather than checking if the repository that it produced is a mock (it would have been).

I'm not sure how we can fix this without regressing 7271 so I'd like to discuss it with the rest of the team. In the meantime, your workaround is a good one. Alternatively, you may want to perform this sort of failure testing in a pure unit test that doesn't involve the application context.

[feliksik]

Thank you for your comment, I actually found a workaround so I can live on :-)

Aside this bug, w.r.t. the testing strategy: I would indeed like to do unit testing instead of loading the entire application context, but I don't believe it's nice or wise to mock all methods of a JPA repo for the unit test. Does the spring/JPA philosophy like mocking so much that it would actually always mock this in the tests?

I know there is some controversy about whether to use mocks always or not; but am I correct that there is no way to obtain an in-memory implementation of a JPA repo, so you can unittest with that dependency, and not having to mock the methods in the test? (i.e.. just observe the behavior from your unit under test)?

[philwebb]

We're going to try calling getBean(...) first and if that fails we'll call getSingleton(...).