Nullpointer Exception when using JUnit 5 @Nested tests

[slu-it]

Hello Spring REST Docs Team!

It seems that JUnit 5's @Nested tests don't work with auto-configured Spring REST Docs.

Executing the following simple test class will result in a NullPointerException:

package example;

import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.restdocs.AutoConfigureRestDocs;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import org.springframework.test.web.servlet.MockMvc;

import static org.hamcrest.CoreMatchers.equalTo;
import static org.springframework.restdocs.mockmvc.MockMvcRestDocumentation.document;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

@SpringBootTest
@AutoConfigureMockMvc
@AutoConfigureRestDocs
@ExtendWith(SpringExtension.class)
class FailingTest {

    @Autowired
    MockMvc mockMvc;

    @Nested
    class NestedTests {

        @Test
        void testAndDocumentation() throws Exception {
            mockMvc.perform(get("/hello"))
                    .andExpect(status().is2xxSuccessful())
                    .andExpect(jsonPath("message", equalTo("Hello World!")))
                    .andDo(document("hello-get-200"));
        }

    }

}

The resulting stacktrace:

java.lang.NullPointerException
	at org.springframework.restdocs.ManualRestDocumentation.beforeOperation(ManualRestDocumentation.java:89)
	at org.springframework.restdocs.mockmvc.MockMvcRestDocumentationConfigurer$ConfigurerApplyingRequestPostProcessor.postProcessRequest(MockMvcRestDocumentationConfigurer.java:100)
	at org.springframework.test.web.servlet.request.MockHttpServletRequestBuilder.postProcessRequest(MockHttpServletRequestBuilder.java:755)
	at org.springframework.test.web.servlet.MockMvc.perform(MockMvc.java:154)
	at example.FailingTest$NestedTests.testAndDocumentation(FailingTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:436)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:170)
	at org.junit.jupiter.engine.execution.ThrowableCollector.execute(ThrowableCollector.java:40)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:166)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:113)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:58)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.lambda$executeRecursively$3(HierarchicalTestExecutor.java:112)
	at org.junit.platform.engine.support.hierarchical.SingleTestExecutor.executeSafely(SingleTestExecutor.java:66)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.executeRecursively(HierarchicalTestExecutor.java:108)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.execute(HierarchicalTestExecutor.java:79)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.lambda$executeRecursively$2(HierarchicalTestExecutor.java:120)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.lambda$executeRecursively$3(HierarchicalTestExecutor.java:120)
	at org.junit.platform.engine.support.hierarchical.SingleTestExecutor.executeSafely(SingleTestExecutor.java:66)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.executeRecursively(HierarchicalTestExecutor.java:108)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.execute(HierarchicalTestExecutor.java:79)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.lambda$executeRecursively$2(HierarchicalTestExecutor.java:120)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.lambda$executeRecursively$3(HierarchicalTestExecutor.java:120)
	at org.junit.platform.engine.support.hierarchical.SingleTestExecutor.executeSafely(SingleTestExecutor.java:66)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.executeRecursively(HierarchicalTestExecutor.java:108)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.execute(HierarchicalTestExecutor.java:79)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.lambda$executeRecursively$2(HierarchicalTestExecutor.java:120)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.lambda$executeRecursively$3(HierarchicalTestExecutor.java:120)
	at org.junit.platform.engine.support.hierarchical.SingleTestExecutor.executeSafely(SingleTestExecutor.java:66)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.executeRecursively(HierarchicalTestExecutor.java:108)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor$NodeExecutor.execute(HierarchicalTestExecutor.java:79)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:55)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:43)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:170)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:154)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:90)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:74)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

Tested with Spring Boot 2.0.1.BUILD-SNAPSHOT, Spring REST Docs 2.0.0.RELEASE and JUnit 5.1.0 on 2018-03-14 at around 11:00 PM.

I created a simple example application (thx start.spring.io ;) ) to reproduce the issue:
https://github.com/slu-it/bug-spring-restdocs-nested-tests

Note: This might be related to the following issue I opened in Spring Boot:
spring-projects/spring-boot#12470

[wilkinsona]

Thanks for the sample. The NullPointerException is occurring because your inner test class isn't a Spring Boot test. I see you've corrected that in AlsoFailingTest that fails for a different reason. It's failing due to what is either a bug in or a limitation of the testing support provided by Spring Framework. I've opened https://jira.spring.io/browse/SPR-16595.

Note that these problems are specific to @AutoConfigureMockMvc and @AutoConfigureRestDocs. If you configure MockMvc and REST Docs without relying on Boot's test auto-configuration, this should work.

[lzhoucs]

Just want to check in on the status of this issue. It seems the limitation mentioned above with junit 5 @Nested class is fixed externally in spring framework 5.1.0, according to spring-projects/spring-framework#21136 (comment), however I am still running into the same NullPointerException in my @Nested class when I use @AutoConfigureMockMvc and @AutoConfigureRestDocs.

We have a lot of tests in Nested class, and they work fine when restdocs logic is removed. So it seems the problem is specific to restdocs. Converting those tests into top level class ones is an option, but not ideal if there's other way around.

Manually configure mockMvc with rest docs might not be an easy option, as there's a large gap to be filled, as mentioned #41 (comment). We do have spring security filters that is not included by default when going manual.

[wilkinsona]

@lzhoucs AFAIK, this should work. If you believe that's not the case, please take the time to provide a minimal sample as @slu-it did.

[lzhoucs]

Thanks @wilkinsona for the response. I thought it might still be a known issue so I just tried to confirm. Knowing it isn't, I just created a quick demo branch: https://github.com/lzhoucs/spring-restdocs/tree/junit5-nested-issue-demo where I modified the SampleJUnit5ApplicationTests as follows:

@SpringBootTest
@AutoConfigureMockMvc
@AutoConfigureRestDocs
public class SampleJUnit5ApplicationTests {

	@Autowired
	private MockMvc mockMvc;

	@Test
	public void sample() throws Exception {
		mockMvc.perform(get("/hello"))
				.andExpect(status().isOk())
				.andDo(document("sample"));
	}

	@Nested
	class NestedTests {
		@Test
		public void sampleNested() throws Exception {
			mockMvc.perform(get("/hello"))
				.andExpect(status().isOk())
				.andDo(document("sample"));
		}
	}
}

sampleNested test is failing right now.

[wilkinsona]

The configuration you've tried doesn't match the recommendations above.

The NullPointerException is occurring because your inner test class isn't a Spring Boot test

With the configuration that you've shared, NestedTests creates a new application context so the bean that should set REST Docs up correctly isn't found.

You can avoid the problem by configuring your nested class like this:

@Nested
@SpringBootTest
@AutoConfigureMockMvc
@AutoConfigureRestDocs
class NestedTests {

	@Autowired
	MockMvc mockMvc;

	@Test
	public void sampleNested() throws Exception {
		this.mockMvc.perform(get("/"))
			.andExpect(status().isOk())
			.andDo(document("sample"));
	}

}

You need to inject MockMvc again as, despite the apparently identical configuration, NestedTests still creates a second application context and you need the MockMvc instance from that context not the context of SampleJUnit5ApplicationTests.

This doesn't feel right to me, but it's out of the control of Spring REST Docs. @sbrannen how are nested tests expected to work in Framework 5.1? What configuration is required to have a nested test reuse the context, test execution listeners, etc from its outer class?

[sbrannen]

This doesn't feel right to me, but it's out of the control of Spring REST Docs. @sbrannen how are nested tests expected to work in Framework 5.1? What configuration is required to have a nested test reuse the context, test execution listeners, etc from its outer class?

The behavior is 5.1 is the same as in 5.0.

The current goal is to address this in 5.2: spring-projects/spring-framework#19930