Test execution exceptions are not supplied to extensions

[sbrannen]

Status Quo

TestExtensionContext does not provide access to any exception thrown during execution of the test method or other extensions.

Deliverables

• Reintroduce a getTestException() method in the TestExtensionContext API.
  • Note: the prototype actually had such a getTestException() method.
• Ensure that the JUnit Jupiter TestEngine infrastructure tracks any exceptions encountered during test method or extension execution and supplies the exception to extensions via the TestExtensionContext.
  • Note: it is assumed that the engine already properly tracks the exceptions, but tests need to be introduced to verify the expected behavior.

Further Resources

See the afterEach() method in the SpringExtension for a concrete example of where the test exception is required.

[jlink]

We must be very explicit about which exceptions will get there. For example: Should an extension context see exceptions that occurred in other extension points and in other implementations of the same extension point? Or just from the same extension? Or just from the current test?

[sbrannen]

Good questions!

The testException should in fact be an exception thrown during invocation of the test method. If we don't have clarity in this regard it would then be difficult if not impossible for an extension to properly react to such an exception. For example, if a developer wants to implement his own version of @ExpectedException using the JUnit 5 extension model, such an extension would need to access exactly the exception thrown from the test method and not from some other extension.

I tend to think that an exception thrown from a @BeforeEach or @AfterEach method should also be considered as a testException.

Regarding exceptions thrown by extensions, we could consider introducing an extensionException for that dedicated purpose. Clients of TestExtensionContext would then have to decide which of the two exceptions they need to support.

I think we should also consider introducing similar support for ContainerExtensionContext.

Thoughts?

[jlink]

Idea: Let's introduce a new extension point à la

interface ExcpetionThrownInTest extends ExtensionPoint {
void exceptionThrown(TestExtensionContext context, Throwable throwable);
}

The callback would be invoked for all exceptions except AssertionError. Impementors could swallow or rethrow the exception.

Advantages:

• An extension can deal with exceptions in one place and not in all implemented extension points
• ExtensionContext need not be changed.

Von meinem iPad gesendet

Am 09.01.2016 um 14:27 schrieb Sam Brannen notifications@github.com:

Good questions!

The testException should should in fact be an exception thrown during invocation of the test method. If we don't have clarity in this regard it would then be difficult if not impossible for an extension to properly react to such an exception. For example, if a developer wants to implement his own version of @ExpectedException using the JUnit 5 extension model, such an extension would need to access exactly the exception thrown from the test method and not from some other extension.

I tend to think that an exception thrown from a @beforeeach or @AfterEach method should also be considered as a testException.

Regarding exceptions thrown by extensions, we could consider introducing an extensionException for that dedicated purpose. Clients of TestExtensionContext would then have to decide which of the two exceptions they need to support.

I think we should also consider introducing similar support for ContainerExtensionContext.

Thoughts?

—
Reply to this email directly or view it on GitHub.

[sbrannen]

That unfortunately won't meet all of our needs.

Extensions that wrap the test invocation with a before/after callback pair need access to the exception in the after callback.

For example, if a DatabaseTransactionExtension starts a transaction in the before callback, it will either need to commit or roll back the transaction in the after callback. In such cases, rollback behavior is typically influenced by the type of exception thrown.

In other words, after callbacks will often need to process a test exception alongside the standard after logic: separating after logic and exception handling logic into two different methods within an extension leads to cumbersome implementations at best.

[jlink]

Well, the exceptionThrown EP can store it as attribute for other EPs to pick up if need be.

Am 09.01.2016 um 15:18 schrieb Sam Brannen notifications@github.com:

That unfortunately won't meet all of our needs.

Extensions that wrap the test invocation with a before/after callback pair need access to the exception in the after callback.

For example, if a DatabaseTransactionExtension starts a transaction in the before callback, it will either need to commit or roll back the transaction in the after callback. In such cases, rollback behavior is typically influenced by the type of exception thrown.

In other words, after callbacks will often need to process a test exception alongside the standard after logic: separating after logic and exception handling logic into two different methods within an extension leads to cumbersome implementations at best.

—
Reply to this email directly or view it on GitHub.

[jlink]

This would play well together with my suggestion about Store in #82.

[sbrannen]

Well, the exceptionThrown EP can store it as attribute for other EPs to pick up if need be.

Sure, it could, but... I don't see any need to make extension developers jump through hoops just to access the exception.

Accessing the exception should be just as easy as accessing the test class, instance, or method.

[sbrannen]

This would play well together with my suggestion about Store in #82.

Let's look at that together next week!

[jlink]

Alright!

2016-01-09 19:08 GMT+01:00 Sam Brannen notifications@github.com:

This would play well together with my suggestion about Store in #82
#82.

Let's look at that together next week!

—
Reply to this email directly or view it on GitHub
#92 (comment)
.