Create annotation to group tests [SPR-3653]

[spring-projects-issues]

Magnus Heino opened SPR-3653 and commented

The only thing I miss comparing in JUnit comparing with TestNG is the ability to group tests using annotations.

org.springframework.test.ConditionalTestCase makes it possible for subclasses to override isDisabledInThisEnvironment(String testMethodName) and in that method, looking at the test method name, decide if the test should run or not.

Add a annotation like this:

package org.springframework.test;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Inherited;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target(value=ElementType.METHOD)
@Retention(value=RetentionPolicy.RUNTIME)
@Inherited
@Documented
public @interface Environment {
	
	String[] value();
}

Add this method to ConditionalTestCase

/**
 * Should this test run?
 * @param environment name of the test method environment
 * @return whether the test should execute in the provided envionment
 */
protected boolean isDisabledInEnvironment(String environment) {
	return false;
}

and change ConditionalTestCase.runBase to this:

public void runBare() throws Throwable {
	Method runMethod = getClass().getMethod(getName(), (Class[])null); 
	
	boolean disabledInEnvironment = false;
	
	if(runMethod.isAnnotationPresent(Environment.class)) {
		String[] environments = runMethod.getAnnotation(Environment.class).value();
		
		for (int i = 0; i < environments.length; i++) {
			if(isDisabledInEnvironment(environments[i])) {
				disabledInEnvironment = true;
			}
		}
	}
	
	// getName will return the name of the method being run
	if (disabledInEnvironment || isDisabledInThisEnvironment(getName())) {
		recordDisabled();
		logger.info("**** " + getClass().getName() + "." + getName() + " disabled in this environment: " +
				"Total disabled tests=" + getDisabledTestCount());
		return;
	}
	
	// Let JUnit handle execution
	super.runBare();
}

Then it's possible to annotate each test method with what environment it should run in. Applications can create own baseclasses of the spring Abstract* test classes and override isDisabledInEnvironment(String environment) . In there, a decision is made based on the environment that the method is annotated with.

Example:

public class AppTest extends ConditionalTestCase {

	private final Log log = LogFactory.getLog(this.getClass());
	
	@Environment({"database", "slow"})
	public void testDatabaseSlow() {
		log.debug("NO");
	}

	@Environment({"slow"})
	public void testSlow() {
		log.debug("YES");
	}
	
	@Environment({"database"})
	public void testDatabase() {
		log.debug("NO");
	}
	
	public void testNoAnnotation() {
		log.debug("YES");
	}
	
	@Environment({})
	public void testEmptyArray() {
		log.debug("YES");
	}
		
	@Override
	protected boolean isDisabledInEnvironment(String environment) {
		if(environment.equalsIgnoreCase("database")) {
			return true;
		}
		
		return false;		
	}	
}

In a real scenario the isDisabledInEnvironment is moved to a application specific testcasebaseclass that inherits ConditionalTestCase, and this example class inherits from that new testcasebaseclass instead.

/Magnus

---

Issue Links:

• Introduce strategy for determining if a profile value is enabled for a particular test environment [SPR-4862] #9538 Introduce strategy for determining if a profile value is enabled for a particular test environment
• Allow multiple values be specified in the runtime for tests filtering by @IfProfileValue [SPR-5903] #10572 Allow multiple values be specified in the runtime for tests filtering by @IfProfileValue

[spring-projects-issues]

Juergen Hoeller commented

Sam, can you please have a look at this for 2.5 RC1? What options are there for this style of grouping in JUnit4? Is there any low-hanging fruit for us to pick there?

Juergen

[spring-projects-issues]

Sam Brannen commented

Hi Magnus,

The only thing I miss comparing in JUnit comparing with TestNG is the
ability to group tests using annotations.

Spring already offers similar functionality since the 2.0 release in
the form of the @IfProfileValue annotation support.

As you pointed out, "ConditionalTestCase makes it possible for subclasses
to override isDisabledInThisEnvironment(String testMethodName) and in that
method, looking at the test method name, decide if the test should run or
not." This remains the recommended approach for JUnit 3.8 based tests which
are run against JDK 1.4 or previous. Note that ConditionalTestCase resides
in the JDK 1.4 source tree and therefore can not rely on annotations.
Thus we can not accept your proposal to make ConditionalTestCase
"annotation aware".

If you are developing against Java 5 or higher, however, you can achieve
your goal via annotations. @IfProfileValue is supported both in the JUnit
3.8 legacy integration testing support from Spring as well as in the new
Spring TestContext Framework (see the Spring 2.1 M4 reference manual
for thorough details).

Since Spring 2.0, AbstractAnnotationAwareTransactionalTests has supported
@IfProfileValue for JUnit 3.8 based tests. Here is an example of how
you could achieve something similar to using your proposed @Environment
annotation (assuming the environment variable 'test-group' is set to
'slow'):

public class AppTest extends AbstractAnnotationAwareTransactionalTests {

public AppTest(String name) {
    super(name);
    System.setProperty("test-group", "slow");
}

@Override
protected String getConfigPath() {
    return "AppTest-context.xml";
}

@IfProfileValue(name = "test-group", value = "database")
public void testDatabase() {
    System.out.println("NO");
}

@IfProfileValue(name = "test-group", value = "slow")
public void testSlow() {
    System.out.println("YES");
}

public void testNoAnnotation() {
    System.out.println("YES");
}

}

Please note that the "test-group" system property would generally be set
either as a command line argument or within a build script.

Also note that @IfProfileValue semantics are actually the opposite of your
proposed @Environment semantics. In other words, with @IfProfileValue
you specify environments in which a test class or method should run;
whereas, with your @Environment annotation, you would specify environments
in which a test method should not run.

One issue that @IfProfileValue does not currently address is the ability
to process multiple values with a single annotation declaration. For
example, for the following @Environment usage you proposed, there would
be no direct @IfProfileValue equivalent, since @IfProfileValue supports
only a single value (i.e., not an array).

@Environment({"database", "slow"})
public void testDatabaseSlow() {
    logger.debug("NO");
}

Please have a look at @IfProfileValue and let me know if it meets
your needs!

In the meantime, I will also look into adding additional support
to @IfProfileValue to allow multiple values.

On the positive side, the mere fact that you brought up this topic
helped me to discover a few small glitches with the implementation
of @IfProfileValue support in AbstractAnnotationAwareTransactionalTests.
I've fixed those in CVS HEAD, and you should see the changes in the
next snapshot here:

http://static.springframework.org/downloads/nightly/snapshot-download.php?project=SPR

So thanks for bringing up these issues! ;)

Regards,

Sam

// -------------------------------------------------------------------------

p.s.

The down side of extending AbstractAnnotationAwareTransactionalTests is
that it extends AbstractTransactionalDataSourceSpringContextTests which
in turn forces you to provide an application context containing a
DataSource and PlatformTransactionManager. For the above AppTest example,
I referenced AppTest-context.xml, which looks like the following:

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:p="http://www.springframework.org/schema/p"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">

<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
    <property name="driverClassName" value="org.hsqldb.jdbcDriver" />
    <property name="url" value="jdbc:hsqldb:mem:transactional_tests" />
    <property name="username" value="sa" />
    <property name="password" value="" />
</bean>

<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager"
    p:data-source-ref="dataSource" />

</beans>

However, if you use the Spring TestContext Framework, you do not necessarily
need an ApplicationContext, DataSource, or PlatformTransactionManager in
order to use @IfProfileValue. For example, here is the same AppTest
configured for use with Spring's JUnit 4 support:

@RunWith(SpringJUnit4ClassRunner.class)
@TestExecutionListeners( {})
public class AppTestJUnit4 {

public AppTestJUnit4() {
    System.setProperty("test-group", "slow");
}

@Test
@IfProfileValue(name = "test-group", value = "database")
public void testDatabase() {
    System.out.println("NO");
}

@Test
@IfProfileValue(name = "test-group", value = "slow")
public void testSlow() {
    System.out.println("YES");
}

@Test
public void testNoAnnotation() {
    System.out.println("YES");
}

}

[spring-projects-issues]

Sam Brannen commented

Hi Magnus,

As promised, I looked into adding additional support to @IfProfileValue
to allow multiple values, and I think we now have a combination that
suits your needs.

@IfProfileValue now accepts either a single 'value' or multiple 'values'.
The following is a JUnit 4 based example displaying this new functionality.
Note, however, that the same functionality applies to JUnit 3.8 support
via AbstractAnnotationAwareTransactionalTests and via the Spring
TestContext Framework. Of course, if you are using TestNG, you would
simply use TestNG's built-in support for groups.

As before, note that the "test-group" system property would generally be
set either as a command line argument or within a build script and thus
not directly within the test class.

@RunWith(SpringJUnit4ClassRunner.class)
@TestExecutionListeners( {})
public class AppTestJUnit4 {

public AppTestJUnit4() {
    System.setProperty("test-group", "slow");
}

@Test
@IfProfileValue(name = "test-group", value = "database")
public void testDatabase() {
    System.out.println("NO");
}

@Test
@IfProfileValue(name = "test-group", value = "slow")
public void testSlow() {
    System.out.println("YES");
}

@Test
@IfProfileValue(name = "test-group", values = { "database", "slow" })
public void testDatabaseSlow() {
    System.out.println("YES");
}

@Test
public void testNoAnnotation() {
    System.out.println("YES");
}

}

Give it a try in one of the upcoming Spring 2.5 RC1 snapshots, and let
me know if it meets your expectations.

Regards,

Sam

[spring-projects-issues]

Magnus Heino commented

The new testcontext framework looks great btw :) Finally a very good reason to switch to junit 4.

I have some questions. I have only looked through fisheye right now, so I haven't seen you latest changes.

1. Ok, I can configure IfProfileValue with multiple values now, but what does that mean. That one of many values must be true, or that all values must be true?

2. To make use of multipe values, I guess I need to implement a custom ProfileValueSource, cause I can only provide one value using the default SystemProfileValueSource I guess? ProfileValueUtils.findUniqueProfileValueSource looks for a unique ProfileValueSource in the context, where is the default set? How do I change the default? Just add it to the context?

3. I guess what I really want, is to be able to annotate the exception, tests that stand out from the others, and its probable those that I want to be able to disable. So like the comment in ProfileValueUtils.java says, "// XXX Optional: add support for @IfNotProfileValue." would be nice I guess.... :)

Thanks,

/Magnus

[spring-projects-issues]

Magnus Heino commented

Hm, giving it another thought, I guess 3 isn't needed after all. If no IfProfileValue is available, it will run. Then you can annotate the exception, and to make that test run, you need to provide a property to System, or whatever ProfileValueSource you have. Sounds reasonable.

/Magnus

[spring-projects-issues]

Magnus Heino commented

Commenting myself once again ;)

Ok, the answer to 1 is OR, which makes sense.

And the answer to the first part of 2 is that it's not possible to pass in many values in any implementation, since ProfileValueSource.get returns a String.

Not 100% as powerful as testng groups, since you then can pass in a list like "-groups windows,linux", but good enough for my purposes since I just want to tag integration tests to be able to run fast unit tests without slow db tests etc.

/Magnus

[spring-projects-issues]

Sam Brannen commented

Hi Magnus,

The new testcontext framework looks great btw :)

Thanks! :)

Finally a very good reason to switch to junit 4.

Agreed.

Now I'll attempt to address all of your follow-up comments in one fell swoop.

1. Ok, I can configure IfProfileValue with multiple values now, but what
   does that mean. That one of many values must be true, or that all values
   must be true?

As you later correctly deduced, multiples values are interpreted with "OR"
semantics.

2. To make use of multipe values, I guess I need to implement a custom
   ProfileValueSource, cause I can only provide one value using the default
   SystemProfileValueSource I guess?

And the answer to the first part of 2 is that it's not possible to pass
in many values in any implementation, since ProfileValueSource.get returns
a String.

That is correct: ProfileValueSource.get(String) takes a single argument and
returns a single value.

ProfileValueUtils.findUniqueProfileValueSource looks for a unique
ProfileValueSource in the context, where is the default set? How do I
change the default? Just add it to the context?

ProfileValueUtils.findUniqueProfileValueSource(ApplicationContext) is
currently only used by the findUniqueProfileValueSourceFromContext(ApplicationContext)
method in AbstractAnnotationAwareTransactionalTests; however, this method is
not called by default. Rather, one must explicitly call it within a subclass.

To make a long story short, in order to override the default ProfileValueSource
(which is an instance of SystemProfileValueSource), you would have to
call findUniqueProfileValueSourceFromContext(ApplicationContext) in a subclass
of AbstractAnnotationAwareTransactionalTests before isDisabledInThisEnvironment(Method)
is called. To accomplish this, you would either have to override runBare() or
isDisabledInThisEnvironment(Method), calling the 'super' implementation in
either case. This is obviously not an ideal mechanism for overriding the
default ProfileValueSource.

I assume it was never truly intended that one could provide a custom ProfileValueSource
implementation (easily); however, I will take a look into making this possible
(or at least easier).

3. I guess what I really want, is to be able to annotate the exception,
   tests that stand out from the others, and its probable those that I want
   to be able to disable.

Hm, giving it another thought, I guess 3 isn't needed after all. If no
IfProfileValue is available, it will run. Then you can annotate the exception,
and to make that test run, you need to provide a property to System, or
whatever ProfileValueSource you have. Sounds reasonable.

Yes, that sounds like a feasable usage scenario.

So like the comment in ProfileValueUtils.java says, "// XXX Optional: add
support for @IfNotProfileValue." would be nice I guess.... :)

That comment has been in the code base since 2.0. So, as long as no one comes up
with a strong need for it, it will remain optional (i.e., unimplemented). ;)

Not 100% as powerful as testng groups, since you then can pass in a list like
"-groups windows,linux", but good enough for my purposes since I just want to
tag integration tests to be able to run fast unit tests without slow db tests etc.

Yes, it is true that the scope of the @IfProfileValue support does not match
100% with TestNG's support for groups, but I don't think @IfProfileValue was
intended to be used in the exactly the same manner. In any case, I am glad I was
able to add the support for multiple values in order to meet your needs!

cheers,

Sam

[spring-projects-issues]

Sam Brannen commented

It is now possible to provide a custom ProfileValueSource implementation for your tests
via the new class-level @ProfileValueSourceConfiguration annotation which is used to
specify what type of ProfileValueSource to use when retrieving "profile values" configured
via @IfProfileValue. Here is an example configuration:

@ProfileValueSourceConfiguration(CustomProfileValueSource.class)
public class CustomProfileValueSourceTests() {
// class body...
}

If @ProfileValueSourceConfiguration is not declared for a test, SystemProfileValueSource
will be used by default.

In addition, the use of findUniqueProfileValueSourceFromContext(ApplicationContext) in
AbstractAnnotationAwareTransactionalTests is now deprecated in favor of the new
@ProfileValueSourceConfiguration configuration option.

Regards,

Sam