Class-level @IfProfileValue overrides method-level @IfProfileValue settings [SPR-5902]

[spring-projects-issues]

Konstantin Boudnik opened SPR-5902 and commented

It seems that current implementation of @IfProfileValue is deficient for the following reasons:

• it is possible to create a mutual-lock situation when none of a class' test methods would be executed no matter what values are set in the runtime
• current semantic contradicts an existing and documented one of @IfProfileValue' values, e.g. OR semantic

The code below clearly demonstrate a situation when no matter which test-group's value is set ('all', 'fast', or 'quick') none of the tests will be executed, because the class belongs to a group 'all', but its test methods are from different test groups.

It seems to be a problem in the SpringJUnit4ClassRunner semantic's implementation that a class level @IfProfileValue actually overwrites any method's level @IfProfileValue. Should it have same semantic as as @IfProfileValue's 'values', i.e. with OR semantic, the problem won't exist (see the documentation at http://static.springsource.org/spring/docs/2.5.x/api/org/springframework/test/annotation/IfProfileValue.html)

import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.annotation.IfProfileValue;
import org.springframework.test.context.TestExecutionListeners;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

@RunWith(SpringJUnit4ClassRunner.class)
@TestExecutionListeners({})
@IfProfileValue(name = "test-group", value = "all")

public class MyTests {
  @BeforeClass
  public static void setUp () {
    System.out.println("test-group is set to: " + System.getProperty("test-group"));
  }
  @Test
  @IfProfileValue(name = "test-group", values = {"fast", "quick"})
  public void testOne() {
    System.out.println("testOne(): Quick stuff");
  }

  @Test
  @IfProfileValue(name = "test-group", value = "fast")
  public void testTwo() {
    System.out.println("testTwo(): Fast stuff");
  }
}

---

Affects: 2.5.6

Issue Links:

• Improve documentation for @IfProfileValue precedence [SPR-11902] #16521 Improve documentation for @IfProfileValue precedence
• ProfileValueUtils#isTestEnabledInThisEnvironment should use current test class to determine IfProfileValue annotation [SPR-4526] #9203 ProfileValueUtils#isTestEnabledInThisEnvironment should use current test class to determine IfProfileValue annotation
• Annotating an integration test with @IfProfileValue at class level still loads the applications contexts even when the profile is not activated [SPR-5043] #9718 Annotating an integration test with @IfProfileValue at class level still loads the applications contexts even when the profile is not activated
• ProfileValueUtils should enforce class-level usage of @IfProfileValue to override method-level usage [SPR-5914] #10583 ProfileValueUtils should enforce class-level usage of @IfProfileValue to override method-level usage
• Allow multiple values be specified in the runtime for tests filtering by @IfProfileValue [SPR-5903] #10572 Allow multiple values be specified in the runtime for tests filtering by @IfProfileValue

Referenced from: commits 2b869d5

[spring-projects-issues]

Konstantin Boudnik commented

Yet another thought on the semantics of @IfProfileValue:

• let's assume I have a test class with say 10 test methods in it
• two of the methods are assigned to test-group with name 'fast'
• the class itself nor other methods (except the two above) have any @IfProfileValue settings

Now, it'd be logical to expect that all methods of such a class should be executed when -Dtest-groups is set at all. However, current implementation only allows non-marked methods to be executed when none of -Dtest-groups is set. Basically it means that a default test-group, e.g. 'run-always' has to be explicitly set for all test methods.

Would it be possible to execute all tests with whatever value(s) of @IfProfileValue in case when no value for this annotation is set in the runtime?

[spring-projects-issues]

Sam Brannen commented

Hi Konstantin,

This is actually not a bug; rather it is by design.

I encourage you to read through the issues I've linked to this issue for further discussions on this functionality.

To address the questions you've raised:

• it is possible to create a mutual-lock situation when none of a class' test methods would be executed no matter what values are set in the runtime
  • This is correct. The use of @IfProfileValue at the class-level overrides method-level use of @IfProfileValue for any methods within that class.
• current semantic contradicts an existing and documented one of @IfProfileValue' values, e.g. OR semantic
  • There is no contradiction here, since (as mentioned above) @IfProfileValue at the class level overrides method-level usage. The following, however, would achieve what you were trying to do:

@RunWith(SpringJUnit4ClassRunner.class)
@TestExecutionListeners( {})
@IfProfileValue(name = "test-group", values = { "fast", "quick" })
public class MyTests {

	@BeforeClass
	public static void setUp() {
		System.out.println("test-group is set to: " + System.getProperty("test-group"));
	}

	@Test
	@IfProfileValue(name = "test-group", values = { "fast", "quick" })
	public void testOne() {
		System.out.println("testOne(): Quick stuff");
	}

	@Test
	@IfProfileValue(name = "test-group", value = "fast")
	public void testTwo() {
		System.out.println("testTwo(): Fast stuff");
	}
}

Run the above test once with -Dtest-group=fast and once with -Dtest-group=quick, and you will see that OR semantics are indeed supported for combinations of class-level and method-level usage of @IfProfileValue.

Regards,

Sam

[spring-projects-issues]

Konstantin Boudnik commented

Hi Sam.

Thanks for the quick resolution and comments. I agree that the execution of the class' methods will be accomplished in the way you've proposed. And if it has been done intentionally then I have no problem with it, although such semantic adds unwelcome maintenance burden to a developer or a test engineer. Such extra maintenance effort is caused by the necessity to keep the class' level list up to date whenever a test method is assigned to a different test-group, which is, however, minor pain).

However, your resolution leaves my second comment kinda unanswered: would it make sense to run tests when no @IfProfileValue is set at all. I can see that this is partially answered by #9718, but I believe my question is wider: should an empty (or null) setting for a test-group be considered as a valid wildcard, including all tests at all?

Please share your thoughts wrt this matter.

[spring-projects-issues]

Konstantin Boudnik commented

I have done tiny experiment with TestNG and found out that TestNG doesn't support this notion of 'empty wildcard' for all tests. So, I guess I'm willing to take my question away unless you guys would be willing to do add such a nice - IMHO - feature to the bucket!

[spring-projects-issues]

Sam Brannen commented

Hi Konstantin,

although such semantic adds unwelcome maintenance burden to a developer or a test engineer. Such extra maintenance effort is caused by the necessity to keep the class' level list up to date whenever a test method is assigned to a different test-group, which is, however, minor pain).

I would argue that using @IfProfileValue at the class-level with a list of values is atypical usage, and in fact I would recommend against it simply because of the maintenance overhead such usage introduces.

Try my above example again but remove @IfProfileValue from the class level. You will see that the tests run exactly the same. In addition, any non-annotated methods within the test class will also be run -- which is likely what you want. Note that if you keep @IfProfileValue at the class level, non-annotated methods in the test class will never run unless the conditions for @IfProfileValue are met at the class level.

The reason that I included a list of values in @IfProfileValue at the class level in my above example was merely to demonstrate that OR semantics work between class-level and method-level usage. Thus I was not trying to imply that you should use a list of values at the class level.

However, your resolution leaves my second comment kinda unanswered: would it make sense to run tests when no @IfProfileValue is set at all. I can see that this is partially answered by #9718, but I believe my question is wider: should an empty (or null) setting for a test-group be considered as a valid wildcard, including all tests at all?

No, I don't believe that an empty or null value should be considered a valid wild card for the currently defined semantics. We are, however, considering adding support for other strategies. See #9538 for details.

Regards,

Sam

[spring-projects-issues]

Konstantin Boudnik commented

Hi Sam.

I would argue that using @IfProfileValue at the class-level with a list of values is atypical usage, and in fact I would recommend against it simply because of the maintenance overhead such usage introduces.

I apologize if it seemed that I was putting words in your mouth - you weren't suggesting to use class level @IfProfileValue at all. It was my original question and I have asked it because of my whishful thinking that I can group the test-groups by using a mix of class and methods level @IfProfileValue settings. Unfortunately, it isn't a case.

Try my above example again but remove @IfProfileValue from the class level. You will see that the tests run exactly the same. In addition, any non-annotated methods within the test class will also be run - which is likely what you want.

Oh, they do - there wasn't a problem with that. If a class level @IfProfileValue is absent then everything works pretty much as expected. Although, I am not sure if including unassigned methods into specific group is what I've expected. I'd rather be happy to see it the other way around:

• when a test-groups value is set in the runtime then only matched tests are executed
• when no value is specified then all available tests are executed

The reason this looks more desirable to me would be better illustrated by an example. Let's assume I have a 100 tests and a half of them are assigned to the integration suite and another have is unassigned. When I run my integration job as

% and -Dtest-groups=integration tests

I will end with all my tests being executed as per current implementation.

However, when no test-groups is set only unassigned tests are ran. I feel like it is an inversion of my expectations, but they are perhaps only mine expectations and no one's else :-)

Also, thanks for pointing me to #9538 - I think if I'd be able to create my own test matching logic by using this new feature it'd the trick.

With regards,
Konstantin