Skip to content

Commit 7d81366

Browse files
olganaumenkoolganaumenko
authored
Added Spring design doc (#2661)
1 parent 8852277 commit 7d81366

File tree

1 file changed

+370
-0
lines changed

1 file changed

+370
-0
lines changed

‎docs/spring.md

Lines changed: 370 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,370 @@
1+
# Automated test generation for Spring-based code
2+
3+
Java developers actively use the Spring framework to implement the inversion of control and dependency injection.
4+
Testing Spring-based applications differs significantly from testing standard Java programs. Thus, we customized
5+
UnitTestBot to analyze Spring projects.
6+
7+
<!-- TOC -->
8+
* [General notes](#general-notes)
9+
* [Limitations](#limitations)
10+
* [Testability](#testability)
11+
* [Standard unit tests](#standard-unit-tests)
12+
* [Example](#example)
13+
* [Use cases](#use-cases)
14+
* [Spring-specific unit tests](#spring-specific-unit-tests)
15+
* [Example](#example-1)
16+
* [Use cases](#use-cases-1)
17+
* [Side effects](#side-effects)
18+
* [Mechanism](#mechanism)
19+
* [Integration tests](#integration-tests)
20+
* [Service layer](#service-layer)
21+
* [Side effects](#side-effects-1)
22+
* [Use cases](#use-cases-2)
23+
* [Controller layer](#controller-layer)
24+
* [Example](#example-2)
25+
* [Microservice layer](#microservice-layer)
26+
<!-- TOC -->
27+
28+
## General notes
29+
30+
UnitTestBot proposes three approaches to automated test generation:
31+
* [standard unit tests](#standard-unit-tests) that mock environmental interactions;
32+
* [Spring-specific unit tests](#spring-specific-unit-tests) that use information about the Spring application context to reduce the number of
33+
mocks;
34+
* and [integration tests](#integration-tests) that validate interactions between application components.
35+
36+
Hereinafter, by _components_ we mean Spring components.
37+
38+
For classes under test, one should select an appropriate type of test generation based on their knowledge
39+
about the Spring specifics of the current class. Recommendations on how to choose the test type are provided below.
40+
For developers who are new to Spring, there is a "default" generation type.
41+
42+
### Limitations
43+
44+
UnitTestBot Java with Spring support uses symbolic execution to generate unit tests, so typical problems
45+
related to this technique may appear: it may be not so efficient for multithreaded programs, functions with calls to
46+
external libraries, processing large collections, etc.
47+
48+
### Testability
49+
50+
Note that UnitTestBot may generate unit tests more efficiently if your code is written to be unit-testable: the
51+
functions are not too complex, each function implements one logical unit, static and global data are used
52+
only if required, etc. Difficulties with automated test generation may have "diagnostic" value: it
53+
may mean that you should refactor your code.
54+
55+
## Standard unit tests
56+
57+
The easiest way to test Spring applications is to generate unit tests for components: to
58+
mock the external calls found in the method under test and to test just this method's
59+
functionality. UnitTestBot Java uses the Mockito framework that allows to mark
60+
the to-be-mocked objects with the `@Mock` annotation and to use the `@InjectMock`
61+
annotation for the tested instance injecting all the mocked fields. See [Mockito](https://site.mockito.org/)
62+
documentation for details.
63+
64+
### Example
65+
66+
Consider generating unit tests for the `OrderService` class that autowires `OrderRepository`:
67+
68+
```java
69+
70+
@Service
71+
public class OrderService {
72+
73+
@Autowired
74+
private OrderRepository orderRepository ;
75+
76+
public List<Order> getOrders () {
77+
return orderRepository.findAll ();
78+
}
79+
}
80+
81+
public interface OrderRepository extends JpaRepository <Order, Long>
82+
```
83+
Then we mock the repository and inject the resulting mock into a service:
84+
85+
```java
86+
public final class OrderServiceTest {
87+
@InjectMocks
88+
private OrderService orderService
89+
90+
@Mock
91+
private OrderRepository orderRepositoryMock
92+
93+
@Test
94+
public void testGetOrders () {
95+
when(orderRepositoryMock .findAll()).thenReturn((List)null)
96+
97+
List actual = orderService .getOrders()
98+
assertNull(actual)
99+
}
100+
```
101+
102+
This test type does not process the Spring context of the original application. The components are tested in
103+
isolation.
104+
105+
It is convenient when the component has its own meaningful logic and may be useless when its main responsibility is to call other components.
106+
107+
Note that if you autowire several beans of one type or a collection into the class under test, the code of test
108+
class will be a bit different: for example, when a collection is autowired, it is marked with `@Spy` annotation due
109+
to Mockito specifics (not with `@Mock`).
110+
111+
### Use cases
112+
113+
When to generate standard unit tests:
114+
* _Service_ or _DAO_ layer of Spring application is tested.
115+
* Class having no Spring specific is tested.
116+
* You would like to test your code in isolation.
117+
* You would like to generate tests as fast as possible.
118+
* You would like to avoid starting application context and be sure the test generation process has no Spring-related side effects.
119+
* You would like to generate tests in one click and avoid creating specific profiles or configuration classes for
120+
testing purposes.
121+
122+
We suggest using this test generation type for the users that are not so experienced in Spring or would like to get
123+
test coverage for their projects without additional efforts.
124+
125+
## Spring-specific unit tests
126+
127+
This is a modification of standard unit tests generated for Spring projects that may allow us to get more
128+
meaningful tests.
129+
130+
### Example
131+
132+
Consider the following class under test
133+
134+
```java
135+
@Service
136+
public class GenderService {
137+
138+
@Autowired
139+
public Human human
140+
141+
public String getGender () {
142+
return human.getGender();
143+
}
144+
}
145+
```
146+
where `Human` is an interface that has just one implementation actually used in current project configuration.
147+
148+
```java
149+
public interface Human {
150+
String getGender();
151+
}
152+
153+
public class Man implements Human {
154+
public String getGender() {
155+
return “man”
156+
}
157+
}
158+
```
159+
160+
The standard unit test generation approach is to mock the _autowired_ objects. It means that the generated test will be
161+
correct but useless. However, there is just one implementation of the `Human` interface, so we may use it directly
162+
and generate a test like this:
163+
164+
```java
165+
@Test
166+
public void testGetGender_HumanGetGender() {
167+
GenderService genderService = new GenderService();
168+
genderService.human = new Man();
169+
String actual = genderService.getGender();
170+
assertEquals(“man”, actual);
171+
}
172+
```
173+
174+
Actually, dependencies in Spring applications are often injected via interfaces, and they often have just one actual
175+
implementation, so it can be used in the generated tests instead of an interface. If a class is injected itself, it
176+
will also be used in tests instead of a mock.
177+
178+
You need to select a configuration to guide the process of creating unit tests. We support all commonly used
179+
approaches to configure the application:
180+
* using an XML file,
181+
* Java annotation,
182+
* or automated configuration in Spring Boot.
183+
184+
Although it is possible to use the development configuration for testing purposes, we strictly recommend creating a separate one.
185+
186+
### Use cases
187+
188+
When to generate Spring-specific unit tests:
189+
* to reduce the amount of mocks in generated tests
190+
* and to use real object types instead of their interfaces, obtaining tests that simulate the method under test execution.
191+
192+
### Side effects
193+
194+
We do not recommend generating Spring-specific unit tests, when you would like to maximize line coverage.
195+
The goal of this approach is to cover the lines that are relevant for the current configuration and are to be used
196+
during the application run. The other lines are ignored.
197+
198+
When a concrete object is created instead of mocks, it is analyzed with symbolic execution. It means that the
199+
generation process may take longer and may exceed the requested timeout.
200+
201+
### Mechanism
202+
203+
A Spring application is created to simulate a user one. It uses configuration importing users one with an additional
204+
bean of a special _bean factory post processor_.
205+
206+
This _post processor_ is called when bean definitions have already been created, but actual bean initialization has
207+
not been started. It gets all accessible information about bean types from the definitions and destroys these
208+
definitions after that.
209+
210+
Further Spring context initialization is gracefully crashed as bean definitions do not exist anymore. Thus, this
211+
test generation type is still safe and will not have any Spring-related side effects.
212+
213+
Bean type information is used in symbolic execution to decide if we should mock the current object or instantiate it.
214+
215+
## Integration tests
216+
217+
The main difference of integration testing is that it tests the current component while taking interactions with
218+
other classes into account.
219+
220+
### _Service_ layer
221+
222+
Consider an `OrderService` class we have already seen. Actually, this class has just one
223+
responsibility: to return the result of a call to the repository. So, if we mock the repository, our unit test is
224+
actually useless. However, we can test this service in interaction with the repository: save some information to the
225+
database and verify if we have successfully read it in our method. Thus, the test method looks as follows.
226+
227+
```java
228+
229+
@Autowired
230+
private OrderService orderService
231+
232+
@Autowired
233+
private OrderRepository orderRepository
234+
235+
@Test
236+
public void testGetOrderById() throws Exception {
237+
Order order = new Order();
238+
Order order1 = orderRepository.save(order);
239+
long id = (Long) getFieldValue(order1, "com.rest.order.models.Order ", "id“);
240+
241+
Order actual = orderService.getOrderById(id);
242+
assertEquals (order1, actual);
243+
}
244+
```
245+
The key idea of integration testing is to initialize the context of a Spring application and to autowire a bean of
246+
the class under test, and the beans it depends on. The main difficulty is to mutate the initial _autowired_ state of the
247+
object under test to another state to obtain meaningful tests (e.g. save some data to related repositories).
248+
Here we use fuzzing methods instead of symbolic execution.
249+
250+
You should take into account that our integration tests do not use mocks at all. It also means that if the method
251+
under test contains calls to other microservices, you need to start the microservice unless you want to test your
252+
component under an assumption that the microservice is not responding.
253+
Writing tests manually, users can investigate the expected behavior of the external service for the current scenario,
254+
but automated test generation tools have no way to do it.
255+
256+
Note that XML configuration files are currently not supported in integration testing. However, you may create a Java
257+
configuration class importing your XML file as a resource. The list of supported test
258+
frameworks is reduced to JUnit 4 and JUnit 5; TestNG is not supported for integration tests.
259+
260+
To run integration tests properly, several annotations are generated for the class with tests (some of them may be
261+
missed: for example, we can avoid setting active profiles via the annotation if a default profile is used).
262+
263+
* `@SpringBootTest` for Spring Boot applications
264+
* `@RunWith(SpringRunner.class)`/`@ExtendWith(SpringExtension.class)` depending on the test framework
265+
* `@BootstrapWith(SpringBootTestContextBootstrapper.class)` for Spring Boot applications
266+
* `@ActiveProfiles(profiles = {profile_names})` to activate requested profiles
267+
* `@ContextConfiguration(classes = {configuration_classes})` to initialize a proper configuration
268+
* `@AutoConfugureTestDatabase`
269+
270+
Two additional annotations are:
271+
272+
* `@Transactional`: using this annotation is not a good idea for some developers because it can
273+
hide problems in the tested code. For example, it leads to getting data from the transaction cache instead of real
274+
communication with database.
275+
However, we need to use this annotation during the test generation process due to the
276+
efficiency reasons and the current fuzzing approach. Generating tests in transaction but not running them in
277+
transaction may sometimes lead to failing tests.
278+
In future, we are going to modify the test generation process and to use `EntityManager` and manual flushing to the
279+
database, so running tests in transaction will not have a mentioned disadvantage any more.
280+
281+
* `@DirtiesContext(classMode=BEFORE_EACH_TEST_METHOD)`: although running test method in transaction rollbacks most
282+
actions in the context, there are two reasons to use `DirtiesContext`. First, we are going to remove
283+
`@Transactional`. After that, the database `id` sequences are not rolled back with the transaction, while we would
284+
like to have a clean context state for each new test to avoid unobvious dependencies between them.
285+
286+
Currently, we do not have proper support for Spring security issues in UnitTestBot. We are going to improve it in
287+
future releases, but to get at least some results on the classes requiring authorization, we use `@WithMockUser` for
288+
applications with security issues.
289+
290+
#### Side effects
291+
292+
Actually, yes! Integration test generation requires Spring context initialization that may contain unexpected
293+
actions: HTTP requests, calls to other microservices, changing the computer parameters. So you need to
294+
validate the configuration carefully before trying to generate integration tests. We strictly recommend avoiding
295+
using _production_ and _development_ configuration classes for testing purposes, and creating separate ones.
296+
297+
#### Use cases
298+
299+
When to generate integration tests:
300+
* You have a properly prepared configuration class for testing
301+
* You would like to test your component in interaction with others
302+
* You would like to generate tests without mocks
303+
* You would like to test a controller
304+
* You consent that generation may be much longer than for unit tests
305+
306+
### _Controller_ layer
307+
308+
When you write tests for controllers manually, it is recommended to do it a bit differently. Of course, you may just
309+
mock the other classes and generate unit tests looking similarly to the tests we created for services, but they may
310+
not be representative. To solve this problem, we suggest a specific integration test generation approach for controllers.
311+
312+
#### Example
313+
314+
Consider testing the following controller method:
315+
316+
```java
317+
318+
@RestController
319+
@RequestMapping(value = "/api")
320+
public class OrderController {
321+
322+
@Autowired
323+
private OrderService orderService;
324+
325+
@GetMapping(path = "/orders")
326+
public ResponseEntity<List<Order>> getAllOrders() {
327+
return ResponseEntity.ok().body(orderService.getOrders());
328+
}
329+
}
330+
```
331+
UnitTestBot generates the following integration test for it:
332+
333+
```java
334+
@Test
335+
public void testGetAllOrders() throws Exception {
336+
Object[] objectArray = {};
337+
MockHttpServletRequestBuilder mockHttpServletRequestBuilder = get("/api/orders", objectArray);
338+
339+
ResultActions actual = mockMvc.perform(mockHttpServletRequestBuilder);
340+
341+
actual.andDo(print());
342+
actual.andExpect((status()).is(200));
343+
actual.andExpect((content()).string("[]"));
344+
}
345+
```
346+
347+
Note that generating specific tests for controllers is now in active development, so some parameter annotations and
348+
types have not been supported yet. For example, we have not supported the `@RequestParam` annotation yet. For now,
349+
specific integration tests for controllers are just an experimental feature.
350+
351+
### _Microservice_ layer
352+
353+
Actually, during integration test generation we create one specific test that can be considered as a test for the
354+
whole microservice. It is the `contextLoads` test, and it checks if a Spring application context has started normally.
355+
If this test fails, it means that your application is not properly configured, so the failure of other tests is not caused by the regression in the tested code.
356+
357+
Normally, this test is very simple:
358+
359+
```java
360+
/**
361+
* This sanity check test fails if the application context cannot start.
362+
*/
363+
@Test
364+
public void contextLoads() {
365+
}
366+
```
367+
368+
If there are context loading problems, the test contains a commented exception type, a message, and a
369+
track trace, so it is easier to investigate why context initialization has failed.
370+

0 commit comments

Comments
 (0)