You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* [rd-refactoring]
Multiple refactorings and enhacements for process starting.
Removed UtSettings.logConcreteExecutionErrors and UtSettings.engineProcessLogLevel switches.
Logs for engine and concrete executor processes are always enabled.
Switched log configuration to utbot-intellij/log4j2.xml.
Use UtSettings#getIdeaProcessLogConfigFile from ~/.utbot/settings.properties to configure logs for engine and concrete executor processes without rebuilding plugin.
* [rd-refactoring]
Enabling suspend and port choosing by utsettings
Run configurations for debugging
Instant process death detection
Orphan processes termination system
Some text loading optimization
ChildProcess -> InstrumentedProcess
Rd-based UtSettings for InstrumentedProcess
* [rd-refactoring]
Rd version bump, model regeneration typos and small adjustments
* [design-docs]
Rewriting design docs for inter process debugging and rd.
Completely new design doc go inter process logging
* [rd-factoring]
review fix
* [rd-factoring]
review fix
* [rd-factoring]
debug run configurations fix
Copy file name to clipboardExpand all lines: docs/RD for UnitTestBot.md
+85-34Lines changed: 85 additions & 34 deletions
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,31 @@
1
-
# RD
2
-
New child process communication involves 3 different things:
1
+
# Multi-process architecture
2
+
3
+
## Table of content
4
+
-[Overview](#overview)
5
+
-[Lifetimes](#lifetimes)
6
+
- [Lifetime](#lifetime)
7
+
- [LifetimeDefinition](#lifetimedefinition)
8
+
-[Rd](#rd)
9
+
-[Rdgen](#rdgen)
10
+
-[Model DSL](#model-dsl)
11
+
-[Gradle](#gradle)
12
+
-[UtBot project](#utbot-project)
13
+
-[IDEA process](#idea-process)
14
+
-[Engine process](#engine-process)
15
+
-[Instrumented process](#instrumented-process)
16
+
-[Commons](#useful)
17
+
18
+
## Overview
19
+
UtBot consists of 3 different processes:
20
+
1.`IDEA process` - the one where plugin part executes. Also can be called `plugin process`, `IDE process`.
21
+
2.`Engine process` - process where unit test generation engine executes.
22
+
3.`InstrumentedProces` - process where concrete execution takes place.
23
+
24
+
These processes are built on top of [JetBrains.RD](https://github.com/JetBrains/rd). It is crucial to understand
25
+
this library, so it's better describing it first(as there are no documentation about it in repo;)).
26
+
27
+
RD is mostly about 3 components:
28
+
3
29
1. Lifetimes
4
30
2. Rd entities
5
31
3. Rdgen
@@ -21,7 +47,7 @@ And so Lifetime was introduced.
21
47
### Lifetime:
22
48
```Lifetime``` is a class, where you can register callbacks and which can be terminated once, thus executing all registered callbacks.
23
49
24
-
```Lifetime``` is an abstract class, it's inheritor - ```LifetimeDefinition```. The only difference - only ```LifetimeDefinition``` can be terminated. Though all ```Lifetime``` are instances of ```LifetimeDefinition```, there are some conventions:
50
+
```Lifetime``` is an abstract class, it's inheritor - ```LifetimeDefinition```. The only difference - ```LifetimeDefinition``` can be terminated. Though all ```Lifetime``` are instances of ```LifetimeDefinition```, there are some conventions:
25
51
1. Do not cast ```Lifetime``` to ```LifetimeDefinion``` unless you are the one who created ```LifetimeDefinition```.
26
52
2. If you introduce somewhere ```LifetimeDefinition``` - either attach it to another ```Lifetime``` or provide code that terminates it.
27
53
@@ -100,42 +126,67 @@ DSL:
100
126
101
127
## UtBot project
102
128
103
-
There is another gradle project ```utbot-rd``` which contains model sources in ```rdgenModels``` sources. Look for ```org.utbot.rd.models.ProtocolRoot```.
129
+
There is another gradle project ```utbot-rd``` which contains model sources in ```rdgenModels```.
130
+
Look at [```utbot-rd/src/main/rdgen/org/utbot/rd/models```](../utbot-rd/src/main/rdgen/org/utbot/rd/models)
131
+
132
+
### IDEA process
133
+
Uses bundled JetBrains JDK. Code in `utbot-intellij`___must___ be compatible will all JDKs and plugin SDKs, which are used by our officially supported IntellijIDEA versions.
134
+
See [`utbot-intellij/build.gradle.kts`](../utbot-intellij/build.gradle.kts), parts `sinceBuild` and `untilBuild`.
135
+
136
+
Starts `Engine process`. Maintains `UtSettings` instance in memory and updates it from IDEA.
137
+
Other processes ask this process for settings via RD RPC.
138
+
139
+
### Engine process
140
+
141
+
`TestCaseGenerator` and `UtBotSymbolicEngine` run here. Process classpath contains all plugin jars(more precisely - it uses plugin classpath).
142
+
143
+
___Must___ run on JDK, which uses project we analyze. Otherwise there will be numerous problems with code analysis, soot, reflection and
144
+
devirgention of generated code Java API.
145
+
146
+
Currently, it is prohibited to run more than 1 generation process simultaneously(something with native libs).
147
+
However, logging for processes relies on that fact, so they can exclusively write to log file.
148
+
149
+
IDEA starting point - class [`EngineProcess`](../utbot-intellij/src/main/kotlin/org/utbot/intellij/plugin/process/EngineProcess.kt).
150
+
Process start file - [`EngineProcessMain`](../utbot-framework/src/main/kotlin/org/utbot/framework/process/EngineProcessMain.kt).
151
+
Starts `Instrumented process`.
152
+
153
+
### Instrumented process
154
+
155
+
Start points at `Engine process`: classes [`InstrumentedProcess`](../utbot-instrumentation/src/main/kotlin/org/utbot/instrumentation/rd/InstrumentedProcess.kt) and [`ConcreteExecutor`](../utbot-instrumentation/src/main/kotlin/org/utbot/instrumentation/ConcreteExecutor.kt).
156
+
First one is state encapsulation, second is used to implement request logic for concrete execution.
157
+
158
+
Runs on the same JDK as `Engine process` to erase deviation from `Engine process`.
159
+
Sometimes might unexpectedly die due concrete execution.
160
+
161
+
162
+
### Useful
104
163
105
-
Usefull:
106
164
1. if you need to use rd somewhere - add following dependencies:
2. There are some usefull classes to work with processes & rd:
170
+
2. There are some useful classes in `utbot-rd` to work with processes & rd:
113
171
- ```LifetimedProcess``` - binds ```Lifetime``` to process. If process dies - lifetime terminates and vice versa. You can terminate lifetime manually - this will destroy process.
114
-
- ```ProcessWithRdServer``` - also starts Rd server and waits for connection.
115
-
- ```UtInstrumentationProcess``` - encapsulates logic for preparing child process for executing arbitary commands. Exposes ```protocolModel``` for communicating with child process.
116
-
- ```ConcreteExecutor``` is convenient wrapper for executing commands and managing resources.
117
-
3. How child communication works:
118
-
- Choosing free port
119
-
- Creating child process, passing port as argument
120
-
- Both processes create protocols and bind model
121
-
- Child process setups all callbacks
122
-
- Parent process cannot send messages before child creates protocol, otherwise messages will be lost. So child process needs to signal that he is ready.
123
-
- Child proces creates special file in temp dir, that is observed by parent process.
124
-
- When parent process spots file - he deletes it, and then sends special message for preparing child proccess instrumentation
125
-
- Only then process is ready for executing commands
126
-
4. How to write custom commands for child process
127
-
- Add new ```call``` in ```ProtocolModel```
128
-
- Regenerate models
129
-
- Add callback for new ```call``` in ```ChildProcess.kt```
130
-
- Use ```ConcreteExecutor.withProcess``` method
131
-
- ___Important___ - do not add `Rdgen` as implementation dependency, it breaks some `.jar`s as it contains `kotlin-compiler-embeddable`.
132
-
5. Logs
133
-
134
-
There is ```UtRdLogger``` where you can configure level via ```log4j2.xml```.
135
-
172
+
- ```ProcessWithRdServer``` - also starts Rd server and waits for connection.
173
+
- `ClientProtocolBuilder` - use in client process to correctly connect to `ProcessWithRdServer`.
174
+
3. How ```ProcessWithRdServer``` communication works:
175
+
- Choose free port
176
+
- Create client process, pass port as argument
177
+
- Both processes create protocols, bind model and setup callbacks
178
+
- Server process cannot send messages before child creates protocol, otherwise messages will be lost. So client process needs to signal that he is ready.
179
+
- Client process creates special file in temp dir, that is observed by parent process.
180
+
- When parent process spots file - he deletes it, and then sends special message for client process confirming communication succeed.
181
+
- Only after client process answer reaches server - then processes are ready.
182
+
4. How to write custom RPC commands
183
+
- Add new ```call``` in some model, for example in ```EngineProcessModel```.
184
+
- Regenerate models: there are special gradle tasks for it in `utbot-rd/build.gradle` file.
185
+
- Add callback for new ```call``` in corresponding start files, for example in `EngineProcessMain.kt`.
186
+
- ___Important___ - do not add [`Rdgen`](https://mvnrepository.com/artifact/com.jetbrains.rd/rd-gen) as implementation dependency, it breaks some `.jar`s as it contains `kotlin-compiler-embeddable`.
187
+
5. Logs & Debug
188
+
- Logs - [inter process logging](./contributing/InterProcessLogging.md)
189
+
- Debug - [inter process debugging](./contributing/InterProcessDebugging.md)
136
190
6. Custom protocol marshalling types
137
-
138
-
Do not spend time on it until:
139
-
- Cyclic dependencies removed from UtModels
140
-
- Kotlinx.serialization is used
191
+
Do not spend time on it until UtModels would get simpler, for example Kotlinx.serialization compatible.
0 commit comments