Doc structure revision (#1387)

* Interprocess debugging doc added

* doc_structure_revision

Author: olganaumenko

CONTRIBUTING.md

@@ -32,13 +32,13 @@ There are so many ways to contribute. Choose yours and find the relevant short g
 ## Reporting a bug
 
 1. Check if the bug (a true bug!) has already been reported: search through [UnitTestBot issues](https://github.com/UnitTestBot/UTBotJava/issues). Please, don't duplicate.
-2. Check with the [Naming and labeling conventions](Conventions.md).
+2. Check with the [Naming and labeling conventions](docs/contributing/Conventions.md).
 3. Make sure you have all the necessary information as per [template](https://github.com/UnitTestBot/UTBotJava/issues/new?assignees=&labels=&template=bug_report.md&title=) and create the bug reporting issue.
 
 ## Requesting a feature
 
 1. Check if the feature has already been requested: search through [UnitTestBot issues](https://github.com/UnitTestBot/UTBotJava/issues).
-2. Check with our [Naming and labeling conventions](Conventions.md).
+2. Check with our [Naming and labeling conventions](docs/contributing/Conventions.md).
 3. Make sure you are able to provide all the necessary information as per [template](https://github.com/UnitTestBot/UTBotJava/issues/new?assignees=&labels=&template=feature_request.md&title=) and create the feature request issue.
 
 ## Contributing the code (bug fix or feature implementation)
@@ -66,7 +66,7 @@ Do you need to create an issue if you want to fix a bug?
    * Before creating the pull request perform the tests you find necessary for your code changes.
    * When implementing something new, it's great to find real users and ask them to try out your feature — to prove
      the necessity and quality of your suggestion.
-6. Check with the [Naming and labeling conventions](Conventions.md).
+6. Check with the [Naming and labeling conventions](docs/contributing/Conventions.md).
 7. Create the pull request, and you'll see if the automated tests pass on GitHub. Your reviewer will possibly recommend
    you more tests.
 
@@ -87,7 +87,7 @@ is the same as for the code. If you'd like to improve the existing docs or to ad
 2. Create your own fork of the code.
 3. Clone the forked repository to your local machine.
 4. Implement changes to docs (change the existing doc or create a new one). Usually, we create a new doc for a new feature, not for the small fixes. You are not obliged to write a detailed text about the feature you implement. You have to only describe it properly in both the related issue and the pull request, but it will be great if you still provide some docs.
-6. Check with the [Naming and labeling conventions](Conventions.md).
+6. Check with the [Naming and labeling conventions](docs/contributing/Conventions.md).
 7. Create the pull request, and we'll review it.
 
 * You can request a new doc — create an issue, using the [guide for a feature request](#Requesting-a-feature).

SystemRequirements.md

@@ -0,0 +1,61 @@
+<!---
+name: CI in UTBot Java
+route: /docs/java/ci/ci-in-utbot-java
+parent: Documentation
+menu: CI
+description: CI processes description
+--->
+
+# CI features
+
+UTBot Java offers contributors bunch of workflows e.g., the workflow _building the project and running tests_, the workflow _archiving plugin and CLI_.
+
+The main CI features in UTBot Java:
+* reproducible environment
+* available monitoring processes
+
+## Reproducible environment
+
+Depending on the resources where you are intended to build and test software environment will be different. The key goal is to provide the same environment on different resources. To do that we use Docker images with appropriate software, environment variables and OS settings.
+
+Crucial CI workflows run in those docker containers thus you can reproduce the environment locally. The environment can be used for running tests or for debugging ([see detailed information](https://github.com/UnitTestBot/UTBotJava/wiki/docker-for-utbot-java)).
+
+If you have any questions of where images are placed, how many they are, what software versions are used, visit [repository](https://github.com/UnitTestBot/infra-images) please (now is private, will be changed in the future), leave an issue with your questions or ask in DM.
+
+## All stages Monitoring
+
+Since the workflow has started you can check access to the metrics on our monitoring service (ask teammates for url). The server offers developers the following dashboards:
+
+* **Node Exporter Full** - metrics of consuming the RAM, CPU, Network and other resources on the host
+* **JVM dashboard** (don't forget to set job to `pushgateway`) - Java metrics
+* **Test executor statistics*** - RAM consuming by Java processes
+* **cAdvisor: container details*** - system resources consuming by certain container
+* **cAdvisor: host summary*** - summarized system resources consuming by all containers
+
+**Note:** * developed by UTBot team
+
+When you open a dashboard you need to choose valid instance. GitHub runs **each job on separate runner** so instance ID (`HOSTNAME` env var) would be different. But all instances have **the same Run ID** (`GITHUB_RUN_ID` env var). Follow this steps:
+
+1. Go to Actions and open your Run;
+2. Expand job list and choose any job you need;
+3. At the right you'll see a list of steps. You need step `Run monitoring`;
+4. Find the string like:
+```
+Find your Prometheus metrics using label {instance="2911909439-7f83f93ff335"}
+```
+5. Copy value between double quotes and go to monitoring dashboard. Set `github` service and expand instance list, CTRL+F and paste copied value. Choose your instance
+
+<img width="398" alt="image" src="https://user-images.githubusercontent.com/25527604/186348770-c4d88867-5656-4733-bf8a-84cf2c2a638c.png">
+
+**Note:** label consists of two part - `${GITHUB_RUN_ID}-${HOSTNAME}`. Use only one part to find all jobs of your Run.
+
+# Available workflows
+
+| Workflow name  | What it's supposed to do | What it triggers on |
+| --- | --- | --- |
+| UTBot Java: build and run tests | Builds the project and runs tests for it  | **push** or **pull request** to the **main** branch |
+| [M] UTBot Java: build and run tests  | Builds the project and runs tests for it | **manual** call or call from **another workflow** |
+| [M] Run chosen tests | Runs a single test or tests in chosen package/class | **manual** call |
+| Plugin and CLI: publish as archives | Archives plugin and CLI and stores them attached to the workflow run report | **push** to the **main** branch |
+| [M] Plugin and CLI: publish as archives | Archives plugin and CLI and stores them attached to the workflow run report | **manual** call or call from **another workflow** |
+| [M] Publish on GitHub Packages | Publishes artifacts such as _utbot-api_, _utbot-core_, _utbot-framework_, etc., on GitHub Packages | **manual** call |

docs/ci/ci-in-utbot-java.md

@@ -0,0 +1,95 @@
+<!---
+name: Docker for UTBot Java
+route: /docs/java/ci/docker-for-utbot-java
+parent: Documentation
+menu: CI
+description: Setting up docker for UTBot Java building/debugging
+--->
+
+# Reproducible environment
+
+It's available to download docker image with the environment for UTBot. The environment is also used in the crucial CI scripts focused on building project and running tests.
+
+The docker image pre-installed environment includes:
+1. *Java 11* + *JDK* package
+3. *Gradle 7.4.2*
+3. *Kotlin compiler 1.7.10*
+
+It's based on Ubuntu [SOME VERSION].
+
+## How to install Docker
+
+Using reproducible environment requires Docker installed.
+
+The detailed information of how to install Docker can be found on the [official site](https://docs.docker.com/engine/install/).
+
+## How to run tests in docker container
+
+Do the following steps to run tests in docker container:
+
+1. Pull docker image
+```
+docker pull unittestbot/java-env:java11-zulu-jdk-gradle7.4.2-kotlinc1.7.0
+```
+2. Run docker container
+```bash
+# -v <utbot-repository-root>:/usr/utbot-debug - mounts the host directory into the container directory
+# -it - make the container look like a terminal connection session
+# -w /usr/utbot-tests - sets up working directory inside the container
+docker run -it -v <utbot-repository-root>:/usr/utbot-tests --name utbot-tests -w /usr/utbot-tests unittestbot/java-env:java11-zulu-jdk-fx-gradle7.4.2-kotlinc1.7.0
+```
+3. Do whatever you want
+
+* Build UTBot and run tests:
+```
+gradle clean build --no-daemon
+```
+* Build UTBot without running tests:
+```
+gradle clean build --no-daemon -x test
+```
+* Run tests for *utbot-framework* project *CustomerExamplesTest* class:
+```
+gradle :utbot-framework:test --no-daemon --tests "org.utbot.examples.collections.CustomerExamplesTest"
+```
+4. Exit container
+```
+exit
+```
+
+## How to debug UTBot in docker container
+
+Do the following steps to debug UTBot in docker container:
+
+1. Set up configuration for remote debug in IntelliJ IDEA
+
+**Run/Debug Configurations** → **Add New Configuration** → Choose **Remote JVM Debug** → Set up **Configuration name** → **Ok**
+
+2. Pull docker image
+```
+docker pull unittestbot/java-env:java11-zulu-jdk-fx-gradle7.4.2-kotlinc1.7.0
+```
+3. Run docker container
+```bash
+# -v <utbot-repository-root>:/usr/utbot-debug - mounts the host directory into the container directory
+# -it - make the container look like a terminal connection session
+# -w /usr/utbot-tests - sets up working directory inside the container
+# -p 5005:5005 - mounts the host port into the container port (debugging port)
+docker run -it -p 5005:5005 -v <utbot-repository-root>:/usr/utbot-debug --name utbot-debug -w /usr/utbot-tests unittestbot/java-env:java11-zulu-jdk-fx-gradle7.4.2-kotlinc1.7.0
+```
+4. Set up gradle options for remote debug:
+```
+export GRADLE_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
+```
+5. Start building and running tests
+```
+gradle clean build --no-daemon
+```
+6. Attach in IntelliJ IDEA to the gradle process in the container
+
+Set up **breakpoints** wherever you want → **Run** new **Configuration** in **Debug** mode
+
+7. Exit container
+```
+exit
+```

docs/ci/docker-for-utbot-java.md

@@ -0,0 +1,21 @@
+<!---
+name: SSH session with GitHub agent
+route: /docs/java/ci/ssh-session-with-github-agent
+parent: Documentation
+menu: CI
+description: How to setup SSH session with GitHub agent
+--->
+
+# SSH session with GitHub agent
+
+It's available to use **action** letting set up SSH session with GitHub agent in your **workflows**. The detailed documentation with the examples of use can be found in the [official repository](https://github.com/mxschmitt/action-tmate).
+
+The action setting SSH session can be easily plugged in your workflow with the example below:
+```
+- name: Setup tmate session
+  uses: mxschmitt/action-tmate@v3
+```
+
+When the action is plugged in the workflow log (the part corresponding to tmate action log) can be found the URL. By the URL you can access the terminal of your host.
+
+There are also some ways to setup action behavior. E.g., the default behavior of the action is to remain SSH session open until the workflow times out. It's available to setup timeout parameter yourself.

docs/ci/ssh-session-with-github-agent.md

@@ -0,0 +1,82 @@
+# Interprocess debugging of UnitTestBot Java
+
+### Background
+
+We have split the UnitTestBot machinery into three processes. This approach has improved UnitTestBot capabilities, e.g. 
+provided support for various JVMs and scenarios, but also complicated the debugging flow.
+
+These are UnitTestBot processes (according to the execution order):
+
+* IDE process
+* Engine process
+* Concrete execution process
+
+Usually, main problems happen in the Engine process, but it is not the process we run first.
+The most straightforward way to debug the Engine process is the following.
+
+### Enable debugging for the Engine process
+
+1. Open `org/utbot/framework/UtSettings.kt`.
+2. Set `runIdeaProcessWithDebug` property to _true_. This enables `EngineProcess.debugArgument`.
+3. Find `EngineProcess.debugArgument` at `org/utbot/intellij/plugin/process/EngineProcess` and check the parameters of the debug run:
+
+    `"-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,quiet=y,address=5005"`
+
+* The `suspend` mode is enabled. Modify it in the case of some tricky timeouts in your scenario.
+* The port that will be used for debugging (`address`) is set to `5005`. Modify it if the port is already in use on your system.
+
+### Create a new run configuration for debugging the Engine process
+
+In addition to the `runIde` Gradle task that is supposed to run a new IDE instance, we should create another run 
+configuration.
+
+1. In your IntelliJ IDEA go to **Ru**n > **Edit configurations…**.
+2. In the **Run/Debug Configuration** dialog, click **`+`** on the toolbar.
+3. In the **Run/Debug Configuration Templates** dialog that opens, select a **Remote JVM Debug** configuration type.
+4. Check that **Port** has the same number as the `address` parameter from the `EngineProcess.debugArgument` mentioned above.
+5. Give the new run configuration a meaningful name and save the run configuration.
+
+### How to debug
+
+1. In your current IntelliJ IDEA, use breakpoints to define where the program needs to be stopped. For example, set the breakpoints at
+
+    `EngineProcess.createTestGenerator()`<br>
+    `engineModel().createTestGenerator.startSuspending()`
+
+2. Start the debugger session (**Shift+F9**) for the `runIde` Gradle task (wait for the debug IDE instance to open).
+3. Generate tests with UnitTestBot in the debug IDE instance. Make sure symbolic execution is turned on.
+4. The debug IDE instance will stop generation (if you have not changed the debug parameters). If you take no action, test generation will be cancelled by timeout.
+5. When the Engine process started (build processes have finished, and the progress bar says: _"Generate tests: read 
+   classes"_), start the debugger session (**Shift+F9**) for your newly created Remote JVM Debug run configuration.
+6. Wait for the program to be suspended upon reaching the first breakpoint.
+
+### Interprocess call mapping
+
+Now you are standing on a breakpoint in the IDE process, for example, the process stopped on:
+
+    `EngineProcess.createTestGenerator()`
+
+If you resume the process it reaches the next breakpoint (you are still in the IDE process):
+
+    `engineModel().createTestGenerator.startSuspending()`
+
+It seems that the test generation itself should occur in the Engine process and there should be an outbound point of the IDE process. How can we find it? An how can we reach the inbound point of the Engine process?
+
+Standing on the breakpoint` engineModel().createTestGenerator.startSuspending()`, you may **Go to Declaration or 
+Usage** and navigate to the definition of `RdCall` (which is responsible for cross-process communication) in `EngineProcessModel.createTestGenerator`.
+
+Now **Find Usages** for `EngineProcessModel.createTestGenerator` and see the point where `RdCall` is passed to the next method:
+
+    synchronizer.measureExecutionForTermination()
+
+This is the point where `RdCall` is called in the Engine process.
+
+Actually you could have skipped the previous step and used **Find Usages** right away, but it is useful to know where `RdCall` is defined.
+
+If you are interested in the trailing lambda of `synchronizer.measureExecutionForTermination()`, set the breakpoint here.
+
+#### Architectural notice
+
+We must place the outbound point of the IDE process and the inbound point of the Engine process as close as possible. 
+They may be two lambda-parameters of the same function. In this case we hope that the developer will not spend time on straying around.
+