|
| 1 | +# Local Setup with Multiple Services |
| 2 | + |
| 3 | +> **NOTE** This guide is adapted for Linux and macOS, if you run it on Windows you would have to adapt it by yourself. |
| 4 | +
|
| 5 | +If you want to change the code only of **Topcoder Project Service** (`tc-project-service`), it's enough to use the main [README](../../../README.md). |
| 6 | + |
| 7 | +Follow this guide in case you want to change the code of **Topcoder Project Service** (`tc-project-service`) **together** with the code of one of the next services: |
| 8 | +- [project-processor-es](https://github.com/topcoder-platform/project-processor-es) |
| 9 | +- [legacy-project-processor](https://github.com/topcoder-platform/legacy-project-processor) |
| 10 | +- [tc-notifications](https://github.com/topcoder-platform/tc-notifications) |
| 11 | +- [tc-bus-api](https://github.com/topcoder-platform/tc-bus-api) |
| 12 | + |
| 13 | +## How it works |
| 14 | + |
| 15 | +To communicate with any of the next services `tc-notifications`, `project-processor-es`, `legacy-project-processor` **Project Service** sends Kafka events and these service listen to the Kafka events do some stuff. **Project Service** don't send Kafka events directly, but uses a special service to send Kafka events called `tc-bus-api`. So no matter what service we want to update, first we have to setup Kafka with Zookeeper and `tc-bus-api`. |
| 16 | + |
| 17 | + |
| 18 | + |
| 19 | +*This diagram shows just some part of relations and services that are important for this guide, it doesn't show all of them.* |
| 20 | + |
| 21 | +### Elasticsearch indexing |
| 22 | + |
| 23 | +It's important to keep in mind how the indexing and reading data from Elasticsearch works. |
| 24 | + |
| 25 | +#### Read data |
| 26 | + |
| 27 | +Endpoints in **Project Service** should get data from the Elasticsearch index first. If no data is found, endpoints should try to get data from Database. |
| 28 | + |
| 29 | +#### Write data |
| 30 | + |
| 31 | +When some data is updated by **Project Service** it's directly changed in the Database. But **Project Service** doesn't change data in Elasticsearch directly. Instead of that, when some data is changed **Project Service** sends event to the Kafka (using `tc-bus-api`), and `project-processor-es` listens to the Kafka event and index updated data in Elasticsearch for **Project Service**. |
| 32 | +As a consequences, data in Elasticsearch is not updated immediately. |
| 33 | + |
| 34 | +## Steps to run locally |
| 35 | + |
| 36 | +To run **Project Service** with any of the services above we has to setup Zookeper + Kafka + [tc-bus-api](https://github.com/topcoder-platform/tc-bus-api) first to produce and receive Kafka events. |
| 37 | + |
| 38 | +### 1. Run Kafka + Zookeeper + create topics |
| 39 | + |
| 40 | + Clone [tc-project-service](https://github.com/topcoder-platform/tc-project-service) repository and run inside the `tc-project-service` folder: |
| 41 | + |
| 42 | + ```bash |
| 43 | + cd local/kafka |
| 44 | + docker-compose up |
| 45 | + ``` |
| 46 | + |
| 47 | + Wait until you see in the log `kafka_kafka-client_1 exited with code 0`. `kafka-client` would create all required topics. You may also make sure that all [required topics](../../../local/kafka/kafka-client/topics.txt) are created by running the next command to list all the topics: |
| 48 | + |
| 49 | + ```bash |
| 50 | + docker exec tc-common-kafka /opt/kafka/bin/kafka-topics.sh --list --zookeeper zookeeper:2181 |
| 51 | + ``` |
| 52 | + |
| 53 | +### 2. Run `tc-bus-api` |
| 54 | + |
| 55 | +Clone [tc-bus-api](https://github.com/topcoder-platform/tc-bus-api) repository. And see its [README](https://github.com/topcoder-platform/tc-bus-api/blob/dev/README.md) for details on how to run it. |
| 56 | + |
| 57 | +Mostly you have to do the next steps inside `tc-bus-api` folder: |
| 58 | + |
| 59 | +1. `npm install` |
| 60 | + |
| 61 | +2. To bypass token validation locally update `/node_modules/tc-core-library-js/lib/auth/verifier.js` manually and add this to line 23: `callback(undefined, decodedToken.payload); return;`. |
| 62 | + |
| 63 | +3. Set the next env mandatory environment variables: |
| 64 | + |
| 65 | + ```bash |
| 66 | + export KAFKA_URL=localhost:9092 |
| 67 | + export JWT_TOKEN_SECRET=secret |
| 68 | + export VALID_ISSUERS="[\"https:\/\/topcoder-newauth.auth0.com\/\",\"https:\/\/api.topcoder-dev.com\"]" |
| 69 | + ``` |
| 70 | + |
| 71 | +3. `npm start` |
| 72 | + |
| 73 | +### 3. Run Project Service |
| 74 | + |
| 75 | +To run **Project Service**, please, follow its README section [Steps to run locally](file:///Users/maks/dev/topcoder/projects/connect/tc-project-service/README.md#steps-to-run-locally). |
| 76 | + |
| 77 | +*Note, that during "Steps to run locally" you would use `local/docker-compose.yml` with a part of services, not the `local/full/docker-compose.yml` with all the services as we are starting them separately in this guide.* |
| 78 | + |
| 79 | +| **Optional** | |
| 80 | +|--| |
| 81 | +| Then next services are optional and you may start them only if need to change them or to test. | |
| 82 | + |
| 83 | +### 4. Run `project-processor-es` (optional) |
| 84 | + |
| 85 | +To run `project-processor-es`, please, follow its README section [Local setup]([file:///Users/maks/dev/topcoder/projects/connect/tc-project-service/README.md#steps-to-run-locally](https://github.com/topcoder-platform/project-processor-es#local-setup)), but **skip** the second step "Run docker compose with dependant services" as we already run the same docker in this guide. |
| 86 | + |
| 87 | +### 5. Run `tc-notifications` (optional) |
| 88 | + |
| 89 | +Clone [tc-notifications](https://github.com/topcoder-platform/tc-notifications) repository. You would have to start 2 services to make the Notification service fully work. |
| 90 | + |
| 91 | +1. `npm install` |
| 92 | + |
| 93 | +2. To bypass token validation locally update `/node_modules/tc-core-library-js/lib/auth/verifier.js` manually and add this to line 23: `callback(undefined, decodedToken.payload); return;`. |
| 94 | + |
| 95 | +3. Create `tc_notifications` database inside the PostgreSQL service that we run with by **Project Service** `local/docker-compose.yml`: |
| 96 | + |
| 97 | + ```bash |
| 98 | + PGPASSWORD=mysecretpassword psql -h localhost -p 5432 -U coder -d projectsdb -c 'CREATE DATABASE tc_notifications;' |
| 99 | + ``` |
| 100 | + |
| 101 | +4. In two separate terminal windows set the same environment variables: |
| 102 | + |
| 103 | + ```bash |
| 104 | + export AUTH0_CLIENT_ID=<insert required value here> |
| 105 | + export AUTH0_CLIENT_SECRET=<insert required value here> |
| 106 | + export AUTH0_URL=<insert required value here> |
| 107 | + export AUTH0_AUDIENCE=<insert required value here> |
| 108 | + export AUTH0_PROXY_SERVER_URL=<insert required value here> |
| 109 | + export AUTH_SECRET=secret |
| 110 | + export VALID_ISSUERS="[\"https:\/\/topcoder-newauth.auth0.com\/\",\"https:\/\/api.topcoder-dev.com\"]" |
| 111 | + export TC_API_V5_BASE_URL=http://localhost:8001/v5 |
| 112 | + export TC_API_V4_BASE_URL=https://api.topcoder-dev.com/v4 |
| 113 | + export TC_API_V3_BASE_URL=https://api.topcoder-dev.com/v3 |
| 114 | + export KAFKA_URL=localhost:9092 |
| 115 | + export DATABASE_URL=postgresql://coder:mysecretpassword@localhost:5432/tc_notifications |
| 116 | + export JWKS_URI=test |
| 117 | + export LOG_LEVEL=debug |
| 118 | + export ENV=development |
| 119 | + ``` |
| 120 | + |
| 121 | +5. In one terminal window: |
| 122 | + - Init DB: `npm run reset:db` |
| 123 | + - Run `PORT=4000 npm run startAPI` - API of Notifications Service |
| 124 | + |
| 125 | +6. In another terminal window: |
| 126 | + - Run `PORT=4001 npm run start` - Processor of Kafka events |
| 127 | + |
| 128 | +### 6. Run Connect App (optional) |
| 129 | + |
| 130 | +You may also run [Connect App](https://github.com/appirio-tech/connect-app) locally together with the current setup. Please, follow section "[Run Connect App with Project Service locally](file:///Users/maks/dev/topcoder/projects/connect/tc-project-service/README.md#run-connect-app-with-project-service-locally)". |
0 commit comments