Skip to content

Commit be40931

Browse files
maxceemmaxceem
authored
Merge pull request #415 from meshde/local-setup-improvements
Easy Local Setup of Project Service with Connect App using docker-compose.
2 parents b8d9382 + 27c8a93 commit be40931

File tree

8 files changed

+328
-0
lines changed

8 files changed

+328
-0
lines changed

‎README.md

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ Microservice to manage CRUD operations for all things Projects.
5454
```
5555
Alternatively, you may update `config/local.js` and replace `dockerhost` with your docker IP address.<br>
5656
You may try using command `docker-machine ip` to get your docker IP, but it works not for all systems.
57+
Also, be sure to update `busApiUrl` if you are running `tc-bus-api` locally. (See below)
5758

5859
Explanation of configs:
5960
- `config/mock.local.js` - Use local `mock-services` from docker to mock Identity and Member services instead of using deployed at Topcoder dev environment.
@@ -92,14 +93,57 @@ This command will create sample metadata entries in the DB (duplicate what is cu
9293

9394
To retrieve data from DEV env we need to provide a valid user token. You may login to http://connect.topcoder-dev.com and find the Bearer token in the request headers using browser dev tools.
9495

96+
### Local Deployment with other Topcoder Services.
97+
98+
* There exists an alternate `docker-compose.yml` file that can be used to spawn containers for the following services:
99+
100+
| Service | Name | Port |
101+
|----------|:-----:|:----:|
102+
| PostGreSQL DB | db | 5432 |
103+
| ElasticSearch | esearch | 9200,9300 |
104+
| RabbitMQ | queue | 5672, 15672 |
105+
| Zookeeper | zookeeper | 2181 |
106+
| Kafka | kafka | 9092 |
107+
| [tc-bus-api](https://github.com/topcoder-platform/tc-bus-api) | tc-bus-api | 8002 |
108+
| [project-processor-es](https://github.com/topcoder-platform/project-processor-es) | project-processor-es | 5000 |
109+
| [tc-notifications-api](https://github.com/topcoder-platform/tc-notifications) | tc-notifications-api | 4000 |
110+
| [tc-notifications-processor](https://github.com/topcoder-platform/tc-notifications) | tc-notifications-processor | 4001 |
111+
112+
* To have kafka create a list of desired topics on startup, there exists a file with the path `local/full/kafka-client/topics.txt`. Each line from the file will be added as a topic.
113+
* To run these services simply run the following commands:
114+
115+
```bash
116+
export AUTH0_CLIENT_ID=<insert required value here>
117+
export AUTH0_CLIENT_SECRET=<insert required value here>
118+
export AUTH0_URL=<insert required value here>
119+
export AUTH0_AUDIENCE=<insert required value here>
120+
export AUTH0_PROXY_SERVER_URL=<insert required value here>
121+
122+
cd local/full
123+
docker-compose up -d
124+
```
125+
126+
* The environment variables specified in the commands above will be passed onto the containers that have been configured to read them.
127+
* The above command will start all containers in the background.
128+
* To view the logs of any of the services use the following command, replacing "SERVICE_NAME" with the corresponding value under the "Name" column in the above table:
129+
130+
```bash
131+
cd local/full
132+
docker-compose logs -f SERVICE_NAME
133+
```
134+
135+
* The containers have been configured such that all Topcoder services will wait until all the topics listed in `local/full/kafka-client/topics.txt` have been created. To monitor the progress of topic creation, you can view the logs of the `kafka-client` service, which will exit when all topics have been created.
136+
95137
### Run Connect App with Project Service locally
96138

97139
To be able to run [Connect App](https://github.com/appirio-tech/connect-app) with the local setup of Project Service we have to do two things:
98140
1. Configurate Connect App to use locally deployed Project service inside `connect-app/config/constants/dev.js` set
99141

100142
```js
101143
PROJECTS_API_URL: 'http://localhost:8001'
144+
TC_NOTIFICATION_URL: 'http://localhost:4000/v5/notifications' # if tc-notfication-api has been locally deployed
102145
```
146+
103147
2. Bypass token validation in Project Service.
104148

105149
In `tc-project-service/node_modules/tc-core-library-js/lib/auth/verifier.js` add this to line 23:
@@ -142,3 +186,33 @@ You can paste **swagger.yaml** to [swagger editor](http://editor.swagger.io/) o
142186

143187
#### Deploying without docker
144188
If you don't want to use docker to deploy to localhost. You can simply run `npm run start:dev` from root of project. This should start the server on default port `8001`.
189+
190+
### Kafka Commands
191+
192+
If you've used `docker-compose` with the file `local/full/docker-compose.yml` to spawn kafka & zookeeper, you can use the following commands to manipulate kafka topics and messages:
193+
(Replace TOPIC_NAME with the name of the desired topic)
194+
195+
**Create Topic**
196+
197+
```bash
198+
docker exec tc-projects-kafka /usr/bin/kafka-topics --create --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic TOPIC_NAME
199+
```
200+
201+
**List Topics**
202+
203+
```bash
204+
docker exec -it tc-projects-kafka /usr/bin/kafka-topics --list --zookeeper zookeeper:2181
205+
```
206+
207+
**Watch Topic**
208+
209+
```bash
210+
docker exec -it tc-projects-kafka /usr/bin/kafka-console-consumer --bootstrap-server localhost:9092 --zookeeper zookeeper:2181 --topic TOPIC_NAME
211+
```
212+
213+
**Post Message to Topic**
214+
215+
```bash
216+
docker exec -it tc-projects-kafka /usr/bin/kafka-console-producer --topic TOPIC_NAME --broker-list localhost:9092
217+
```
218+
The message can be passed using `stdin`

‎local/full/docker-compose.base.yml

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
version: "2"
2+
services:
3+
tc-notifications-base:
4+
build:
5+
context: ./generic-tc-service
6+
args:
7+
NODE_VERSION: 8.2.1
8+
GIT_URL: https://github.com/topcoder-platform/tc-notifications
9+
GIT_BRANCH: v5-upgrade
10+
BYPASS_TOKEN_VALIDATION: 1
11+
environment:
12+
- VALID_ISSUERS="[\"https://topcoder-newauth.auth0.com/\",\"https://api.topcoder-dev.com\"]"
13+
- TC_API_V5_BASE_URL=http://host.docker.internal:8001/v5
14+
- TC_API_V4_BASE_URL=https://api.topcoder-dev.com/v4
15+
- TC_API_V3_BASE_URL=https://api.topcoder-dev.com/v3
16+
- KAFKA_URL=kafka:9092
17+
- AUTH_SECRET=secret
18+
- DATABASE_URL=postgresql://coder:mysecretpassword@notifications_db:5432/tc_notifications
19+
- JWKS_URI=test
20+
- LOG_LEVEL=debug
21+
- ENV=development
22+
- AUTH0_CLIENT_ID
23+
- AUTH0_CLIENT_SECRET
24+
- AUTH0_URL
25+
- AUTH0_AUDIENCE
26+
- AUTH0_PROXY_SERVER_URL

‎local/full/docker-compose.yml

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
version: "2"
2+
services:
3+
jsonserver:
4+
extends:
5+
file: ../docker-compose.yml
6+
service: jsonserver
7+
db:
8+
extends:
9+
file: ../docker-compose.yml
10+
service: db
11+
notifications_db:
12+
image: "postgres:9.5"
13+
expose:
14+
- "5432"
15+
environment:
16+
- POSTGRES_PASSWORD=mysecretpassword
17+
- POSTGRES_USER=coder
18+
- POSTGRES_DB=tc_notifications
19+
db_test:
20+
extends:
21+
file: ../docker-compose.yml
22+
service: db_test
23+
esearch:
24+
extends:
25+
file: ../docker-compose.yml
26+
service: esearch
27+
queue:
28+
extends:
29+
file: ../docker-compose.yml
30+
service: queue
31+
zookeeper:
32+
image: confluent/zookeeper
33+
ports:
34+
- "2181:2181"
35+
environment:
36+
zk_id: "1"
37+
kafka:
38+
image: confluent/kafka
39+
container_name: tc-projects-kafka
40+
depends_on:
41+
- zookeeper
42+
ports:
43+
- "9092:9092"
44+
kafka-client:
45+
build: ./kafka-client
46+
depends_on:
47+
- kafka
48+
- zookeeper
49+
tc-bus-api:
50+
build:
51+
context: ./generic-tc-service
52+
args:
53+
NODE_VERSION: 8.2.1
54+
GIT_URL: https://github.com/topcoder-platform/tc-bus-api
55+
GIT_BRANCH: dev
56+
BYPASS_TOKEN_VALIDATION: 1
57+
command: start kafka-client
58+
expose:
59+
- "3000"
60+
ports:
61+
- "8002:3000"
62+
depends_on:
63+
- kafka-client
64+
environment:
65+
- PORT=3000
66+
- KAFKA_URL=kafka:9092
67+
- JWT_TOKEN_SECRET=secret
68+
- VALID_ISSUERS="[\"https://topcoder-newauth.auth0.com/\",\"https://api.topcoder-dev.com\"]"
69+
project-processor-es:
70+
build:
71+
context: ./generic-tc-service
72+
args:
73+
NODE_VERSION: 8.11.3
74+
GIT_URL: https://github.com/topcoder-platform/project-processor-es
75+
GIT_BRANCH: develop
76+
command: start kafka-client
77+
expose:
78+
- "5000"
79+
ports:
80+
- "5000:5000"
81+
depends_on:
82+
- kafka-client
83+
environment:
84+
- PORT=5000
85+
- KAFKA_URL=kafka:9092
86+
- ES_HOST=esearch:9200
87+
tc-notifications-reset-db:
88+
extends:
89+
file: ./docker-compose.base.yml
90+
service: tc-notifications-base
91+
command: reset:db
92+
restart: on-failure
93+
expose:
94+
- "4002"
95+
ports:
96+
- "4002:4002"
97+
depends_on:
98+
- notifications_db
99+
- esearch
100+
environment:
101+
- PORT=4002
102+
tc-notifications-processor:
103+
extends:
104+
file: ./docker-compose.base.yml
105+
service: tc-notifications-base
106+
command: start kafka-client
107+
depends_on:
108+
- tc-notifications-reset-db
109+
- kafka-client
110+
expose:
111+
- "4001"
112+
ports:
113+
- "4001:4001"
114+
environment:
115+
- PORT=4001
116+
tc-notifications-api:
117+
extends:
118+
file: ./docker-compose.base.yml
119+
service: tc-notifications-base
120+
command: startAPI kafka-client
121+
depends_on:
122+
- tc-notifications-reset-db
123+
- kafka-client
124+
expose:
125+
- "4000"
126+
ports:
127+
- "4000:4000"
128+
environment:
129+
- PORT=4000

‎local/full/generic-tc-service/Dockerfile

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
ARG NODE_VERSION=8.2.1
2+
3+
FROM node:$NODE_VERSION
4+
ARG GIT_URL
5+
ARG GIT_BRANCH
6+
ARG BYPASS_TOKEN_VALIDATION
7+
8+
RUN git clone $GIT_URL /opt/app
9+
WORKDIR /opt/app
10+
RUN git checkout -b node-branch origin/$GIT_BRANCH
11+
12+
RUN npm install
13+
RUN if [ $BYPASS_TOKEN_VALIDATION -eq 1 ]; then sed -i '/decodedToken = jwt.decode/a \ callback(undefined, decodedToken.payload); return;' node_modules/tc-core-library-js/lib/auth/verifier.js; fi
14+
COPY docker-entrypoint.sh /opt/
15+
ENTRYPOINT ["/opt/docker-entrypoint.sh"]

‎local/full/generic-tc-service/docker-entrypoint.sh

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/bin/bash
2+
3+
HOST_DOMAIN="host.docker.internal"
4+
ping -q -c1 $HOST_DOMAIN > /dev/null 2>&1
5+
if [ $? -ne 0 ]; then
6+
HOST_IP=$(ip route | awk 'NR==1 {print $3}')
7+
echo -e "$HOST_IP\t$HOST_DOMAIN" >> /etc/hosts
8+
fi
9+
10+
if [ $# -eq 2 ]; then
11+
echo "Waiting for $2 to exit...."
12+
while ping -c1 $2 &>/dev/null
13+
do
14+
sleep 1
15+
done
16+
echo "$2 exited!"
17+
fi
18+
19+
cd /opt/app/ && npm run $1

‎local/full/kafka-client/Dockerfile

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
From confluent/kafka
2+
WORKDIR /app/
3+
COPY topics.txt .
4+
COPY create-topics.sh .
5+
ENTRYPOINT ["/app/create-topics.sh"]

‎local/full/kafka-client/create-topics.sh

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/bash
2+
3+
while read topic; do
4+
/usr/bin/kafka-topics --create --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic $topic
5+
done < topics.txt

‎local/full/kafka-client/topics.txt

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
connect.notification.email.project.member.invite.created
2+
connect.notification.project.active
3+
connect.notification.project.approved
4+
connect.notification.project.canceled
5+
connect.notification.project.completed
6+
connect.notification.project.created
7+
connect.notification.project.fileUploaded
8+
connect.notification.project.files.updated
9+
connect.notification.project.linkCreated
10+
connect.notification.project.member.assignedAsOwner
11+
connect.notification.project.member.copilotJoined
12+
connect.notification.project.member.invite.approved
13+
connect.notification.project.member.invite.created
14+
connect.notification.project.member.invite.rejected
15+
connect.notification.project.member.invite.requested
16+
connect.notification.project.member.invite.updated
17+
connect.notification.project.member.joined
18+
connect.notification.project.member.left
19+
connect.notification.project.member.managerJoined
20+
connect.notification.project.member.removed
21+
connect.notification.project.paused
22+
connect.notification.project.phase.transition.active
23+
connect.notification.project.phase.transition.completed
24+
connect.notification.project.phase.update.payment
25+
connect.notification.project.phase.update.progress
26+
connect.notification.project.phase.update.scope
27+
connect.notification.project.plan.ready
28+
connect.notification.project.plan.updated
29+
connect.notification.project.post.created
30+
connect.notification.project.post.edited
31+
connect.notification.project.product.update.spec
32+
connect.notification.project.submittedForReview
33+
connect.notification.project.team.updated
34+
connect.notification.project.timeline.adjusted
35+
connect.notification.project.timeline.milestone.added
36+
connect.notification.project.timeline.milestone.removed
37+
connect.notification.project.timeline.milestone.transition.active
38+
connect.notification.project.timeline.milestone.transition.completed
39+
connect.notification.project.timeline.milestone.transition.paused
40+
connect.notification.project.timeline.milestone.updated
41+
connect.notification.project.timeline.milestone.waiting.customer
42+
connect.notification.project.topic.created
43+
connect.notification.project.topic.updated
44+
connect.notification.project.updated
45+
connect.notification.project.updated.progress
46+
connect.notification.project.updated.spec
47+
connect.notification.project.work.transition.active
48+
connect.notification.project.work.transition.completed
49+
connect.notification.project.work.update.payment
50+
connect.notification.project.work.update.progress
51+
connect.notification.project.work.update.scope
52+
connect.notification.project.workitem.update.spec
53+
project.notification.create
54+
project.notification.delete
55+
project.notification.update

0 commit comments

Comments
 (0)