diff --git a/README.md b/README.md index e2790ac4..2a764cab 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,7 @@ Microservice to manage CRUD operations for all things Projects. ``` Alternatively, you may update `config/local.js` and replace `dockerhost` with your docker IP address.
You may try using command `docker-machine ip` to get your docker IP, but it works not for all systems. + Also, be sure to update `busApiUrl` if you are running `tc-bus-api` locally. (See below) Explanation of configs: - `config/mock.local.js` - Use local `mock-services` from docker to mock Identity and Member services instead of using deployed at Topcoder dev environment. @@ -92,6 +93,47 @@ This command will create sample metadata entries in the DB (duplicate what is cu To retrieve data from DEV env we need to provide a valid user token. You may login to http://connect.topcoder-dev.com and find the Bearer token in the request headers using browser dev tools. +### Local Deployment with other Topcoder Services. + +* There exists an alternate `docker-compose.yml` file that can be used to spawn containers for the following services: + +| Service | Name | Port | +|----------|:-----:|:----:| +| PostGreSQL DB | db | 5432 | +| ElasticSearch | esearch | 9200,9300 | +| RabbitMQ | queue | 5672, 15672 | +| Zookeeper | zookeeper | 2181 | +| Kafka | kafka | 9092 | +| [tc-bus-api](https://github.com/topcoder-platform/tc-bus-api) | tc-bus-api | 8002 | +| [project-processor-es](https://github.com/topcoder-platform/project-processor-es) | project-processor-es | 5000 | +| [tc-notifications-api](https://github.com/topcoder-platform/tc-notifications) | tc-notifications-api | 4000 | +| [tc-notifications-processor](https://github.com/topcoder-platform/tc-notifications) | tc-notifications-processor | 4001 | + +* To have kafka create a list of desired topics on startup, there exists a file with the path `local/full/kafka-client/topics.txt`. Each line from the file will be added as a topic. +* To run these services simply run the following commands: + + ```bash + export AUTH0_CLIENT_ID= + export AUTH0_CLIENT_SECRET= + export AUTH0_URL= + export AUTH0_AUDIENCE= + export AUTH0_PROXY_SERVER_URL= + + cd local/full + docker-compose up -d + ``` + +* The environment variables specified in the commands above will be passed onto the containers that have been configured to read them. +* The above command will start all containers in the background. +* To view the logs of any of the services use the following command, replacing "SERVICE_NAME" with the corresponding value under the "Name" column in the above table: + + ```bash + cd local/full + docker-compose logs -f SERVICE_NAME + ``` + +* The containers have been configured such that all Topcoder services will wait until all the topics listed in `local/full/kafka-client/topics.txt` have been created. To monitor the progress of topic creation, you can view the logs of the `kafka-client` service, which will exit when all topics have been created. + ### Run Connect App with Project Service locally To be able to run [Connect App](https://github.com/appirio-tech/connect-app) with the local setup of Project Service we have to do two things: @@ -99,7 +141,9 @@ To be able to run [Connect App](https://github.com/appirio-tech/connect-app) wit ```js PROJECTS_API_URL: 'http://localhost:8001' + TC_NOTIFICATION_URL: 'http://localhost:4000/v5/notifications' # if tc-notfication-api has been locally deployed ``` + 2. Bypass token validation in Project Service. In `tc-project-service/node_modules/tc-core-library-js/lib/auth/verifier.js` add this to line 23: @@ -142,3 +186,33 @@ You can paste **swagger.yaml** to [swagger editor](http://editor.swagger.io/) o #### Deploying without docker If you don't want to use docker to deploy to localhost. You can simply run `npm run start:dev` from root of project. This should start the server on default port `8001`. + +### Kafka Commands + +If you've used `docker-compose` with the file `local/full/docker-compose.yml` to spawn kafka & zookeeper, you can use the following commands to manipulate kafka topics and messages: +(Replace TOPIC_NAME with the name of the desired topic) + +**Create Topic** + +```bash +docker exec tc-projects-kafka /usr/bin/kafka-topics --create --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic TOPIC_NAME +``` + +**List Topics** + +```bash +docker exec -it tc-projects-kafka /usr/bin/kafka-topics --list --zookeeper zookeeper:2181 +``` + +**Watch Topic** + +```bash +docker exec -it tc-projects-kafka /usr/bin/kafka-console-consumer --bootstrap-server localhost:9092 --zookeeper zookeeper:2181 --topic TOPIC_NAME +``` + +**Post Message to Topic** + +```bash +docker exec -it tc-projects-kafka /usr/bin/kafka-console-producer --topic TOPIC_NAME --broker-list localhost:9092 +``` +The message can be passed using `stdin` diff --git a/local/full/docker-compose.base.yml b/local/full/docker-compose.base.yml new file mode 100644 index 00000000..6020be65 --- /dev/null +++ b/local/full/docker-compose.base.yml @@ -0,0 +1,26 @@ +version: "2" +services: + tc-notifications-base: + build: + context: ./generic-tc-service + args: + NODE_VERSION: 8.2.1 + GIT_URL: https://github.com/topcoder-platform/tc-notifications + GIT_BRANCH: v5-upgrade + BYPASS_TOKEN_VALIDATION: 1 + environment: + - VALID_ISSUERS="[\"https://topcoder-newauth.auth0.com/\",\"https://api.topcoder-dev.com\"]" + - TC_API_V5_BASE_URL=http://host.docker.internal:8001/v5 + - TC_API_V4_BASE_URL=https://api.topcoder-dev.com/v4 + - TC_API_V3_BASE_URL=https://api.topcoder-dev.com/v3 + - KAFKA_URL=kafka:9092 + - AUTH_SECRET=secret + - DATABASE_URL=postgresql://coder:mysecretpassword@notifications_db:5432/tc_notifications + - JWKS_URI=test + - LOG_LEVEL=debug + - ENV=development + - AUTH0_CLIENT_ID + - AUTH0_CLIENT_SECRET + - AUTH0_URL + - AUTH0_AUDIENCE + - AUTH0_PROXY_SERVER_URL diff --git a/local/full/docker-compose.yml b/local/full/docker-compose.yml new file mode 100644 index 00000000..b6f0f207 --- /dev/null +++ b/local/full/docker-compose.yml @@ -0,0 +1,129 @@ +version: "2" +services: + jsonserver: + extends: + file: ../docker-compose.yml + service: jsonserver + db: + extends: + file: ../docker-compose.yml + service: db + notifications_db: + image: "postgres:9.5" + expose: + - "5432" + environment: + - POSTGRES_PASSWORD=mysecretpassword + - POSTGRES_USER=coder + - POSTGRES_DB=tc_notifications + db_test: + extends: + file: ../docker-compose.yml + service: db_test + esearch: + extends: + file: ../docker-compose.yml + service: esearch + queue: + extends: + file: ../docker-compose.yml + service: queue + zookeeper: + image: confluent/zookeeper + ports: + - "2181:2181" + environment: + zk_id: "1" + kafka: + image: confluent/kafka + container_name: tc-projects-kafka + depends_on: + - zookeeper + ports: + - "9092:9092" + kafka-client: + build: ./kafka-client + depends_on: + - kafka + - zookeeper + tc-bus-api: + build: + context: ./generic-tc-service + args: + NODE_VERSION: 8.2.1 + GIT_URL: https://github.com/topcoder-platform/tc-bus-api + GIT_BRANCH: dev + BYPASS_TOKEN_VALIDATION: 1 + command: start kafka-client + expose: + - "3000" + ports: + - "8002:3000" + depends_on: + - kafka-client + environment: + - PORT=3000 + - KAFKA_URL=kafka:9092 + - JWT_TOKEN_SECRET=secret + - VALID_ISSUERS="[\"https://topcoder-newauth.auth0.com/\",\"https://api.topcoder-dev.com\"]" + project-processor-es: + build: + context: ./generic-tc-service + args: + NODE_VERSION: 8.11.3 + GIT_URL: https://github.com/topcoder-platform/project-processor-es + GIT_BRANCH: develop + command: start kafka-client + expose: + - "5000" + ports: + - "5000:5000" + depends_on: + - kafka-client + environment: + - PORT=5000 + - KAFKA_URL=kafka:9092 + - ES_HOST=esearch:9200 + tc-notifications-reset-db: + extends: + file: ./docker-compose.base.yml + service: tc-notifications-base + command: reset:db + restart: on-failure + expose: + - "4002" + ports: + - "4002:4002" + depends_on: + - notifications_db + - esearch + environment: + - PORT=4002 + tc-notifications-processor: + extends: + file: ./docker-compose.base.yml + service: tc-notifications-base + command: start kafka-client + depends_on: + - tc-notifications-reset-db + - kafka-client + expose: + - "4001" + ports: + - "4001:4001" + environment: + - PORT=4001 + tc-notifications-api: + extends: + file: ./docker-compose.base.yml + service: tc-notifications-base + command: startAPI kafka-client + depends_on: + - tc-notifications-reset-db + - kafka-client + expose: + - "4000" + ports: + - "4000:4000" + environment: + - PORT=4000 diff --git a/local/full/generic-tc-service/Dockerfile b/local/full/generic-tc-service/Dockerfile new file mode 100644 index 00000000..3c93b79b --- /dev/null +++ b/local/full/generic-tc-service/Dockerfile @@ -0,0 +1,15 @@ +ARG NODE_VERSION=8.2.1 + +FROM node:$NODE_VERSION +ARG GIT_URL +ARG GIT_BRANCH +ARG BYPASS_TOKEN_VALIDATION + +RUN git clone $GIT_URL /opt/app +WORKDIR /opt/app +RUN git checkout -b node-branch origin/$GIT_BRANCH + +RUN npm install +RUN if [ $BYPASS_TOKEN_VALIDATION -eq 1 ]; then sed -i '/decodedToken = jwt.decode/a \ callback(undefined, decodedToken.payload); return;' node_modules/tc-core-library-js/lib/auth/verifier.js; fi +COPY docker-entrypoint.sh /opt/ +ENTRYPOINT ["/opt/docker-entrypoint.sh"] diff --git a/local/full/generic-tc-service/docker-entrypoint.sh b/local/full/generic-tc-service/docker-entrypoint.sh new file mode 100755 index 00000000..312b4f0a --- /dev/null +++ b/local/full/generic-tc-service/docker-entrypoint.sh @@ -0,0 +1,19 @@ +#!/bin/bash + +HOST_DOMAIN="host.docker.internal" +ping -q -c1 $HOST_DOMAIN > /dev/null 2>&1 +if [ $? -ne 0 ]; then + HOST_IP=$(ip route | awk 'NR==1 {print $3}') + echo -e "$HOST_IP\t$HOST_DOMAIN" >> /etc/hosts +fi + +if [ $# -eq 2 ]; then + echo "Waiting for $2 to exit...." + while ping -c1 $2 &>/dev/null + do + sleep 1 + done + echo "$2 exited!" +fi + +cd /opt/app/ && npm run $1 diff --git a/local/full/kafka-client/Dockerfile b/local/full/kafka-client/Dockerfile new file mode 100644 index 00000000..3d332b83 --- /dev/null +++ b/local/full/kafka-client/Dockerfile @@ -0,0 +1,5 @@ +From confluent/kafka +WORKDIR /app/ +COPY topics.txt . +COPY create-topics.sh . +ENTRYPOINT ["/app/create-topics.sh"] diff --git a/local/full/kafka-client/create-topics.sh b/local/full/kafka-client/create-topics.sh new file mode 100755 index 00000000..c5b33e62 --- /dev/null +++ b/local/full/kafka-client/create-topics.sh @@ -0,0 +1,5 @@ +#!/bin/bash + +while read topic; do + /usr/bin/kafka-topics --create --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic $topic +done < topics.txt diff --git a/local/full/kafka-client/topics.txt b/local/full/kafka-client/topics.txt new file mode 100644 index 00000000..85ac3f0f --- /dev/null +++ b/local/full/kafka-client/topics.txt @@ -0,0 +1,55 @@ +connect.notification.email.project.member.invite.created +connect.notification.project.active +connect.notification.project.approved +connect.notification.project.canceled +connect.notification.project.completed +connect.notification.project.created +connect.notification.project.fileUploaded +connect.notification.project.files.updated +connect.notification.project.linkCreated +connect.notification.project.member.assignedAsOwner +connect.notification.project.member.copilotJoined +connect.notification.project.member.invite.approved +connect.notification.project.member.invite.created +connect.notification.project.member.invite.rejected +connect.notification.project.member.invite.requested +connect.notification.project.member.invite.updated +connect.notification.project.member.joined +connect.notification.project.member.left +connect.notification.project.member.managerJoined +connect.notification.project.member.removed +connect.notification.project.paused +connect.notification.project.phase.transition.active +connect.notification.project.phase.transition.completed +connect.notification.project.phase.update.payment +connect.notification.project.phase.update.progress +connect.notification.project.phase.update.scope +connect.notification.project.plan.ready +connect.notification.project.plan.updated +connect.notification.project.post.created +connect.notification.project.post.edited +connect.notification.project.product.update.spec +connect.notification.project.submittedForReview +connect.notification.project.team.updated +connect.notification.project.timeline.adjusted +connect.notification.project.timeline.milestone.added +connect.notification.project.timeline.milestone.removed +connect.notification.project.timeline.milestone.transition.active +connect.notification.project.timeline.milestone.transition.completed +connect.notification.project.timeline.milestone.transition.paused +connect.notification.project.timeline.milestone.updated +connect.notification.project.timeline.milestone.waiting.customer +connect.notification.project.topic.created +connect.notification.project.topic.updated +connect.notification.project.updated +connect.notification.project.updated.progress +connect.notification.project.updated.spec +connect.notification.project.work.transition.active +connect.notification.project.work.transition.completed +connect.notification.project.work.update.payment +connect.notification.project.work.update.progress +connect.notification.project.work.update.scope +connect.notification.project.workitem.update.spec +project.notification.create +project.notification.delete +project.notification.update