Extended Local Setup

README.md

@@ -54,6 +54,7 @@ Microservice to manage CRUD operations for all things Projects.
   ```
   Alternatively, you may update `config/local.js` and replace `dockerhost` with your docker IP address.<br>
   You may try using command `docker-machine ip` to get your docker IP, but it works not for all systems.
+  Also, be sure to update `busApiUrl` if you are running `tc-bus-api` locally. (See below)
 
   Explanation of configs:
   - `config/mock.local.js` - Use local `mock-services` from docker to mock Identity and Member services instead of using deployed at Topcoder dev environment.
@@ -92,14 +93,57 @@ This command will create sample metadata entries in the DB (duplicate what is cu
 
 To retrieve data from DEV env we need to provide a valid user token. You may login to http://connect.topcoder-dev.com and find the Bearer token in the request headers using browser dev tools.
 
+### Local Deployment with other Topcoder Services.
+
+* There exists an alternate `docker-compose.yml` file that can be used to spawn containers for the following services:
+
+|  Service | Name | Port  |
+|----------|:-----:|:----:|
+| PostGreSQL DB | db | 5432  |
+| ElasticSearch | esearch | 9200,9300 |
+| RabbitMQ | queue | 5672, 15672  |
+| Zookeeper | zookeeper | 2181  |
+| Kafka | kafka | 9092  |
+| [tc-bus-api](https://github.com/topcoder-platform/tc-bus-api) | tc-bus-api | 8002  |
+| [project-processor-es](https://github.com/topcoder-platform/project-processor-es) | project-processor-es | 5000  |
+| [tc-notifications-api](https://github.com/topcoder-platform/tc-notifications) | tc-notifications-api | 4000  |
+| [tc-notifications-processor](https://github.com/topcoder-platform/tc-notifications) | tc-notifications-processor | 4001  |
+
+* To have kafka create a list of desired topics on startup, there exists a file with the path `local/full/kafka-client/topics.txt`. Each line from the file will be added as a topic.
+* To run these services simply run the following commands:
+
+  ```bash
+  export AUTH0_CLIENT_ID=<insert required value here>
+  export AUTH0_CLIENT_SECRET=<insert required value here>
+  export AUTH0_URL=<insert required value here>
+  export AUTH0_AUDIENCE=<insert required value here>
+  export AUTH0_PROXY_SERVER_URL=<insert required value here>
+
+  cd local/full
+  docker-compose up -d
+  ```
+
+* The environment variables specified in the commands above will be passed onto the containers that have been configured to read them.
+* The above command will start all containers in the background.
+* To view the logs of any of the services use the following command, replacing "SERVICE_NAME" with the corresponding value under the "Name" column in the above table:
+
+  ```bash
+  cd local/full
+  docker-compose logs -f SERVICE_NAME
+  ```
+
+* The containers have been configured such that all Topcoder services will wait until all the topics listed in `local/full/kafka-client/topics.txt` have been created. To monitor the progress of topic creation, you can view the logs of the `kafka-client` service, which will exit when all topics have been created.
+
 ### Run Connect App with Project Service locally
 
 To be able to run [Connect App](https://github.com/appirio-tech/connect-app) with the local setup of Project Service we have to do two things:
 1. Configurate Connect App to use locally deployed Project service inside `connect-app/config/constants/dev.js` set
 
    ```js
    PROJECTS_API_URL: 'http://localhost:8001'
+   TC_NOTIFICATION_URL: 'http://localhost:4000/v5/notifications' # if tc-notfication-api has been locally deployed
    ```
+
 2. Bypass token validation in Project Service.
 
    In `tc-project-service/node_modules/tc-core-library-js/lib/auth/verifier.js` add this to line 23:
@@ -142,3 +186,33 @@ You can paste **swagger.yaml** to  [swagger editor](http://editor.swagger.io/) o
 
 #### Deploying without docker
 If you don't want to use docker to deploy to localhost. You can simply run `npm run start:dev` from root of project. This should start the server on default port `8001`.
+
+### Kafka Commands
+
+If you've used `docker-compose` with the file `local/full/docker-compose.yml` to spawn kafka & zookeeper, you can use the following commands to manipulate kafka topics and messages:
+(Replace TOPIC_NAME with the name of the desired topic)
+
+**Create Topic**
+
+```bash
+docker exec tc-projects-kafka /usr/bin/kafka-topics --create --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic TOPIC_NAME
+```
+
+**List Topics**
+
+```bash
+docker exec -it tc-projects-kafka /usr/bin/kafka-topics --list --zookeeper zookeeper:2181
+```
+
+**Watch Topic**
+
+```bash
+docker exec -it tc-projects-kafka /usr/bin/kafka-console-consumer --bootstrap-server localhost:9092 --zookeeper zookeeper:2181 --topic TOPIC_NAME
+```
+
+**Post Message to Topic**
+
+```bash
+docker exec -it tc-projects-kafka /usr/bin/kafka-console-producer --topic TOPIC_NAME --broker-list localhost:9092
+```
+The message can be passed using `stdin`

local/full/docker-compose.base.yml

@@ -0,0 +1,26 @@
+version: "2"
+services:
+  tc-notifications-base:
+    build:
+      context: ./generic-tc-service
+      args:
+        NODE_VERSION: 8.2.1
+        GIT_URL: https://github.com/topcoder-platform/tc-notifications
+        GIT_BRANCH: v5-upgrade
+        BYPASS_TOKEN_VALIDATION: 1
+    environment:
+      - VALID_ISSUERS="[\"https://topcoder-newauth.auth0.com/\",\"https://api.topcoder-dev.com\"]"
+      - TC_API_V5_BASE_URL=http://host.docker.internal:8001/v5
+      - TC_API_V4_BASE_URL=https://api.topcoder-dev.com/v4
+      - TC_API_V3_BASE_URL=https://api.topcoder-dev.com/v3
+      - KAFKA_URL=kafka:9092
+      - AUTH_SECRET=secret
+      - DATABASE_URL=postgresql://coder:mysecretpassword@notifications_db:5432/tc_notifications
+      - JWKS_URI=test
+      - LOG_LEVEL=debug
+      - ENV=development
+      - AUTH0_CLIENT_ID
+      - AUTH0_CLIENT_SECRET
+      - AUTH0_URL
+      - AUTH0_AUDIENCE
+      - AUTH0_PROXY_SERVER_URL

local/full/docker-compose.yml

@@ -0,0 +1,129 @@
+version: "2"
+services:
+  jsonserver:
+    extends:
+      file: ../docker-compose.yml
+      service: jsonserver
+  db:
+    extends:
+      file: ../docker-compose.yml
+      service: db
+  notifications_db:
+    image: "postgres:9.5"
+    expose:
+      - "5432"
+    environment:
+      - POSTGRES_PASSWORD=mysecretpassword
+      - POSTGRES_USER=coder
+      - POSTGRES_DB=tc_notifications
+  db_test:
+    extends:
+      file: ../docker-compose.yml
+      service: db_test
+  esearch:
+    extends:
+      file: ../docker-compose.yml
+      service: esearch
+  queue:
+    extends:
+      file: ../docker-compose.yml
+      service: queue
+  zookeeper:
+    image: confluent/zookeeper
+    ports:
+      - "2181:2181"
+    environment:
+      zk_id: "1"
+  kafka:
+    image: confluent/kafka
+    container_name: tc-projects-kafka
+    depends_on:
+      - zookeeper
+    ports:
+      - "9092:9092"
+  kafka-client:
+    build: ./kafka-client
+    depends_on:
+      - kafka
+      - zookeeper
+  tc-bus-api:
+    build:
+      context: ./generic-tc-service
+      args:
+        NODE_VERSION: 8.2.1
+        GIT_URL: https://github.com/topcoder-platform/tc-bus-api
+        GIT_BRANCH: dev
+        BYPASS_TOKEN_VALIDATION: 1
+    command: start kafka-client
+    expose:
+      - "3000"
+    ports:
+      - "8002:3000"
+    depends_on:
+      - kafka-client
+    environment:
+      - PORT=3000
+      - KAFKA_URL=kafka:9092
+      - JWT_TOKEN_SECRET=secret
+      - VALID_ISSUERS="[\"https://topcoder-newauth.auth0.com/\",\"https://api.topcoder-dev.com\"]"
+  project-processor-es:
+    build:
+      context: ./generic-tc-service
+      args:
+        NODE_VERSION: 8.11.3
+        GIT_URL: https://github.com/topcoder-platform/project-processor-es
+        GIT_BRANCH: develop
+    command: start kafka-client
+    expose:
+      - "5000"
+    ports:
+      - "5000:5000"
+    depends_on:
+      - kafka-client
+    environment:
+      - PORT=5000
+      - KAFKA_URL=kafka:9092
+      - ES_HOST=esearch:9200
+  tc-notifications-reset-db:
+    extends:
+      file: ./docker-compose.base.yml
+      service: tc-notifications-base
+    command: reset:db
+    restart: on-failure
+    expose:
+      - "4002"
+    ports:
+      - "4002:4002"
+    depends_on:
+      - notifications_db
+      - esearch
+    environment:
+      - PORT=4002
+  tc-notifications-processor:
+    extends:
+      file: ./docker-compose.base.yml
+      service: tc-notifications-base
+    command: start kafka-client
+    depends_on:
+      - tc-notifications-reset-db
+      - kafka-client
+    expose:
+      - "4001"
+    ports:
+      - "4001:4001"
+    environment:
+      - PORT=4001
+  tc-notifications-api:
+    extends:
+      file: ./docker-compose.base.yml
+      service: tc-notifications-base
+    command: startAPI kafka-client
+    depends_on:
+      - tc-notifications-reset-db
+      - kafka-client
+    expose:
+      - "4000"
+    ports:
+      - "4000:4000"
+    environment:
+      - PORT=4000

local/full/generic-tc-service/Dockerfile

@@ -0,0 +1,15 @@
+ARG NODE_VERSION=8.2.1
+
+FROM node:$NODE_VERSION
+ARG GIT_URL
+ARG GIT_BRANCH
+ARG BYPASS_TOKEN_VALIDATION
+
+RUN git clone $GIT_URL /opt/app
+WORKDIR /opt/app
+RUN git checkout -b node-branch origin/$GIT_BRANCH
+
+RUN npm install
+RUN if [ $BYPASS_TOKEN_VALIDATION -eq 1 ]; then sed -i '/decodedToken = jwt.decode/a \      callback(undefined, decodedToken.payload); return;' node_modules/tc-core-library-js/lib/auth/verifier.js; fi
+COPY docker-entrypoint.sh /opt/
+ENTRYPOINT ["/opt/docker-entrypoint.sh"]

local/full/generic-tc-service/docker-entrypoint.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+HOST_DOMAIN="host.docker.internal"
+ping -q -c1 $HOST_DOMAIN > /dev/null 2>&1
+if [ $? -ne 0 ]; then
+  HOST_IP=$(ip route | awk 'NR==1 {print $3}')
+  echo -e "$HOST_IP\t$HOST_DOMAIN" >> /etc/hosts
+fi
+
+if [ $# -eq 2 ]; then
+  echo "Waiting for $2 to exit...."
+  while ping -c1 $2 &>/dev/null
+    do
+      sleep 1
+    done
+  echo "$2 exited!"
+fi
+
+cd /opt/app/ && npm run $1

local/full/kafka-client/Dockerfile

@@ -0,0 +1,5 @@
+From confluent/kafka
+WORKDIR /app/
+COPY topics.txt .
+COPY create-topics.sh .
+ENTRYPOINT ["/app/create-topics.sh"]

local/full/kafka-client/create-topics.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+while read topic; do
+  /usr/bin/kafka-topics --create --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic $topic
+done < topics.txt

local/full/kafka-client/topics.txt

@@ -0,0 +1,55 @@
+connect.notification.email.project.member.invite.created
+connect.notification.project.active
+connect.notification.project.approved
+connect.notification.project.canceled
+connect.notification.project.completed
+connect.notification.project.created
+connect.notification.project.fileUploaded
+connect.notification.project.files.updated
+connect.notification.project.linkCreated
+connect.notification.project.member.assignedAsOwner
+connect.notification.project.member.copilotJoined
+connect.notification.project.member.invite.approved
+connect.notification.project.member.invite.created
+connect.notification.project.member.invite.rejected
+connect.notification.project.member.invite.requested
+connect.notification.project.member.invite.updated
+connect.notification.project.member.joined
+connect.notification.project.member.left
+connect.notification.project.member.managerJoined
+connect.notification.project.member.removed
+connect.notification.project.paused
+connect.notification.project.phase.transition.active
+connect.notification.project.phase.transition.completed
+connect.notification.project.phase.update.payment
+connect.notification.project.phase.update.progress
+connect.notification.project.phase.update.scope
+connect.notification.project.plan.ready
+connect.notification.project.plan.updated
+connect.notification.project.post.created
+connect.notification.project.post.edited
+connect.notification.project.product.update.spec
+connect.notification.project.submittedForReview
+connect.notification.project.team.updated
+connect.notification.project.timeline.adjusted
+connect.notification.project.timeline.milestone.added
+connect.notification.project.timeline.milestone.removed
+connect.notification.project.timeline.milestone.transition.active
+connect.notification.project.timeline.milestone.transition.completed
+connect.notification.project.timeline.milestone.transition.paused
+connect.notification.project.timeline.milestone.updated
+connect.notification.project.timeline.milestone.waiting.customer
+connect.notification.project.topic.created
+connect.notification.project.topic.updated
+connect.notification.project.updated
+connect.notification.project.updated.progress
+connect.notification.project.updated.spec
+connect.notification.project.work.transition.active
+connect.notification.project.work.transition.completed
+connect.notification.project.work.update.payment
+connect.notification.project.work.update.progress
+connect.notification.project.work.update.scope
+connect.notification.project.workitem.update.spec
+project.notification.create
+project.notification.delete
+project.notification.update