Improve local setup with docker compose

README.md

@@ -40,40 +40,24 @@ Also note that there is a `/health` endpoint that checks for the health of the a
 
 Config for tests are at `config/test.js`, it overrides some default config.
 
-## Local Kafka setup
-- Call extracted directory kafka_2.11-0.11.0.1 : `path_to_kafka`
-- Call our project root directory : `our_project_root_directory`
-- `http://kafka.apache.org/quickstart` contains details to setup and manage Kafka server,
-  below provides details to setup Kafka server in Mac, Windows will use bat commands in bin/windows instead
-- Download kafka at `https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/kafka_2.11-1.1.0.tgz`
-- Extract out the doanlowded tgz file
-- Go to extracted directory kafka_2.11-0.11.0.1
-- Start ZooKeeper server:
-  `bin/zookeeper-server-start.sh config/zookeeper.properties`
-- Use another terminal, go to same directory, start the Kafka server:
-  `bin/kafka-server-start.sh config/server.properties`
-- Note that the zookeeper server is at localhost:2181, and Kafka server is at localhost:9092
-- Use another terminal, go to same directory, create some topics:
-  `bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic project.action.create`
-  `bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic project.action.update`
-  `bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic project.action.delete`
-- Verify that the topics are created:
-  `bin/kafka-topics.sh --list --zookeeper localhost:2181`,
-  it should list out the created topics
-- run the producer and then write some message into the console to send to the `project.action.create` topic:
-  `bin/kafka-console-producer.sh --broker-list localhost:9092 --topic project.action.create`
-  in the console, write message, one message per line:
-  `{"topic":"project.action.create","originator":"project-api","timestamp":"2019-06-20T13:43:25.817Z","mime-type":"application/json","payload":{"resource":"project","createdAt":"2019-06-20T13:43:23.554Z","updatedAt":"2019-06-20T13:43:23.555Z","terms":[],"id":1,"name":"test project","description":"Hello I am a test project","type":"app","createdBy":40051333,"updatedBy":40051333,"projectEligibility":[],"bookmarks":[],"external":null,"status":"draft","lastActivityAt":"2019-06-20T13:43:23.514Z","lastActivityUserId":"40051333","members":[{"createdAt":"2019-06-20T13:43:23.555Z","updatedAt":"2019-06-20T13:43:23.625Z","id":2,"isPrimary":true,"role":"manager","userId":40051333,"updatedBy":40051333,"createdBy":40051333,"projectId":2,"deletedAt":null,"deletedBy":null}],"version":"v2","directProjectId":null,"billingAccountId":null,"estimatedPrice":null,"actualPrice":null,"details":null,"cancelReason":null,"templateId":null,"deletedBy":null,"attachments":null,"phases":null,"projectUrl":"https://connect.topcoder-dev.com/projects/2"}}`
-- Optionally, use another terminal, go to same directory, start a consumer to view the messages:
-  `bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic project.action.create --from-beginning`
-- If the kafka don't allow to input long message you can use this script to write message from file:
-  `path_to_kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic project.action.create < our_project_root_directory/test/data/project/project.action.create.json`
-- Writing/reading messages to/from other topics are similar. All example for messages are in:
-`our_project_root_directory/test/data`
 
-## Local Elasticsearch setup
+### Local Deployment for Kafka
+
+* There exists an alternate `docker-compose.yml` file that can be used to spawn containers for the following services:
+
+  |  Service | Name | Port  |
+  |----------|:-----:|:----:|
+  | ElasticSearch | esearch | 9200 |
+  | Zookeeper | zookeeper | 2181  |
+  | Kafka | kafka | 9092  |
 
-- In the `docker-es` folder, run `docker-compose up`
+* To have kafka create a list of desired topics on startup, there exists a file with the path `local/kafka-client/topics.txt`. Each line from the file will be added as a topic.
+* To run these services simply run the following commands:
+
+  ```bash
+  cd local
+  docker-compose up -d
+  ```
 
 ## Local deployment
 - Install dependencies `npm i`
@@ -135,12 +119,10 @@ npm run test:cov
 ```
 
 ## Verification
-
-- Call extracted directory kafka_2.11-0.11.0.1 : `path_to_kafka`
 - Call our project root directory : `our_project_root_directory`
-- Start kafka server, start elasticsearch, initialize Elasticsearch, start processor app
+- Start Docker servicees, initialize Elasticsearch, start processor app
 - Send message:
-  `path_to_kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic project.action.create < our_project_root_directory/test/data/project/project.action.create.json`
+    `docker exec -i project-processor-es-kafka /opt/kafka/bin/kafka-console-producer.sh --topic project.action.create --broker-list localhost:9092 < our_project_root_directory/test/data/project/project.action.create.json`
 - run command `npm run view-data projects 1` to view the created data, you will see the data are properly created:
 
 ```bash
@@ -193,7 +175,8 @@ info: {
 
 
 - Run the producer and then write some invalid message into the console to send to the `project.action.create` topic:
-  `bin/kafka-console-producer.sh --broker-list localhost:9092 --topic project.action.create`
+
+  `docker exec -it project-processor-es-kafka /opt/kafka/bin/kafka-console-producer.sh --topic project.action.create --broker-list localhost:9092`
   in the console, write message, one message per line:
   `{ "topic": "project.action.create", "originator": "project-api", "timestamp": "2019-02-16T00:00:00", "mime-type": "application/json", "payload": { "id": "invalid", "typeId": "8e17090c-465b-4c17-b6d9-dfa16300b0ff", "track": "Code", "name": "test", "description": "desc", "timelineTemplateId": "8e17090c-465b-4c17-b6d9-dfa16300b0aa", "phases": [{ "id": "8e17090c-465b-4c17-b6d9-dfa16300b012", "name": "review", "isActive": true, "duration": 10000 }], "prizeSets": [{ "type": "prize", "prizes": [{ "type": "winning prize", "value": 500 }] }], "reviewType": "code review", "tags": ["code"], "projectId": 123, "forumId": 456, "status": "Active", "created": "2019-02-16T00:00:00", "createdBy": "admin" } }`
 
@@ -203,7 +186,8 @@ info: {
 - Then in the app console, you will see error messages
 
 - Sent message to update data:
-  `path_to_kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic project.action.update < our_project_root_directory/test/data/project/project.action.update.json`
+
+   `docker exec -i project-processor-es-kafka /opt/kafka/bin/kafka-console-producer.sh --topic project.action.update --broker-list localhost:9092 < our_project_root_directory/test/data/project/project.action.update.json`
 - Run command `npm run view-data projects 1` to view the updated data, you will see the data are properly updated:
 
 ```bash
@@ -258,7 +242,7 @@ info: {
 
 
 - Run the producer and then write some invalid message into the console to send to the `project.action.create` topic:
-  `bin/kafka-console-producer.sh --broker-list localhost:9092 --topic project.action.create`
+  `docker exec -it project-processor-es-kafka /opt/kafka/bin/kafka-console-producer.sh --topic project.action.create`
   in the console, write message, one message per line:
   `{ "topic": "project.action.update", "originator": "project-api", "timestamp": "2019-02-17T01:00:00", "mime-type": "application/json", "payload": { "id": "173803d3-019e-4033-b1cf-d7205c7f774c", "typeId": "123", "track": "Code", "name": "test3", "description": "desc3", "timelineTemplateId": "8e17090c-465b-4c17-b6d9-dfa16300b0dd", "groups": ["group2", "group3"], "updated": "2019-02-17T01:00:00", "updatedBy": "admin" } }`
 
@@ -270,3 +254,42 @@ info: {
 
 - To test the health check API, run `export PORT=5000`, start the processor, then browse `http://localhost:5000/health` in a browser,
   and you will see result `{"checksRun":1}`
+
+
+
+### Kafka Commands
+
+If you've used `docker-compose` with the file `local/docker-compose.yml` to spawn kafka & zookeeper, you can use the following commands to manipulate kafka topics and messages:
+(Replace TOPIC_NAME with the name of the desired topic)
+
+**Create Topic**
+
+```bash
+docker exec project-processor-es-kafka /opt/kafka/bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic TOPIC_NAME
+```
+
+**List Topics**
+
+```bash
+docker exec project-processor-es-kafka /opt/kafka/bin/kafka-topics.sh --list --zookeeper zookeeper:2181
+```
+
+**Watch Topic**
+
+```bash
+docker exec  project-processor-es-kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --zookeeper zookeeper:2181 --topic TOPIC_NAME
+```
+
+**Post Message to Topic**
+
+```bash
+docker exec -it project-processor-es-kafka /opt/kafka/bin/kafka-console-producer.sh --topic TOPIC_NAME --broker-list localhost:9092
+```
+The message can be passed using `stdin`
+
+### Test
+```bash
+docker exec -i project-processor-es-kafka /opt/kafka/bin/kafka-console-producer.sh --topic project.action.create --broker-list localhost:9092 < test_message.json
+
+```
+All example for messages are in: our_project_root_directory/test/data

local/docker-compose.yml

@@ -0,0 +1,28 @@
+version: "2"
+services:
+  esearch:
+    image: "elasticsearch:2.3"
+    ports:
+      - "9200:9200"
+  zookeeper:
+    image: wurstmeister/zookeeper
+    ports:
+      - "2181:2181"
+    environment:
+      zk_id: "1"
+  kafka:
+    image: wurstmeister/kafka
+    container_name: project-processor-es-kafka
+    depends_on:
+      - zookeeper
+    ports:
+      - "9092:9092"
+    environment:
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
+      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+  kafka-client:
+    build: ./kafka-client
+    depends_on:
+      - kafka
+      - zookeeper

local/kafka-client/Dockerfile

@@ -0,0 +1,5 @@
+From wurstmeister/kafka
+WORKDIR /app/
+COPY topics.txt .
+COPY create-topics.sh .
+ENTRYPOINT ["/app/create-topics.sh"]

local/kafka-client/create-topics.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+while read topic; do
+  /usr/bin/kafka-topics --create --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic $topic
+done < topics.txt

local/kafka-client/topics.txt

@@ -0,0 +1,3 @@
+project.action.create
+project.action.delete
+project.action.update