Merge pull request #3 from hesibo/develop

Directly using Informix database

Author: ThomasKranitsas

ReadMe.md

@@ -2,8 +2,10 @@
 
 ## Dependencies
 
-- nodejs https://nodejs.org/en/ (v10+)
+- nodejs https://nodejs.org/en/ (v8)
 - Kafka
+- Informix
+- Docker, Docker Compose
 
 ## Configuration
 
@@ -26,11 +28,9 @@ The following parameters can be set in config files or in env variables:
 - AUTH0_CLIENT_ID: Auth0 client id, used to get TC M2M token
 - AUTH0_CLIENT_SECRET: Auth0 client secret, used to get TC M2M token
 - AUTH0_PROXY_SERVER_URL: Proxy Auth0 URL, used to get TC M2M token
-- V4_CHALLENGE_API_URL: V4 challenge api url, default value is 'https://api.topcoder-dev.com/v4/challenges'
-- V4_PLATFORMS_API_URL: v4 platforms api url, default value is 'https://api.topcoder-dev.com/v4/platforms'
-- V4_TECHNOLOGIES_API_URL:v4 technologies api url, default value is 'https://api.topcoder-dev.com/v4/technologies'
 - V5_CHALLENGE_API_URL: v5 challenge api url, default value is 'http://localhost:4000/v5/challenges'
 - V5_CHALLENGE_TYPE_API_URL: v5 challenge type api url, default value is 'http://localhost:4000/v5/challengeTypes'
+- INFORMIX: Informix database configuration parameters, refer `config/default.js` for more information
 
 There is a `/health` endpoint that checks for the health of the app. This sets up an expressjs server and listens on the environment variable `PORT`. It's not part of the configuration file and needs to be passed as an environment variable
 
@@ -67,33 +67,40 @@ Configuration for the tests is at `config/test.js`, only add such new configurat
   `bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic challenge.notification.create --from-beginning`
 - writing/reading messages to/from other topics are similar
 
+## Topcoder Informix Database Setup
+We will use Topcoder Informix database setup on Docker.
+
+Go to `docker-ifx` folder and run `docker-compose up`
+After the database has initialized, You can use a database GUI tool(example [DBeaver](https://dbeaver.io)) to run the sql script `docker-ifx/update.sql`.
+
 ## Mock V5 Challenge API
 Mock V5 challenge api is under `test/mock` folder. You can use command `npm run mock-api` to start the server.
 
 ## Local deployment
-1. Make sure that Kafka is running as per instructions above.
-2. From the project root directory, run the following command to install the dependencies
-```
-npm install
-```
-3. To run linters if required
-```
-npm run lint
+- Given the fact that the library used to access Informix DB depends on Informix Client SDK.
+We will run the application on Docker using a base image with Informix Client SDK installed and properly configured.
+For deployment, please refer to next section 'Local Deployment with Docker'
 
-npm run lint:fix # To fix possible lint errors
-```
-4. Start the mock server
+## Local Deployment with Docker
+
+1. Make sure that Kafka, mock server and Informix are running as per instructions above.
+
+2. Go to `docker` folder
+
+3. Rename the file `sample.api.env` to `api.env` And properly update the IP addresses to match your environment for the variables : KAFKA_URL, INFORMIX_HOST and V5_CHALLENGE_TYPE_API_URL( make sure to use IP address instead of hostname ( i.e localhost will not work)).Here is an example:
 ```
-npm run mock-api
+KAFKA_URL=192.168.31.8:9092
+INFORMIX_HOST=192.168.31.8
+V5_CHALLENGE_TYPE_API_URL=http://192.168.31.8:4000/v5/challengeTypes
 ```
-5. Start the processor and health check dropin
+
+4. Once that is done, go to run the following command
+
 ```
-npm start
+docker-compose up
 ```
 
-## Testing
-- Run `npm run test` to execute unit tests and generate coverage report.
-- RUN `npm run e2e` to execute e2e tests and generate coverage report.
+5. When you are running the application for the first time, It will take some time initially to download the image and install the dependencies
 
 ## Verification
 Refer `Verification.md`

Verification.md

@@ -1,57 +1,40 @@
-# Topcoder - Legacy Groups Processor
+# Topcoder - Legacy Challenge Processor
 
 ## Verification
-start Kafka server, start mock api server and start the processor
+start Kafka server, start Informix database, start mock api server and start the processor
 
 1. start kafka-console-producer to write messages to `challenge.notification.create` topic:
   `bin/kafka-console-producer.sh --broker-list localhost:9092 --topic challenge.notification.create`
 2. write message:
-  `{ "topic": "challenge.notification.create","originator": "challenge-api","timestamp": "2019-05-14T00:00:00.000Z","mime-type": "application/json","payload": { "id": "1a4ef3a8-ed35-40d1-b8a6-7371a700d011","typeId": "2f4ef3a8-ed35-40d1-b8a6-7371a700d098","track": "CODE","name": "test-for-legacy-challenge-processor","description": "<p>test</p>","phases": [{ "id": "id-1","name": "registration","isActive": true, "duration": 345600000 }, { "id": "id-2","name": "submission","isActive": true, "duration": 345600000 }, { "id": "id-3","name": "checkpoint","isActive": true, "duration": 172800000 } ], "prizeSets": [{ "type": "Code","prizes": [{ "type": "first-place", "value": 1000 }, { "type": "second-place","value": 500 }] }, { "type": "Check Point", "prizes": [{ "type": "first-place","value": 200 }, { "type": "second-place","value": 200 }, { "type": "third-place","value": 200 }] }],"reviewType": "COMMUNITY","markdown": false,"tags": ["Node.js","NodeJS","MongoDB","AWS"],"projectId": 5087,"forumId": 33059 } }`
-3. Watch the app console, It will show message successfully handled. And has info log `Create challenge entity in legacy system, the legacy id is XXX`. Now mark down the legacy id.
-4. Wait for a short period(1 minute), Then go to `https://api.topcoder-dev.com/v4/challenges/<legacy_id>` to verify the challenge has been created in legacy system. Note CODE challenge doesn't have checkpoint prize/phase, so the checkpoint prize is 0 in the legacy system.
+  `{ "topic": "challenge.notification.create","originator": "challenge-api","timestamp": "2019-05-14T00:00:00.000Z","mime-type": "application/json","payload": { "id": "1a4ef3a8-ed35-40d1-b8a6-7371a700d011","typeId": "2f4ef3a8-ed35-40d1-b8a6-7371a700d098","track": "CODE","name": "test-for-legacy-challenge-processor","description": "<p>test</p>","phases": [{ "id": "id-1","name": "registration","isActive": true, "duration": 345600000 }, { "id": "id-2","name": "submission","isActive": true, "duration": 345600000 }, { "id": "id-3","name": "checkpoint","isActive": true, "duration": 172800000 } ], "prizeSets": [{ "type": "Code","prizes": [{ "type": "first-place", "value": 1000 }, { "type": "second-place","value": 500 }] }, { "type": "Check Point", "prizes": [{ "type": "first-place","value": 200 }, { "type": "second-place","value": 200 }, { "type": "third-place","value": 200 }] }],"reviewType": "COMMUNITY","markdown": false,"tags": ["Java","JUnit","MongoDB","AWS"],"projectId": 5087,"forumId": 33059 } }`
+3. Use database GUI tool to execute following statements to verify AssetDTO data.
+  ```sql
+  select * from tcs_catalog:comp_catalog where component_id >= 2010;
+  select * from tcs_catalog:comp_categories where comp_categories_id >= 2010;
+  select * from tcs_catalog:comp_versions where comp_vers_id >= 2010;
+  select * from tcs_catalog:comp_version_dates where comp_version_dates_id >= 3000;
+  select * from tcs_catalog:comp_technology where comp_tech_id >= 2200;
+  ```
+4. Use database GUI tool to insert the following test data(Only AssetDTO data has been persisted right now, so you can't retrieve the corresponding AssetDTO using the legacyId provided by message payload). project_info table will contain component version id of challenge, you can use the comp_vers_id retrieve from step 3 to replace the `value` column.
+  ```sql
+  insert into tcs_catalog:project(project_id, project_status_id, project_category_id, create_user, create_date, modify_user, modify_date) values(2000, 2, 39, 132456, current, 132456, current);
+  insert into tcs_catalog:project_info(project_id, project_info_type_id, value, create_user, create_date, modify_user, modify_date) values(2000, 1, 2020, 132456, current, 132456, current);
+  ```
 5. start kafka-console-producer to write messages to `challenge.notification.update` topic:
   `bin/kafka-console-producer.sh --broker-list localhost:9092 --topic challenge.notification.update`
-6. write message(Use the legacy id you mark down in step 3, as a number):
-  `{ "topic": "challenge.notification.update","originator": "challenge-api","timestamp": "2019-05-14T00:00:00.000Z","mime-type": "application/json","payload": { "legacyId": <legacy_id>, "id": "1a4ef3a8-ed35-40d1-b8a6-7371a700d011","typeId": "2f4ef3a8-ed35-40d1-b8a6-7371a700d098","track": "CODE","name": "update-for-legacy-challenge-processor","description": "#Title\n##sub title 1\ntext\n##sub title2\nanother text\n","phases": [{ "id": "id-1","name": "registration","isActive": true, "duration": 345600000 }, { "id": "id-2","name": "submission","isActive": true, "duration": 345600000 }], "prizeSets": [{ "type": "Code","prizes": [{ "type": "first-place", "value": 800 }, { "type": "second-place","value": 400 }]}],"reviewType": "COMMUNITY","markdown":true,"tags": ["Node.js","NodeJS"],"projectId": 5087,"forumId": 33059 } }`
-7. Wait for a short period(1 minute), Then go to `https://api.topcoder-dev.com/v4/challenges/<legacy_id>` to verify the challenge has been updated in legacy system.
-8. Repeat step 1 to 7, to create other type of challenge. Here is the message payload for creating a Design challenge with checkpoint prizes:
-  `{ "topic": "challenge.notification.create","originator": "challenge-api","timestamp": "2019-05-14T00:00:00.000Z","mime-type": "application/json","payload": { "id": "1a4ef3a8-ed35-40d1-b8a6-7371a700d011","typeId": "3f4ef3a8-ed35-40d1-b8a6-7371a700d098","track": "WEB_DESIGNS","name": "test-design-demo","description": "<p>test design</p>","phases": [{ "id": "id-1","name": "registration","isActive": true, "duration": 345600000 }, { "id": "id-2","name": "submission","isActive": true, "duration": 345600000 }, { "id": "id-3","name": "checkpoint","isActive": true, "duration": 172800000 } ], "prizeSets": [{ "type": "Code","prizes": [{ "type": "first-place", "value": 1000 }, { "type": "second-place","value": 500 }] }, { "type": "Check Point", "prizes": [{ "type": "first-place","value": 200 }, { "type": "second-place","value": 200 }, { "type": "third-place","value": 200 }] }],"reviewType": "COMMUNITY","markdown": false,"tags": ["Node.js","NodeJS","MongoDB","AWS"],"projectId": 5087,"forumId": 33059 } }`
-
+6. write message
+  `{ "topic": "challenge.notification.update","originator": "challenge-api","timestamp": "2019-05-14T00:00:00.000Z","mime-type": "application/json","payload": { "legacyId": 2000, "id": "1a4ef3a8-ed35-40d1-b8a6-7371a700d011","typeId": "2f4ef3a8-ed35-40d1-b8a6-7371a700d098","track": "CODE","name": "update-for-legacy-challenge-processor","description": "#Title\n##sub title 1\ntext\n##sub title2\nanother text\n","phases": [{ "id": "id-1","name": "registration","isActive": true, "duration": 345600000 }, { "id": "id-2","name": "submission","isActive": true, "duration": 345600000 }], "prizeSets": [{ "type": "Code","prizes": [{ "type": "first-place", "value": 800 }, { "type": "second-place","value": 400 }]}],"reviewType": "COMMUNITY","markdown":true,"tags": ["Node.js","NodeJS", "PostgreSQL"],"projectId": 5087,"forumId": 33059 } }`
+7. Use database GUI tool to execute following statements to verify AssetDTO data.
+  ```sql
+  select * from tcs_catalog:comp_catalog where component_id >= 2010;
+  select * from tcs_catalog:comp_technology where comp_tech_id >= 2200;
+  ```
+8. Try to change track, write the following message
+  `{ "topic": "challenge.notification.update","originator": "challenge-api","timestamp": "2019-05-14T00:00:00.000Z","mime-type": "application/json","payload": { "legacyId": 2000, "id": "1a4ef3a8-ed35-40d1-b8a6-7371a700d011","typeId": "2f4ef3a8-ed35-40d1-b8a6-7371a700d098","track": "WEB_DESIGNS","name": "update-for-legacy-challenge-processor","description": "#Title\n##sub title 1\ntext\n##sub title2\nanother text\n","phases": [{ "id": "id-1","name": "registration","isActive": true, "duration": 345600000 }, { "id": "id-2","name": "submission","isActive": true, "duration": 345600000 }], "prizeSets": [{ "type": "Code","prizes": [{ "type": "first-place", "value": 800 }, { "type": "second-place","value": 400 }]}],"reviewType": "COMMUNITY","markdown":true,"tags": ["Node.js","NodeJS", "PostgreSQL"],"projectId": 5087,"forumId": 33059 } }`
+9. You would see error message in app console.
 
 ## Unit test Coverage
-  51 passing (23s)
-
-File                  |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
-----------------------|----------|----------|----------|----------|-------------------|
-All files             |    94.05 |    86.21 |    96.67 |    93.75 |                   |
- config               |      100 |      100 |      100 |      100 |                   |
-  default.js          |      100 |      100 |      100 |      100 |                   |
-  test.js             |      100 |      100 |      100 |      100 |                   |
- src                  |      100 |      100 |      100 |      100 |                   |
-  bootstrap.js        |      100 |      100 |      100 |      100 |                   |
-  constants.js        |      100 |      100 |      100 |      100 |                   |
- src/common           |    87.34 |       50 |    94.74 |    87.34 |                   |
-  helper.js           |    73.33 |        0 |    83.33 |    73.33 |       16,17,18,20 |
-  logger.js           |    90.63 |       60 |      100 |    90.63 |32,55,60,84,98,118 |
- src/services         |      100 |      100 |      100 |      100 |                   |
-  ProcessorService.js |      100 |      100 |      100 |      100 |                   |
+  It is not available right now
 
 ## E2E test Coverage
-
-  54 passing (2m)
-
-File                  |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
-----------------------|----------|----------|----------|----------|-------------------|
-All files             |    95.37 |    87.88 |     97.5 |    95.17 |                   |
- config               |      100 |      100 |      100 |      100 |                   |
-  default.js          |      100 |      100 |      100 |      100 |                   |
-  test.js             |      100 |      100 |      100 |      100 |                   |
- src                  |    94.12 |    66.67 |       90 |       94 |                   |
-  app.js              |    93.75 |    66.67 |       90 |    93.62 |          48,61,86 |
-  bootstrap.js        |      100 |      100 |      100 |      100 |                   |
-  constants.js        |      100 |      100 |      100 |      100 |                   |
- src/common           |    91.14 |    66.67 |      100 |    91.14 |                   |
-  helper.js           |    93.33 |       50 |      100 |    93.33 |                18 |
-  logger.js           |    90.63 |       70 |      100 |    90.63 |32,55,60,84,98,118 |
- src/services         |      100 |      100 |      100 |      100 |                   |
-  ProcessorService.js |      100 |      100 |      100 |      100 |                   |
+  It is not available right now

config/default.js

@@ -20,13 +20,23 @@ module.exports = {
   AUTH0_URL: process.env.AUTH0_URL || 'https://topcoder-dev.auth0.com/oauth/token',
   AUTH0_AUDIENCE: process.env.AUTH0_AUDIENCE || 'https://m2m.topcoder-dev.com/',
   TOKEN_CACHE_TIME: process.env.TOKEN_CACHE_TIME || 90,
-  AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID || 'e6oZAxnoFvjdRtjJs1Jt3tquLnNSTs0e',
-  AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET || 'OGCzOnQkhYTQpZM3NI0sD--JJ_EPcm2E7707_k6zX11m223LrRK1-QZL4Pon4y-D',
+  AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID || 'EkE9qU3Ey6hdJwOsF1X0duwskqcDuElW',
+  AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET || 'Iq7REiEacFmepPh0UpKoOmc6u74WjuoJriLayeVnt311qeKNBvhRNBe9BZ8WABYk',
   AUTH0_PROXY_SERVER_URL: process.env.AUTH0_PROXY_SERVER_URL,
 
-  V4_CHALLENGE_API_URL: process.env.V4_CHALLENGE_API_URL || 'https://api.topcoder-dev.com/v4/challenges',
-  V4_PLATFORMS_API_URL: process.env.V4_PLATFORMS_API_URL || 'https://api.topcoder-dev.com/v4/platforms',
-  V4_TECHNOLOGIES_API_URL: process.env.V4_TECHNOLOGIES_API_URL || 'https://api.topcoder-dev.com/v4/technologies',
   V5_CHALLENGE_API_URL: process.env.V5_CHALLENGE_API_URL || 'http://localhost:4000/v5/challenges',
-  V5_CHALLENGE_TYPE_API_URL: process.env.V5_CHALLENGE_TYPE_API_URL || 'http://localhost:4000/v5/challengeTypes'
+  V5_CHALLENGE_TYPE_API_URL: process.env.V5_CHALLENGE_TYPE_API_URL || 'http://localhost:4000/v5/challengeTypes',
+
+  // informix database configuration
+  INFORMIX: {
+    SERVER: process.env.IFX_SERVER || 'informixoltp_tcp', // informix server
+    DATABASE: process.env.IFX_DATABASE || 'tcs_catalog', // informix database
+    HOST: process.env.INFORMIX_HOST || 'localhost', // host
+    PROTOCOL: process.env.IFX_PROTOCOL || 'onsoctcp',
+    PORT: process.env.IFX_PORT || '2021', // port
+    DB_LOCALE: process.env.IFX_DB_LOCALE || 'en_US.57372',
+    USER: process.env.IFX_USER || 'informix', // user
+    PASSWORD: process.env.IFX_PASSWORD || '1nf0rm1x', // password
+    POOL_MAX_SIZE: parseInt(process.env.IFX_POOL_MAX_SIZE) || 10 // use connection pool in processor, the pool size
+  }
 }

docker-ifx/docker-compose.yml

@@ -0,0 +1,6 @@
+version: "3"
+services:
+  informix:
+    image: "appiriodevops/informix:cbbd0fa"
+    ports:
+      - "2021:2021"

docker-ifx/update.sql

@@ -0,0 +1 @@
+update tcs_catalog:id_sequences set next_block_start = 3000 where name = 'COMPVERSIONDATES_SEQ';

docker/Dockerfile

@@ -0,0 +1,46 @@
+FROM ibmcom/informix-innovator-c:12.10.FC12W1IE
+
+ARG servername=informix
+
+USER root
+RUN mkdir /app
+WORKDIR /home/informix
+
+RUN sed -i '/jessie-updates/d' /etc/apt/sources.list
+RUN apt-get -qq update && apt-get -qq install -y \
+  wget gcc g++ make xz-utils python2.7 git curl
+
+RUN wget -q -O node8.tar.xz https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.xz \
+  && tar xfJ node8.tar.xz && rm -rf node8.tar.xz
+
+ENV SERVERNAME=$servername
+
+COPY docker/esql /opt/ibm/informix/bin/
+
+RUN chmod +x /opt/ibm/informix/bin/esql
+RUN echo "informixoltp_tcp        onsoctcp        $SERVERNAME               sqlexec" \
+  > /opt/ibm/informix/etc/sqlhosts.informixoltp_tcp
+
+ENV INFORMIXDIR /opt/ibm/informix
+ENV INFORMIX_HOME /home/informix
+ENV INFORMIXSERVER informixoltp_tcp
+ENV INFORMIXTERM terminfo
+ENV CLIENT_LOCALE=en_US.utf8
+ENV DB_LOCALE=en_US.utf8
+ENV DBDATE Y4MD-
+ENV DBDELIMITER "|"
+ENV PATH /home/informix/node-v8.11.3-linux-x64/bin:${INFORMIXDIR}/bin:${INFORMIXDIR}/lib:${INFORMIXDIR}/lib/esql:${PATH}
+ENV LD_LIBRARY_PATH ${INFORMIXDIR}/lib:${INFORMIXDIR}/lib/esql:${INFORMIXDIR}/lib/cli
+ENV INFORMIXSQLHOSTS /opt/ibm/informix/etc/sqlhosts.informixoltp_tcp
+ENV USER root
+ENV LICENSE accept
+
+RUN ln -s /usr/bin/python2.7 /usr/bin/python
+RUN echo "sqlexec 2021/tcp" >> /etc/services
+
+COPY . /app
+
+WORKDIR /app
+RUN rm -rf node_modules && npm install --unsafe-perm
+
+ENTRYPOINT [ "npm" ]

docker/api.env

@@ -0,0 +1,3 @@
+KAFKA_URL=192.168.31.8:9092
+INFORMIX_HOST=192.168.31.8
+V5_CHALLENGE_TYPE_API_URL=http://192.168.31.8:4000/v5/challengeTypes

docker/docker-compose.yml

@@ -0,0 +1,12 @@
+version: '3'
+services:
+  legacy-challenge-processor:
+    image: legacy-challenge-processor:latest
+    container_name: legacy-challenge-processor
+    build:
+      context: ../
+      dockerfile: docker/Dockerfile
+    env_file:
+      - api.env
+    network_mode: "host"
+    command: run start