Merge pull request #52 from topcoder-platform/dev

Prod - M2M and Other changes

Author: sachin-maheshwari

.circleci/config.yml

@@ -82,7 +82,7 @@ workflows:
       - "build-dev":
           filters:
             branches:
-              only: [dev, 'feature/notification-email-improvements']
+              only: [dev]
       - "build-prod":
           filters:
             branches:

.gitignore

@@ -1,5 +1,6 @@
 .idea
 node_modules
 *.log
+log.txt
 .DS_Store
 dist

README.md

@@ -6,48 +6,74 @@
 - Heroku Toolbelt https://toolbelt.heroku.com
 - git
 - PostgreSQL 9.5
- 
+
 
 ## Configuration
+
+### Notification server
 Configuration for the notification server is at `config/default.js`.
 The following parameters can be set in config files or in env variables:
-- LOG_LEVEL: the log level
-- PORT: the notification server port
-- authSecret: TC auth secret
-- authDomain: TC auth domain
-- validIssuers: TC auth valid issuers
-- jwksUri: TC auth JWKS URI
-- DATABASE_URL: URI to PostgreSQL database
-- DATABASE_OPTIONS: database connection options
-- KAFKA_URL: comma separated Kafka hosts
-- KAFKA_TOPIC_IGNORE_PREFIX: ignore this prefix for topics in the Kafka
-- KAFKA_GROUP_ID: Kafka consumer group id
-- KAFKA_CLIENT_CERT: Kafka connection certificate, optional;
-    if not provided, then SSL connection is not used, direct insecure connection is used;
-    if provided, it can be either path to certificate file or certificate content
-- KAFKA_CLIENT_CERT_KEY: Kafka connection private key, optional;
-    if not provided, then SSL connection is not used, direct insecure connection is used;
-    if provided, it can be either path to private key file or private key content
-- BUS_API_BASE_URL: Bus API url
-- BUS_API_AUTH_TOKEN: Bus API auth token
-- REPLY_EMAIL_PREFIX: prefix of the genereated reply email address
-- REPLY_EMAIL_DOMAIN: email domain
-- DEFAULT_REPLY_EMAIL: default reply to email address, for example no-reply@topcoder.com
-- MENTION_EMAIL: recipient email used for email.project.post.mention event
-
+- **General**
+  - `LOG_LEVEL`: the log level
+  - `PORT`: the notification server port
+  - `DATABASE_URL`: URI to PostgreSQL database
+  - `DATABASE_OPTIONS`: database connection options
+- **JWT authentication**
+  - `AUTH_SECRET`: TC auth secret
+  - `VALID_ISSUERS`: TC auth valid issuers
+  - `JWKS_URI`: TC auth JWKS URI (need only for local deployment)
+- **KAFKA**
+  - `KAFKA_URL`: comma separated Kafka hosts
+  - `KAFKA_TOPIC_IGNORE_PREFIX`: ignore this prefix for topics in the Kafka
+  - `KAFKA_GROUP_ID`: Kafka consumer group id
+  - `KAFKA_CLIENT_CERT`: Kafka connection certificate, optional;
+      if not provided, then SSL connection is not used, direct insecure connection is used;
+      if provided, it can be either path to certificate file or certificate content
+  - `KAFKA_CLIENT_CERT_KEY`: Kafka connection private key, optional;
+      if not provided, then SSL connection is not used, direct insecure connection is used;
+      if provided, it can be either path to private key file or private key content
+- **Topcoder API**
+  - `TC_API_V5_BASE_URL`: the TopCoder API V5 base URL
+- **Notifications API**
+  - `API_CONTEXT_PATH`: path to serve API on
+- **Machine to machine auth0 token**
+  - `AUTH0_URL`: auth0 URL
+  - `AUTH0_AUDIENCE`: auth0 audience
+  - `TOKEN_CACHE_TIME`: time period of the cached token
+  - `AUTH0_CLIENT_ID`: auth0 client id
+  - `AUTH0_CLIENT_SECRET`: auth0 client secret
+
+### Connect notification server
 Configuration for the connect notification server is at `connect/config.js`.
 The following parameters can be set in config files or in env variables:
-- TC_API_V3_BASE_URL: the TopCoder API V3 base URL
-- TC_API_V4_BASE_URL: the TopCoder API V4 base URL
-- TC_ADMIN_TOKEN: the admin token to access TopCoder API - same for V3 and V4<br>
-  Also it has probably temporary variables of TopCoder role ids for 'Connect Manager', 'Connect Copilot' and 'administrator':
-- CONNECT_MANAGER_ROLE_ID: 8,
-- CONNECT_COPILOT_ROLE_ID: 4,
-- ADMINISTRATOR_ROLE_ID: 1<br>
-  Provided values are for development backend. For production backend they may be different.
-  These variables are currently being used to retrieve above role members using API V3 `/roles` endpoint. As soon as this endpoint is replaced with more suitable one, these variables has to be removed if no need anymore.
-- TCWEBSERVICE_ID - id of the BOT user which creates post with various events in discussions
-
+- **Topcoder API**
+  - `TC_API_V3_BASE_URL`: the TopCoder API V3 base URL
+  - `TC_API_V4_BASE_URL`: the TopCoder API V4 base URL
+  - `MESSAGE_API_BASE_URL`: the TopCoder message service API base URL
+- **Topcder specific**<br>
+    Also it has probably temporary variables of TopCoder role ids for 'Connect Manager', 'Connect Copilot' and 'administrator':
+  - `CONNECT_MANAGER_ROLE_ID`: 8,
+  - `CONNECT_COPILOT_ROLE_ID`: 4,
+  - `ADMINISTRATOR_ROLE_ID`: 1<br>
+    Provided values are for development backend. For production backend they may be different.
+    These variables are currently being used to retrieve above role members using API V3 `/roles` endpoint. As soon as this endpoint is replaced with more suitable one, these variables has to be removed if no need anymore.
+  - `TCWEBSERVICE_ID` - id of the BOT user which creates post with various events in discussions
+- **Machine to machine auth0 token**
+  - `AUTH0_URL`: auth0 URL
+  - `AUTH0_AUDIENCE`: auth0 audience
+  - `TOKEN_CACHE_TIME`: time period of the cached token
+  - `AUTH0_CLIENT_ID`: auth0 client id
+  - `AUTH0_CLIENT_SECRET`: auth0 client secret
+- **Email notification service**
+  - `ENV`: environment variable (used to generate reply emails)
+  - `AUTH_SECRET`: auth secret (used to sign reply emails)
+  - `ENABLE_EMAILS`: if email service has to be enabled
+  - `ENABLE_DEV_MODE`: send all emails to the `DEV_MODE_EMAIL` email address
+  - `DEV_MODE_EMAIL`: address to send all email when `ENABLE_DEV_MODE` is enabled
+  - `MENTION_EMAIL`: recipient email used for `notifications.action.email.connect.project.post.mention` event
+  - `REPLY_EMAIL_PREFIX`: prefix of the genereated reply email address
+  - `REPLY_EMAIL_DOMAIN`: email domain
+  - `DEFAULT_REPLY_EMAIL`: default reply to email address, for example no-reply@topcoder.com
 
 Note that the above two configuration are separate because the common notification server config
 will be deployed to a NPM package, the connect notification server will use that NPM package,
@@ -67,31 +93,19 @@ so we can run `node test/token 305384` to generate a token to manage notificatio
 The generated token is already configured in the Postman notification server API environment TOKEN variable.
 You may reuse it during review.
 
-
-## TC API Admin Token
-
-An admin token is needed to access TC API. This is already configured Postman notification
-server API environment TC_ADMIN_TOKEN variable.
-In case it expires, you may get a new token in this way:
-
-- use Chrome to browse connect.topcoder-dev.com
-- open developer tools, click the Network tab
-- log in with suser1 / Topcoder123, or mess / appirio123
-- once logged in, open some project, for example https://connect.topcoder-dev.com/projects/1936 and in the network inspector
-  look for the call to the project api and get the token from the auth header, see
-  http://pokit.org/get/img/68cdd34f3d205d6d9bd8bddb07bdc216.jpg
-
-
 ## Local deployment
 - for local development environment you can set variables as following:
-  - `authSecret`, `authDomain`, `validIssuers` can get from [tc-project-service config](https://github.com/topcoder-platform/tc-project-service/blob/dev/config/default.json)
+  - `AUTH_SECRET`,`VALID_ISSUERS` can get from [tc-project-service config](https://github.com/topcoder-platform/tc-project-service/blob/dev/config/default.json)
   - `PORT=4000` because **connect-app** call this port by default
-  - `jwksUri` - any
   - `KAFKA_TOPIC_IGNORE_PREFIX=joan-26673.` (with point at the end)
   - `TC_API_V4_BASE_URL=https://api.topcoder-dev.com/v4`
   - `TC_API_V3_BASE_URL=https://api.topcoder-dev.com/v3`
-  - `TC_ADMIN_TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyb2xlcyI6WyJUb3Bjb2RlciBVc2VyIiwiYWRtaW5pc3RyYXRvciJdLCJpc3MiOiJodHRwczovL2FwaS50b3Bjb2Rlci1kZXYuY29tIiwiaGFuZGxlIjoic3VzZXIxIiwiZXhwIjoxNTEzNDAxMjU4LCJ1c2VySWQiOiI0MDE1MzkzOCIsImlhdCI6MTUwOTYzNzYzOSwiZW1haWwiOiJtdHdvbWV5QGJlYWtzdGFyLmNvbSIsImp0aSI6IjIzZTE2YjA2LWM1NGItNDNkNS1iY2E2LTg0ZGJiN2JiNDA0NyJ9.REds35fdBvY7CMDGGFyT_tOD7DxGimFfVzIyEy9YA0Y` or follow section **TC API Admin Token** to obtain a new one if expired
   - `KAFKA_URL`, `KAFKA_CLIENT_CERT` and `KAFKA_CLIENT_CERT_KEY` get from [tc-bus-api readme](https://github.com/topcoder-platform/tc-bus-api/tree/dev)
+- if you are willing to use notifications API which is hosted by the notifications server locally, you will need to use some patched `tc-core-library-js` module, which skips verification of user token. Because we don't know Topcoder `AUTH_SECRET` locally. So you can install this fork:
+  ```
+  npm i https://github.com/maxceem/tc-core-library-js/tree/skip-validation
+  ```
+  **WARNING** do not push package.json with this dependency as it skips users token validation.
 - start local PostgreSQL db, create an empty database, update the config/default.js DATABASE_URL param to point to the db
 - install dependencies `npm i`
 - run code lint check `npm run lint`
@@ -130,5 +144,5 @@ In case it expires, you may get a new token in this way:
 ## Swagger
 
 Swagger API definition is provided at `docs/swagger_api.yaml`,
-you may check it at `http://editor.swagger.io`. 
+you may check it at `http://editor.swagger.io`.
 

config/default.js

@@ -2,12 +2,8 @@
  * The configuration file.
  */
 module.exports = {
-  ENV: process.env.ENV,
   LOG_LEVEL: process.env.LOG_LEVEL,
   PORT: process.env.PORT,
-  authSecret: process.env.authSecret,
-  authDomain: process.env.authDomain,
-  jwksUri: process.env.jwksUri,
   DATABASE_URL: process.env.DATABASE_URL,
   DATABASE_OPTIONS: {
     dialect: 'postgres',
@@ -21,27 +17,28 @@ module.exports = {
     },
   },
 
-  validIssuers: process.env.validIssuers ? process.env.validIssuers.replace(/\\"/g, '') : null,
+  AUTH_SECRET: process.env.authSecret,
+  VALID_ISSUERS: process.env.validIssuers ? process.env.validIssuers.replace(/\\"/g, '') : null,
+  // keep it here for dev purposes, it's only needed by modified version of tc-core-library-js
+  // which skips token validation when locally deployed
+
   KAFKA_URL: process.env.KAFKA_URL,
   KAFKA_TOPIC_IGNORE_PREFIX: process.env.KAFKA_TOPIC_IGNORE_PREFIX,
   KAFKA_GROUP_ID: process.env.KAFKA_GROUP_ID,
   KAFKA_CLIENT_CERT: process.env.KAFKA_CLIENT_CERT ? process.env.KAFKA_CLIENT_CERT.replace('\\n', '\n') : null,
   KAFKA_CLIENT_CERT_KEY: process.env.KAFKA_CLIENT_CERT_KEY ?
     process.env.KAFKA_CLIENT_CERT_KEY.replace('\\n', '\n') : null,
 
-  BUS_API_AUTH_TOKEN: process.env.BUS_API_AUTH_TOKEN,
-  MENTION_EMAIL: process.env.MENTION_EMAIL,
-  REPLY_EMAIL_PREFIX: process.env.REPLY_EMAIL_PREFIX,
-  REPLY_EMAIL_DOMAIN: process.env.REPLY_EMAIL_DOMAIN,
-
-  TC_ADMIN_TOKEN: process.env.TC_ADMIN_TOKEN,
-  TC_API_BASE_URL: process.env.TC_API_BASE_URL || 'https://api.topcoder-dev.com',
-  TC_API_V3_BASE_URL: process.env.TC_API_V3_BASE_URL || 'https://api.topcoder-dev.com/v3',
-  TC_API_V4_BASE_URL: process.env.TC_API_V4_BASE_URL || 'https://api.topcoder-dev.com/v4',
   TC_API_V5_BASE_URL: process.env.TC_API_V5_BASE_URL || 'https://api.topcoder-dev.com/v5',
-  MESSAGE_API_BASE_URL: process.env.MESSAGE_API_BASE_URL || 'https://api.topcoder-dev.com/v4',
-  ENABLE_EMAILS: process.env.ENABLE_EMAILS || true,
-  ENABLE_DEV_MODE: process.env.ENABLE_DEV_MODE || true,
-  DEV_MODE_EMAIL: process.env.DEV_MODE_EMAIL,
   API_CONTEXT_PATH: process.env.API_CONTEXT_PATH || '/v5/notifications',
+
+  // Configuration for generating machine to machine auth0 token.
+  // The token will be used for calling another internal API.
+  AUTH0_URL: process.env.AUTH0_URL,
+  AUTH0_AUDIENCE: process.env.AUTH0_AUDIENCE,
+  // The token will be cached.
+  // We define the time period of the cached token.
+  TOKEN_CACHE_TIME: process.env.TOKEN_CACHE_TIME || 86400000,
+  AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID,
+  AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET,
 };

connect/config.js

@@ -3,11 +3,10 @@
  */
 
 module.exports = {
+  // TC API related variables
   TC_API_V3_BASE_URL: process.env.TC_API_V3_BASE_URL || 'https://api.topcoder-dev.com/v3',
   TC_API_V4_BASE_URL: process.env.TC_API_V4_BASE_URL || 'https://api.topcoder-dev.com/v4',
-  MESSAGE_API_BASE_URL: process.env.MESSAGE_API_BASE_URL || 'https://api.topcoder-dev.com/v4',
-  // eslint-disable-next-line max-len
-  TC_ADMIN_TOKEN: process.env.TC_ADMIN_TOKEN,
+  MESSAGE_API_BASE_URL: process.env.MESSAGE_API_BASE_URL || 'https://api.topcoder-dev.com/v5',
 
   // Probably temporary variables for TopCoder role ids for 'Connect Manager', 'Connect Copilot' and 'administrator'
   // These are values for development backend. For production backend they may be different.
@@ -16,7 +15,28 @@ module.exports = {
   CONNECT_MANAGER_ROLE_ID: 8,
   CONNECT_COPILOT_ROLE_ID: 4,
   ADMINISTRATOR_ROLE_ID: 1,
-
   // id of the BOT user which creates post with various events in discussions
   TCWEBSERVICE_ID: process.env.TCWEBSERVICE_ID || '22838965',
+
+  // Configuration for generating machine to machine auth0 token.
+  // The token will be used for calling another internal API.
+  AUTH0_URL: process.env.AUTH0_URL,
+  AUTH0_AUDIENCE: process.env.AUTH0_AUDIENCE,
+  // The token will be cached.
+  // We define the time period of the cached token.
+  TOKEN_CACHE_TIME: process.env.TOKEN_CACHE_TIME || 86400000,
+  AUTH0_CLIENT_ID: process.env.AUTH0_CLIENT_ID,
+  AUTH0_CLIENT_SECRET: process.env.AUTH0_CLIENT_SECRET,
+
+  // email notification service related variables
+  ENV: process.env.ENV,
+  AUTH_SECRET: process.env.authSecret,
+  ENABLE_EMAILS: process.env.ENABLE_EMAILS || true,
+  ENABLE_DEV_MODE: process.env.ENABLE_DEV_MODE || true,
+  DEV_MODE_EMAIL: process.env.DEV_MODE_EMAIL,
+  MENTION_EMAIL: process.env.MENTION_EMAIL,
+  REPLY_EMAIL_PREFIX: process.env.REPLY_EMAIL_PREFIX,
+  REPLY_EMAIL_DOMAIN: process.env.REPLY_EMAIL_DOMAIN,
+  REPLY_EMAIL_FROM: process.env.REPLY_EMAIL_FROM,
+  DEFAULT_REPLY_EMAIL: process.env.DEFAULT_REPLY_EMAIL,
 };

connect/connectNotificationServer.js

@@ -9,11 +9,12 @@ const config = require('./config');
 const notificationServer = require('../index');
 const _ = require('lodash');
 const service = require('./service');
-const { BUS_API_EVENT } = require('../src/constants')
+const { BUS_API_EVENT } = require('./constants');
 const EVENTS = require('./events-config').EVENTS;
 const TOPCODER_ROLE_RULES = require('./events-config').TOPCODER_ROLE_RULES;
 const PROJECT_ROLE_RULES = require('./events-config').PROJECT_ROLE_RULES;
 const PROJECT_ROLE_OWNER = require('./events-config').PROJECT_ROLE_OWNER;
+const emailNotificationServiceHandler = require('./notificationServices/email').handler;
 
 /**
  * Get TopCoder members notifications
@@ -62,14 +63,13 @@ const getTopCoderMembersNotifications = (eventConfig) => {
  * @return {Promise}            resolves to a list of notifications
  */
 const getNotificationsForMentionedUser = (eventConfig, content) => {
-  if (!eventConfig.toMentionedUsers) {
+  if (!eventConfig.toMentionedUsers || !content) {
     return Promise.resolve([]);
   }
 
   let notifications = [];
   // eslint-disable-next-line
-  const regexUserHandle = /title=\"@([a-zA-Z0-9-_.{}\[\]]+)\"|\[.*\]\(.*\"\@(.*)\"\)/g;
-  const handles = [];
+  const regexUserHandle = /title=\"@([a-zA-Z0-9-_.{}\[\]]+)\"|\[.*?\]\(.*?\"\@(.*?)\"\)/g;
   let matches = regexUserHandle.exec(content);
   while (matches) {
     const handle = matches[1] ? matches[1].toString() : matches[2].toString();
@@ -81,18 +81,22 @@ const getNotificationsForMentionedUser = (eventConfig, content) => {
       },
     });
     matches = regexUserHandle.exec(content);
-    handles.push(handle);
   }
   // only one per userHandle
   notifications = _.uniqBy(notifications, 'userHandle');
 
   return new Promise((resolve) => {
-    service.getUsersByHandle(handles).then((users) => {
-      _.map(notifications, (notification) => {
-        notification.userId = _.find(users, { handle: notification.userHandle }).userId.toString();
+    const handles = _.map(notifications, 'userHandle');
+    if (handles.length > 0) {
+      service.getUsersByHandle(handles).then((users) => {
+        _.forEach(notifications, (notification) => {
+          notification.userId = _.find(users, { handle: notification.userHandle }).userId.toString();
+        });
+        resolve(notifications);
       });
-      resolve(notifications);
-    });
+    } else {
+      resolve([]);
+    }
   });
 };
 
@@ -243,6 +247,7 @@ const excludeNotifications = (notifications, eventConfig, message, data) => {
   return Promise.all([
     getNotificationsForTopicStarter(excludeEventConfig, message.topicId),
     getNotificationsForUserId(excludeEventConfig, message.userId),
+    getNotificationsForMentionedUser(eventConfig, message.postContent),
     getProjectMembersNotifications(excludeEventConfig, project),
     getTopCoderMembersNotifications(excludeEventConfig),
   ]).then((notificationsPerSource) => (
@@ -296,7 +301,7 @@ const handler = (topic, message, callback) => {
       //       - check that event has everything required or throw error
       getNotificationsForTopicStarter(eventConfig, message.topicId),
       getNotificationsForUserId(eventConfig, message.userId),
-      message.postContent ? getNotificationsForMentionedUser(eventConfig, message.postContent) : Promise.resolve([]),
+      getNotificationsForMentionedUser(eventConfig, message.postContent),
       getProjectMembersNotifications(eventConfig, project),
       getTopCoderMembersNotifications(eventConfig),
     ]).then((notificationsPerSource) => (
@@ -344,6 +349,11 @@ EVENTS.forEach(eventConfig => {
   notificationServer.addTopicHandler(eventConfig.type, handler);
 });
 
+// add notification service handlers
+if (config.ENABLE_EMAILS) {
+  notificationServer.addNotificationServiceHandler(emailNotificationServiceHandler);
+}
+
 // init database, it will clear and re-create all tables
 notificationServer
   .initDatabase()

src/constants.js renamed to connect/constants.js

@@ -1,17 +1,29 @@
 module.exports = {
+  // periods of time in cron format (node-cron)
+  SCHEDULED_EVENT_PERIOD: {
+    every10minutes: '*/10 * * * *',
+    hourly: '0 * * * *',
+    daily: '0 7 * * *', // every day at 7am
+    weekly: '0 7 * * 6', // every Saturday at 7am
+  },
+
+  // email service id for settings
+  SETTINGS_EMAIL_SERVICE_ID: 'email',
+
   BUS_API_EVENT: {
-    CONNECT : {
+    CONNECT: {
       TOPIC_CREATED: 'notifications.connect.project.topic.created',
       TOPIC_DELETED: 'notifications.connect.project.topic.deleted',
       POST_CREATED: 'notifications.connect.project.post.created',
       POST_UPDATED: 'notifications.connect.project.post.edited',
       POST_DELETED: 'notifications.connect.project.post.deleted',
       MENTIONED_IN_POST: 'notifications.connect.project.post.mention',
     },
-    EMAIL : {
+    EMAIL: {
       TOPIC_CREATED: 'notifications.action.email.connect.project.topic.created',
       POST_CREATED: 'notifications.action.email.connect.project.post.created',
       MENTIONED_IN_POST: 'notifications.action.email.connect.project.post.mention',
+      BUNDLED: 'notifications.action.email.connect.project.bundled',
     },
   },
 };