Bundling support for email notifications and email settings

[gondzo]

Add support for bundling emails end email settings.

Notifications settings endpoints should be updated to save preferences for emails (same as what we are doing for web preferences now). Additionally, there should be a bundle setting for emails with options: no bundling, 10 minutes, 1 hour, 24 hours, weekly (these should be configurable levels).
When a notification is created, email should not be sent right away, but we should first check email settings and send it immediately if email bundling is off. Otherwise, the email for that notification will be sent later.

All bundle intervals should be handled via scheduled jobs. At each interval check for new notifications that require emails to be sent and send bundle email containing all notifications created since last sending the email.

@vikasrohit should we create a new email event in kafka and new template in sendgrid for this one?

[vikasrohit]

Otherwise, the email for that notification will be sent later.

Is there a good way to log such emails? I mean apart from logging in log files, can we have them in database table where we can mark their status to be something which says 'not sent immediately, would be sent in bundle of 24 hours'?

should we create a new email event in kafka and new template in sendgrid for this one?

Yes, definitely. We need new template in sendgrid and I guess new topic in kafka can not be avoided as well.

[vikasrohit]

notifications.action.email.connect.project.bundled is created in kafka.

[maxceem]

1. Should we be able to config bundling time per notifications type? or for all notifications types there is same bundling time?

2. Should we bundle notifications of all types to one email? Or we bundle separately notifications of different types?

3. Do you know any good module for scheduling jobs for node?
   As I understand we need our job to be persistent between service relaunches, so the scheduler has to be able to store jobs somewhere. For example, agenda stores in the MongoDB, though I would prefer not to introduce another DB for this service as we already use PostgreSQL.
   I'll research now, but if you have something to advise it would be great.

[maxceem]

4. For example we bundle notifications by 24 hours. Should we count 24h from some fixed time, like all notifications which comes from 00.00 till 23:59. Or from the first notification time occure, so if we got some first notification at 3:47am, than we wait 24h since than, and after we send the bundle at 3:47am the next day.

[vikasrohit]

Great questions @maxceem.! 1 and 2 we should group config and bundle per notification server e.g. in this case connectNotificationServer. 3. We are thinking of using node cron module for scheduling and simple storage of all job runs in existing posters db. Or you can try quartz with posters. Still I would prefer simple node cron unless there is a specific reason. 4.I would come back for this. @gondzo what do you think? On May 8, 2018 4:33 PM, "maxceem" <notifications@github.com> wrote: 1. For example we bundle notifications by 24 hours. Should we count 24h from some fixed time, like all notifications which comes from 00.00 till 23:59. Or from the first notification time occure, so if we got some first notification at 3:47am, than we wait 24h since than, and after we send the bundle at 3:47am the next day. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#40 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACTj4Dn3zqw1xXe1jJ67KgsUMuSWhiy4ks5twXtogaJpZM4TydfF> .

[maxceem]

5. Is it possible to obtain long last admin token for dev environment to configure TC_ADMIN_TOKEN?
   The one that provided in the docs is expired. And if I create by myself they expire in few minutes and I have to create again.

[maxceem]

6. As I understand from 1 and 2 in one bundled notification could be notifications about different posts and projects. How in this case should we build replyTo email address? So far it's created for one particular topic.

    const replyTo = `${config.REPLY_EMAIL_PREFIX}+${topicId}/${token}@${config.REPLY_EMAIL_DOMAIN}`;

[maxceem]

7. SImilar to previous one. What do you think we should put to email from name? Currently we put the handle of the user who triggered the event. For example user who added some post. In bundled email could be posts from various users.

[vikasrohit]

How in this case should we build replyTo email address?

We don't want users replying bundled emails. Please use noreply AT topcoder DOT com.

What do you think we should put to email from name?

We don't need the name here. Just use the same noreply AT topcoder DOT com

[maxceem]

8. Do you have any preferences what should be a bundled event payload? So far I'm thinking to have values like recipients, replyTo, cc, from, categories only one per events as it is. And send data as arrays of data of all bundled events:

   

   Do you have any requirements here?

[gondzo]

4. I think just having fixed intervals should be good enough, without taking into account when the first notification was created. For daily and weekly intervals, it would be good to configure time of day that we send the emails (so we don't end up sending all emails at midnight)

[gondzo]

8. I'm not really sure here - we use sendgrid templating to populate the email using that data field, but looks like they don't yet support loops in templates How to iterate array of objects in Sendgrid template? sendgrid/sendgrid-nodejs#221

As I see it, we can either

• create the final HTML in notifications service (containing the entire list of emails), or
• add loop support to our email service (manually unroll sendgrid templates), for example as done here How to iterate array of objects in Sendgrid template? sendgrid/sendgrid-nodejs#221 (comment)

@vikasrohit thoughts?

[maxceem]

Is there a good way to log such emails? I mean apart from logging in log files, can we have them in database table where we can mark their status to be something which says 'not sent immediately, would be sent in bundle of 24 hours'?

@vikasrohit Regarding this. I see implementation in the next way:

• each time event which has to be bundled comes, I ADD it to the DB
• scheduled job runs every 10 minutes, for example
  • scheduled job retrieves all events, which has to be run in 10 minutes from DB
  • job bundles them per user
  • job send bundled events to bus API
  • job removes all sent events from DB

So we don't have records in the DB per email, but per event which will be bundled before sending.
Does this way provide the logging level which you were looking for?