[$50] Always pass data to Kafka Events from DB

[maxceem]

At the moment when we send data to Kafka using helper.postEvent we usually send raw data which were received from the user:

await job.update(data) // save raw data form the user
await helper.postEvent(config.TAAS_JOB_UPDATE_TOPIC, { id, ...data }, { oldValue: oldValue }) // send raw data to the Kafka

This leads to several inconsistencies:

• When we are creating a record there are no values for updatedAt and updatedBy. So object is send to Kafka without these values and they are indexed with updatedBy = undefined and updatedBy = undefined. Though in the DB these values are stored as null. This leads to the inconsistency between data in DB and ES. While generally we shouldn't rely on the difference between null and undefined sometimes we do. Also, our commands to reindex data from DB to ES would index these fields with null because the data would be taken from the DB.
• If if PATCH data, then we can pass just a few fields and as a result payload for the Kafka Event would have only a few fields. In my opinion, it would be better for consistency if Kafka events always have the whole document in the payload.

So to make data consistent between DB and ES, we have to always pass to the Kafka Events data from DB, and the code above should be replaced with something like:

const updatedJob = await job.update(data) // save raw data form the user
await helper.postEvent(config.TAAS_JOB_UPDATE_TOPIC, updatedJob.toJSON(), { oldValue: oldValue }) // send raw data to the Kafka

Another benefit of such approach is that if we implement some logic at the model level it would automatically reflected in the event payloads, though for now I prefer not to rely on it. So for now we still have to implement custom logic explicitly like here https://github.com/topcoder-platform/taas-apis/issues and not to include it inside the models.

• This has to be done for all the places where we pass data to the Kafka events during creating and updating
• As we use method helper.postEvent to send a message not only to Kafka, but also to our internal event handler, so this would make event bodies unified there too, which seems good too

[nkumar-topcoder]

@maxceem You mean trigger event from postgres post any DB action and then update ES ?
Let me know if this is what you are thinking.

• https://gist.github.com/colophonemes/9701b906c5be572a40a84b08f4d2fa4e
• Similar thing, we have implemented for sync from postgres to informix.
• https://github.com/topcoder-platform/postgres-ifx-processer/blob/9c283cdf4ac4f9a80ab8abeb95823985ab417e29/pg-identity-func-trig-seq.sql#L143
• Once, above code is trigger, this will listen here
• https://github.com/topcoder-platform/postgres-ifx-processer/blob/9c283cdf4ac4f9a80ab8abeb95823985ab417e29/src/producer.js#L26
• And then post to kafka at line 39 same js file.

We can connect and discuss tomorrow.

[maxceem]

@nkumar-topcoder you have a great idea for extension of my intention, though my initial there is less ambitious.

What I would like to solve are cases like this:

Inconsistency during POST

1. User creates a new job, but sending all required fields:

   {
     "projectId": {{projectId}},
     "numPositions": 13,
     "skills": [
       "23e00d92-207a-4b5b-b3c9-4c5662644941",
       "7d076384-ccf6-4e43-a45d-1b24b1e624aa",
       "cbac57a3-7180-4316-8769-73af64893158",
       "a2b4bc11-c641-4a19-9eb7-33980378f82e"
     ],
     "title": "Dummy title - at most 64 characters"
   }

   We populate some default values and send Job to ES as:

    {
     "projectId": 111,
     "numPositions": 13,
     "skills": [
         "23e00d92-207a-4b5b-b3c9-4c5662644941",
         "7d076384-ccf6-4e43-a45d-1b24b1e624aa",
         "cbac57a3-7180-4316-8769-73af64893158",
         "a2b4bc11-c641-4a19-9eb7-33980378f82e"
     ],
     "title": "Dummy title - at most 64 characters",
     "status": "sourcing",
     "id": "57bc04fe-0343-495c-9478-6cbee2b2a304",
     "createdAt": "2021-02-05T08:52:13.036Z",
     "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c"
   }

   So all the missing fields are not added to the ES at all. While in DB all of the missing fields are stored as null. If now we re-index this job, then we would get this record from DB with all the null values, and after reindexing the same records would look like this in ES:

       {
     "id": "57bc04fe-0343-495c-9478-6cbee2b2a304",
     "projectId": 111,
     "externalId": null,
     "description": null,
     "title": "Dummy title - at most 64 characters",
     "startDate": null,
     "endDate": null,
     "numPositions": 13,
     "resourceType": null,
     "rateType": null,
     "workload": null,
     "skills": [
         "23e00d92-207a-4b5b-b3c9-4c5662644941",
         "7d076384-ccf6-4e43-a45d-1b24b1e624aa",
         "cbac57a3-7180-4316-8769-73af64893158",
         "a2b4bc11-c641-4a19-9eb7-33980378f82e"
     ],
     "status": "sourcing",
     "createdAt": "2021-02-05T08:52:13.036Z",
     "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
     "updatedAt": null,
     "updatedBy": null
   }

   This leads to inconsistency:

   • Values which are not defined are always stored as null in DB. While in ES sometimes undefined values could be just ignored, and sometimes could be stored as null. This might affect search queries or it can affect client-side code, if client-side code happens to check for null or undefined.
   • This issue happens, because when we create a new Job we are sending event to Kafka using data provided by the user. While if we reindex we use data from DB.
   • So the solution is when user creates a Job, we don't send user input directly to Kafka, but we first save it to the DB, and get values from the DB and send it to Kafka. This way we would always send to Kafka values from DB, so data in ES would be always consistent and the same like in DB.

Inconsistency during PATCH or PUT

There is another type of inconsistency. During PATCH request user can send just one field, for example { title: "updated" }.

As we are sending to the Kafka user input, and not value form DB, then we would send Kafka event with the next payload:

{
  title: "updated",
  updatedAt: "",
  updatedBy: ""
}

But if a user sends PUT request, then the user is required to provide all the required values, and payload would contain more values.

This could be treated as a benefit as we can immediately see which values very provided by user initially. But for me it looks more like a disadvantage, in my opinion, it would be better if the Kafka event with type update always has all the fields of the updated document.

So again, the solution would be, instead of sending data provided by user to the ES, we can updated data in DB, and then get data from DB and send it to Kafka. This way even payload would have the whole updated document and it would equivalent to what we have in DB.

[maxceem]

Challenge https://www.topcoder.com/challenges/c7c30b20-ace8-40b3-88f9-47ff0d7d0e38 has been created for this ticket.

This is an automated message for maxceem via Topcoder X

[maxceem]

@imcaizheng this is ready for pickup.

The fix by itself is trivial. But please, verify that this fix doesn't break any of the affected requests (including team requests). If you see any drawbacks, please, let me know.

[maxceem]

Challenge https://www.topcoder.com/challenges/c7c30b20-ace8-40b3-88f9-47ff0d7d0e38 has been assigned to aaron2017.

This is an automated message for maxceem via Topcoder X

[imcaizheng]

PR created #136 and topcoder-platform/taas-es-processor#23