[$100] Add "duration" field to Job and ResourceBooking models

[maxceem]

• Add duration field to Job and ResourceBooking models. This field would define the duration in weeks.
• This field is optional and should contain an integer value equal or greater than 1 - add Joi validation.
• As we already have startDate and endDate fields we should keep all of these 3 fields consistent. If 2 of these fields are defined, we should automatically calculate the third one.
  • start + duration => calculate end
  • end + duration => calculate start
  • start + end => calculate duration (we should validate that calculated duration is more than 1 and it's an integer value or throw validation error).
  • start + end + duration => check if (end - start = duration) and if no, then throw validation error because data is inconsistent.
  • please, create some reusable logic for this, could it be done on the Model level if it makes sense so we never forget to add it?
• Update Swagger and Postman
• Update ES Processor
• Create migration script
• Mention which requests to call to update ES mapping

[maxceem]

Challenge https://www.topcoder.com/challenges/a902c186-691e-4353-b958-24f65731959c has been created for this ticket.

This is an automated message for maxceem via Topcoder X

[maxceem]

@imcaizheng this is ready for pickup.

[maxceem]

Challenge https://www.topcoder.com/challenges/a902c186-691e-4353-b958-24f65731959c has been assigned to aaron2017.

This is an automated message for maxceem via Topcoder X

[imcaizheng]

We could add a hook to do the calculation before saving(create/update) a record on the model level. The only thing should be aware of is the result need also being saved in ES.

Currently when an entity is going to be created it is saved in DB and ES independently. We could slightly modify the logic so that we have entity => save in DB = (result from DB) => save in ES

For example, https://github.com/topcoder-platform/taas-apis/blob/dev/src/services/JobService.js#L157-L158 can be changed to as below:

  const created = await Job.create(job)
  await helper.postEvent(config.TAAS_JOB_CREATE_TOPIC, created)

The following places need to be updated as well:
https://github.com/topcoder-platform/taas-apis/blob/dev/src/services/ResourceBookingService.js#L110-L111
https://github.com/topcoder-platform/taas-apis/blob/dev/src/services/JobCandidateService.js#L97-L98

[maxceem]

@imcaizheng Good catch about ES. Now putting this logic into the model sounds not like the best idea. I guess it would be very easy to forget about this detail and in the future, someone can just send raw data to ES without using Model methods.

So I guess it would be better just to create a reusable method and apply it explicitly inside each of the endpoints create/update. So anyone would be aware that we have such logic.

https://github.com/topcoder-platform/taas-apis/blob/dev/src/services/JobCandidateService.js#L97-L98

Just in case, we wouldn't add "duration" for the Job Candidates.

Also, I did a few updates to the tasks:

• duration field should be in weeks not in months
• duration value can only be 1 or more (integer)
• if we pass start + end dates and we calculate duration automatically, then we should validate that calculated duration is more than 1 and it's an integer value or throw a validation error

[imcaizheng]

One more question, to update existing data via migration script, in the case a record have a duration calculated to be less than 1 week, we could

1. silently ignore it
2. modify its end date to one week after start date

@maxceem which one is preferred, or any other idea?

[maxceem]

@imcaizheng If it requires some essential efforts, we can ignore it, as we have only demo data at the moment. But if it's relatively easy to achieve then let's follow 2 to make the demo data consistent for easier development.

[imcaizheng]

Sorry for the delay. Here are the PR #115 and topcoder-platform/taas-es-processor#16

Please run the following requests to update existing ES mappings with the new duration field:

PUT /job/_mapping

{
  "properties": {
    "duration": {
      "type": "integer"
    }
  }
}

PUT /resource_booking/_mapping

{
  "properties": {
    "duration": {
      "type": "integer"
    }
  }
}

Here is an example operating on local ES service, with the curl command:

> curl -X PUT localhost:9200/job/_mapping \
  -H 'Content-Type: application/json' \
  -d '{
    "properties": {
      "duration": {
        "type": "integer"
      }
    }
  }'
{"acknowledged":true}

> curl -X PUT localhost:9200/resource_booking/_mapping \
  -H 'Content-Type: application/json' \
  -d '{
    "properties": {
      "duration": {
        "type": "integer"
      }
    }
  }'
{"acknowledged":true}

Note on the migration script, since I didn't find an easy way to update existing records via a migration script, to update existing data we may need to create a temporary script to do the job.

[maxceem]

@nkumar-topcoder I've tested solution for this task locally and it works good.

There is one thing that would need your confirmation. At the moment we've added duration field, but we didn't update existent data to populate this field. So for all existent data duration would have value null. And current data in DEV for sure doesn't follow the new restriction of having integer number of weeks between start and end dates.
As we don't have any prod data yet, we didn't put efforts into creating a script for this, but if you think we need it, please, let us know.

If you think it's ok like this, then we would need your help to deploy this change to DEV.

Steps to deploy:

1. Merge PR Add duration field #116
2. Run migration script npm run migrate
3. Update ES mapping using request in the comment above [$100] Add "duration" field to Job and ResourceBooking models #113 (comment)