ES/DB compare script

[maxceem]

@vikasrohit The script for comparing data in ES and DB is ready.

It would compare all metadata objects and projects with related objects like invites, members, attachments, phases, products, timelines and milestones.

See README on how to use it.

I propose for the first testing to run it on the DEV environment once to chech projects with ids from 8000 to 9000:

REPORT_S3_BUCKET=<S3 bucket name> PROJECT_START_ID=8000 PROJECT_END_ID=9000 npm run es-db-compare

This command requires that AWS credential env variables are already set: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

[vikasrohit]

Thanks @maxceem. PR looks good to me. I have couple of questions:

1. what are the dump json files? Are these used for unit testing only? Just wanted to see if they contain the real data from dev env or test data.
2. what is usage of insertDB/ESData scripts?

[maxceem]

1. what are the dump json files? Are these used for unit testing only? Just wanted to see if they contain the real data from dev env or test data.

Oh, yes indeed these files contain metadata from DEV. So we have to clear out some data first from them before merging.

2. what is usage of insertDB/ESData scripts?

These scripts together with DB and ES dumps are only necessary for verification of the script. So work on this script locally, we have to create some test data in ES and DB which has some mismatches on purpose. So after we can test how the script works.

[vikasrohit]

Oh, yes indeed these files contain metadata from DEV. So we have to clear out some data first from them before merging.

You are doing that, right? And again to be on same page, these files are only needed for verification just like insertXXData scripts?

[maxceem]

@vikasrohit

You are doing that, right?

This is done. Prices are random now so this PR is ready for merging.

And again to be on same page, these files are only needed for verification just like insertXXData scripts?

RIght, everything in verification folder including ES/DB dump JSONs and insertXXData scripts are only for local verification of the script. It has prepared data to be inserted in DB and ES which already has some mismatches.

[maxceem]

I would suggest to Squash commits for this PR, and after we would remove feature/es-db-compare which has real data in commits history.