Improve setup guide/process for local developing

[maxceem]

Currently we several kinds of issues which raise during setting up the project service locally for development. Especially it's crucial for F2F challenges, where setup process may take the same or even more efforts than solving the actual issue.

We can improve setup process in several ways:

1. ✅ Create a separate step-by-step guide of how to run project service locally if we provided AUTH credentials for M2M authentication.

   • currently if we just project AUTH credentials, it wouldb't be used, we have to also adjust code to make use of it, something like this:

     

     Is there a better way?

2. ❌ no need anymore Create a separate step-by-step guide of how to run project service locally if we DIDN'T provide AUTH credentials for M2M authentication. So mock service should be used.

3. ❌ no need anymore as we have mock data not for all third-party endpoints used by project-service, we should either create missing endpints or maybe document somehow which functionality we support with mock data and which we don't. Is there any other way of how to make it clear what would work if we don't provide M2M credentials?

4. ✅ Guide how to setup Connect App together with project service locally.

   • where to update project service URL
   • how to bypass token validation

5. ✅ (done via Import and Export Data Commands #546) A script/command which would populate DB with test data.

   • this has been started here Local test data #102.
   • we should improve it so it would also create some test projects with all the necessary dependencies (if any)
   • I think instead of getting data directly form dev, we can have two kind of scripts:
     • one script gets data from dev and save it to some JSON files inside project service
     • another script (main script) create test data in DB using saved JSON files, which could be automatically saved from the DEV or created manually - this way we can fix and control test data, rather each time get it from the DEV, as it could bring some unexpected issues
   • The test data should be created in such a way so it could be used when we run Project Service with mock service or when we run it against DEV environment. So the test data should use the same data which we have in mock service and in DEV.

6. ✅ We should be able to somehow run tests without rebuilding and re-running Docker containers.

7. ✅ (done via Improve Local Setup - Support Windows, Single config file, Updated Docker Guide #559) Support local developing on Windows

[maxceem]

@vikasrohit I would like to have a script to populate demo data from a JSON file instead of importing data from DEV as we do now.

• The thing is, that now we have to index all the data to ES, and when we import data from DEV, it around 10 minutes to all the data to be indexed in ES. So it makes it very slow to test locally when we have to recreate data from scratch multiple times.

• We already have some scripts which do this as a side result of several challenges:

  • https://github.com/topcoder-platform/tc-project-service/tree/b2dbc560fbe7f83789bd1317fa562db9a26980eb/scripts/es-db-compare/verification
  • https://github.com/topcoder-platform/tc-project-service/tree/af2b4c4bfb5b94c841ab08787454a2d60b27c843/migrations/bookmarks

  And these scripts were very helpful during to validate the requirements of the challenges.

• We can make this script to populate data from JSON to work with any arbitrary file, so we could use also use it during the challenge, and participants could create demo data to test the challenge scope properly.

• Also, we can control what data we have in these files. So we can import data from DEV once. Clear it from the sensitive data and save it to files. So no more sensitive data would be in work during working on the challenges.

• And the last things, sometimes data on DEV got inconsistent which causes issues during importing and confuses challenge participants.

Sum up:

• Implement a script that would save all the current data in DB to file
• Implement another script that would populate DB and ES from the data in the file

PS I constantly feel the lack of such a script when I work on Project Service locally or testing it locally.

Please, let me know if we can run a challenge for creating such a script.

[vikasrohit]

@maxceem I am good with it. Only thing I would like to know is that how much new scripts we need to create now provided that we already have many scripts which fetches data from dev and writes it to the file system and then read it into local database/es.

[maxceem]

@maxceem I am good with it. Only thing I would like to know is that how much new scripts we need to create now provided that we already have many scripts which fetches data from dev and writes it to the file system and then read it into local database/es.

The script for importing data which we have now has a completely different logic from the one which is proposed above:

• it retrieves data from DEV using API
• and calls LOCAL API to create the same data without writing data to the file-system
• the big downsides of such logic are:
  • as we use API to create data, then we have to wait until all the created data is properly indexed in ES
    • this takes around 10 minutes
    • we cannot create all possible situations which we want, for example we cannot create a project and immediately after that invite a member, because it would fail during indexing
  • another downside is that we cannot recreate all the possible situations, as API cannot let recreate all possible data in DB/ES

The new script would have different logic:

• it would take data from file
• it would use sequalize to recreate data directly in DB (including soft-deleted records)
• after recreating data in DB it should index all the data to ES directly, same as we do using admin endpoints
• most complicated logic of the new script for indexing data from DB to ES would be reused between this new script and existent admin endpoints for reindexing data.

If apparently we want to be able to import some data from DEV, we can change the current import script to import data from DEV to file and after populate data from file using a new script.

[vikasrohit]

Okay, go ahead.

If apparently we want to be able to import some data from DEV, we can change the current import script to import data from DEV to file and after populate data from file using a new script.

I thought we are writing the DEV api call response to file and then using that to call against local api.

[maxceem]

A script to export and import data from and to file with fast indexing of data to Elasticsearch is done via #546.

Readme is improved and simplified.

[vikasrohit]

We can close this now, right @maxceem?

[maxceem]

Not yet. The last task is left on the list - support Windows for local setup.

[vikasrohit]

ok, I think we can keep windows efforts in separate task.

[maxceem]

ok, I think we can keep windows efforts in separate task.

@vikasrohit created a separate issue #553. So this one may be closed.