Admin endpoint for reindexing project shouldn't populate member details

[maxceem]

TL;DR

Admin endpoint POST /v4/projects/admin/es/project/index for indexing project from DB to ES sends multiple requests to Member Service to populate user details but we never use this data.

Details

When index projects using admin endpoint it retrieves data from the Member Service for each project member and additionally populates 'handle', 'firstName', 'lastName', 'email' for each them: https://github.com/topcoder-platform/tc-project-service/blob/dev/src/routes/admin/project-index-create.js#L74-L88

This was implemented this way because we do the same when we index members in ES during adding them: https://github.com/topcoder-platform/tc-project-service/blob/dev/src/events/projectMembers/index.js#L90-L91

Even though we add these additional details to the ES index, we actually never use them in Connect App V4. As by default, these fields are not returned by project listing endpoint, and we don't request them: https://api.topcoder-dev.com/v4/projects/?limit=20&offset=0&fields=id,name,description,members,status,type,actualPrice,estimatedPrice,createdAt,updatedAt,createdBy,updatedBy,details,lastActivityAt,lastActivityUserId,version,templateId&sort=updatedAt+desc, see a screenshot of returned data:

In Project Service V5 we even don't populate user details to the project member records anymore https://github.com/topcoder-platform/project-processor-es/blob/develop/src/services/ProcessorServiceProjectMember.js#L53-L64 and Connect App still works well without such data added to ES.

Summary

So I propose to remove this logic from this admin endpoint and don't request user details for members, this should dramatically speed up indexing via this endpoint.

FYI @vikasrohit

[vikasrohit]

@maxceem I was about to say yes, but I tried looking at history of the these files and recalled that we are using these member details for searching the projects by project members i.e. when you put user handle or email, we use this data stored in the ES, to filter the projects.

[maxceem]

Ah, good catch @vikasrohit so we have to fix v5 then. As I understand as per new logic we should populate the member details in project-processor-es rather than tc-project-service.

[vikasrohit]

Yep, agreed. ES indexing part should be in project-processor-es. I thought it got ported to the processor already with our v5 efforts.

[maxceem]

Yep, agreed. ES indexing part should be in project-processor-es. I thought it got ported to the processor already with our v5 efforts.

It wasn't, though I've already got it done via F2F topcoder-platform/project-processor-es#8.

For testing that PR, I used a mock service instead of real Member Service, due to the issue in Member Service https://github.com/appirio-tech/ap-member-microservice/issues I cannot verify it fully with the M2M token authorization.

[vikasrohit]

Thanks @maxceem for bearing the issues and finding work around for time being.

[maxceem]

The PR for this issue is ready to be merged topcoder-platform/project-processor-es#9. It should work no matter if Member Service fails or no. If it fails to get user details it still adds a new member to the project as having user details is not critical while failing to add a member in ES would create data inconsistency between DB and ES.

[vikasrohit]

Closing the issue. For records, we can not remove the logic of populating the member details while indexing a project because we need it for searching purposes.