Stream Admin Index Endpoint output to response

[maxceem]

We have endpoint /v5/projects/admin/es/project/index which we call to index multiple projects from DB to ES by defining a range of projects ids from projectIdStart to projectIdEnd.

At the moment this endpoint only responds after we prepare data for indexing and it doesn't wait until the actual indexing is finished. So we have to see the console of the server to see the logs if we want to understand when the indexing is ready and if there were any errors during indexing or no.

• We want to see the progress with log when we run this endpoint in response. But we don't want to wait until everything is done, so the response hangs for a long time and only after all is done we can see the log. Response should be returned in parts using streams see https://stackoverflow.com/questions/38788721/how-do-i-stream-response-in-express, so we would see the constant update of response when we are calling this endpoint.

The can be implemented the next way:

• create a new logger which would do 2 things:
  • log to the console as it's done now by req.log logger
  • write to response stream
  • for example such a custom logger can be created like this https://github.com/topcoder-platform/tc-project-service/blob/develop/src/routes/admin/es-fix-metadata-for-es.js#L11-L38, but instead of writing to an array it should write to the response stream
• this new logger might have a special method to mark the stream end so we can close the connection when everything is done.
• while we log to console in JSON format, in the response we should return in the next format:
  <log_type>: <message>, examples:
  • INFO: Getting user details.
  • ERROR: Cannot get user details.
  • DEBUG: User details: {"id:"11232,"name":"Test"}.
  • WARNING: ...

[maxceem]

@vikasrohit would you like to have a such a feature?

[vikasrohit]

I am good with this. There are points that come to my mind when thinking about it:

1. Mostly when I do full reindexing of projects, I do see many projects not being indexed. But still I do not reindex them because I leave it to fix on adhoc basis when some user complaints that he/she is not able to view the project in project listing page. So, my concern is that as of now it does not give me much benefit for the bulk indexing. However, it does makes a lot of sense when we are debugging/indexing a particular project. So, it would be great to have this.
2. However, my major concern is the security. I don't think any of the topcoder API uses streaming as of now so I am not sure what are the best practices here and is there any potential risk of hacking into the systems with streaming.

[maxceem]

However, my major concern is the security. I don't think any of the topcoder API uses streaming as of now so I am not sure what are the best practices here and is there any potential risk of hacking into the systems with streaming.

It feels for me, that the use who is calling the endpoint can see the output. Streaming in this terms is not different from direct returning all the errors output at one time. The only benefit of streaming here, is that we would get errors and other logs at they time they happens, so we would always know what is going on.

The only thing I can think, if all the logs which now we log into console is safe to log into response. In reality it feels safe, as the admin how can call this endpoint can see logs on the sever usually.

But if you want to give me more consideration before implementing, let me know, otherwise we can try to implement it in the coming release.

[vikasrohit]

Lets wait on this for now.