React Skill not showing as verified

[ajefts]

Describe the bug
The "React.js" skill is not showing on profiles as verified even in cases where they have successfully competed on a challenge with React.js tagged.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'https://www.topcoder.com/members/diazz
2. Click on 'Show More" skills
3. Scroll down to 'React.js'
4. See that there is no blue badge indicating it's verified.

This seems to be true for all users with React.js.

Expected behavior
Skill should show blue badge.

Screenshots

NOTE: Ritesh checked Dynamo and apparently the skills are not stored as verified for the user there.

[ajefts]

@Oanh-and-only-Oanh @sushilshinde @urwithat Can you help with this one? We have a lot of TaaS leads for React resources and this is slowing us down a lot since we can't accurately search for React developers.

[gondzo]

@Oanh-and-only-Oanh I'm not sure which data loader manages skills verification. Any ideas @ajefts @RishiRajSahu ?

[RishiRajSahu]

@gondzo Are they part of loaders in first place, I mean does loaders push skills data to dynamo ?

@urwithat Which table in dynamo is responsible for skills data and how does that tables gets populated ?

[urwithat]

@RishiRajSahu - There are two tables - MemberAggregatedSkills & MemberEnteredSkills
MemberEnteredSkills - Gets updated via https://api.topcoder.com/v3/members/upbeat/skills get / patch
MemberAggregatedSkills - contains details on skills that have been added from participating in. Not sure how this is added!

[RishiRajSahu]

@Oanh-and-only-Oanh and @ajefts I dived around this issue and found out that aws emr is responsible for verifying and aggregating the user skills. So basically it works on MR architecture by hadoop and number of jobs are sechuled under the emr cluster. Below are the details regarding that -

1. The jobs scheduling for emr cluster is done via aws datapipeline
2. The required jars and other files are present under S3 bucket
3. How/Which are the jobs that does the required thing

• hive pulls all user's entered skills from dynamo
• hive pulls all user's stackoverflow skills from dynamo
• scoop pulls data from DW for each user's project and tech
• Actual hadoop job to aggregate/verify all above three skills
• exported aggregated skills to dyanmodb table

Now, why we were seeing the above issue. With the passage of time the user skills data has been increased but the data pipeline has been assigned just 2 hours for running all the jobs, and the export job has been failing since Jan'2018 and hence no skill was getting verified, the verified skills we are seeing are from older succesfull runs before Jan'2018.

I tried increasing the time for datapipeline execution but it didn't help seems our export to dynamo job always fails.(we have to figure out solution around that)

Next step I tried is running it for single user, it succeeded but it didn't updated the user skills for that single user, the reason thereafter I found was because of mismatch in tech name present in DW versus the one used by MR for compare the skills verification process.

I have added the synonym for the Tech on MR side and then running for a single user succeeded so now you can see React.js as verified skill for diazz. But this solves particular git issue I think we still have to design new solution around verifying skills for all users successfully.

fyi @urwithat

[ajefts]

@RishiRajSahu Is there a particular part of the job that is taking a long time that we can tune?

[RishiRajSahu]

@ajefts Yep export is taking a lot which never completes even after providing duration of ~6 hours, even importing user entered skills takes 1 hours(though it completes successfully) but would always increase with increase in users. So we need to tune -

1. Hive query to import user entered skills
2. Hive export for aggregated skills to dynamo

[RishiRajSahu]

@ajefts Also,

• There is no support of Delta on these jobs, they run for all data we have every-time. Maybe we can optimize it to run based on delta but have to check how that affects the skills being verified.

[RishiRajSahu]

@Oanh-and-only-Oanh what's the priority on this, Do we have to plan for solutions for this or are there any other dependent issues/features related to this ?