[$250] DynamoDB Timeout

[jmgasper]

@afrisalyp - We're seeing timeouts now in other operations (not just git access links). See below for a log when a user tried to archive a project.

What we're thinking is that:

1. Maybe we should use query instead of scanOne?
2. Are the indices correct for how we are querying / scanning? I see we have an index on tcDirectId, but do we need any other indices to speed things up?

From @rootelement :

i don’t believe this is correct:

https://github.com/topcoder-platform/topcoder-x-ui/blob/develop/src/common/db-helper.js#L85
all() ?

you’re trying to pull the whole table and filter it?

i know query is better than scan, but scan might need a better hash and range key based on the model you’re trying to pull from

2021-05-25T08:08:07.246-04:00	error: DynamoDB scanOne error Error: Wait for Active timed out after 180000 ms.
2021-05-25T08:08:07.246-04:00	error: Error happened in undefined
2021-05-25T08:08:07.246-04:00	Error: Wait for Active timed out after 180000 ms.
2021-05-25T08:08:07.246-04:00	at Timeout.waitForActive [as _onTimeout] (/topcoder-x-ui/node_modules/dynamoose/dist/Table.js:198:36)
2021-05-25T08:08:07.246-04:00	at listOnTimeout (internal/timers.js:554:17)
2021-05-25T08:08:07.246-04:00	at processTimers (internal/timers.js:497:7)
2021-05-25T08:08:07.247-04:00	error: Error happened in system
2021-05-25T08:08:07.247-04:00	TypeError: Cannot read property 'count' of undefined
2021-05-25T08:08:07.247-04:00	at /topcoder-x-ui/src/common/db-helper.js:91:29
2021-05-25T08:08:07.247-04:00	at /topcoder-x-ui/node_modules/dynamoose/dist/Scan.js:267:24
2021-05-25T08:08:07.247-04:00	at _rejected (/topcoder-x-ui/node_modules/q/q.js:864:24)
2021-05-25T08:08:07.247-04:00	at /topcoder-x-ui/node_modules/q/q.js:890:30
2021-05-25T08:08:07.247-04:00	at Promise.when (/topcoder-x-ui/node_modules/q/q.js:1142:31)
2021-05-25T08:08:07.247-04:00	at Promise.promise.promiseDispatch (/topcoder-x-ui/node_modules/q/q.js:808:41)
2021-05-25T08:08:07.247-04:00	at /topcoder-x-ui/node_modules/q/q.js:624:44
2021-05-25T08:08:07.247-04:00	at runSingle (/topcoder-x-ui/node_modules/q/q.js:137:13)
2021-05-25T08:08:07.247-04:00	at flush (/topcoder-x-ui/node_modules/q/q.js:125:13)
2021-05-25T08:08:07.247-04:00	at processTicksAndRejections (internal/process/task_queues.js:75:11)

Standards

https://platform.topcoder-dev.com/model/help/copilots-link

[jmgasper]

Challenge https://www.topcoder.com/challenges/632ec9db-e0e6-46d6-8db2-5141f72c2e08 has been created for this ticket.

This is an automated message for ghostar2020 via Topcoder X

[jmgasper]

Challenge https://www.topcoder.com/challenges/632ec9db-e0e6-46d6-8db2-5141f72c2e08 has been assigned to afrisalyp.

This is an automated message for ghostar2020 via Topcoder X

[afrisalyp]

@jmgasper

I'm not sure that changing to query will solve this issue. Both of the methods calls waitForActive function in dynamoose lib

https://github.com/dynamoose/dynamoose/blob/v1.11.1/lib/Scan.js#L312

https://github.com/dynamoose/dynamoose/blob/v1.11.1/lib/Query.js#L302

But I agree that it's better to migrate to use query. I remember that we've tried to address it.

#256

We've planed it actually :)

#256 (comment)

This is the explanation why we call the all() function.

#256 (comment)

[afrisalyp]

Abiut the error Error: Wait for Active timed out after 180000 ms.

I think it's possible that the error came either from the dynamodb server or from the dynamoose library since we're using old version of dynamoose v 1.

Any idea?

[jmgasper]

@afrisalyp - How much effort would it be to update to a newer version of Dynamoose?

[afrisalyp]

Seems pretty complex https://dynamoosejs.com/other/FAQ/#how-do-i-migrate-from-v1-to-v2

Btw do we have other Topcoder services that run DynamoDB and dynamoose?

[jmgasper]

I'll ask.

[rootelement]

Everything in v5 uses Dynamoose 1, but most other services that require any sort of query have Elasticsearch in front of Dynamoose and the searching is done against that ES index.

…

On Mon, May 31, 2021 at 5:41 PM Justin Gasper ***@***.***> wrote: I'll ask. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#396 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJGMXH6OPKK7WU765JOJULTQP7ARANCNFSM45QHRLKQ> .

[rootelement]

I haven't looked closely at the code, but what I do believe is that the problem stems from us not using properly defined hash and range keys, and using a generic "get model" function in the code. I think if we switched to Query and use a range key defined on the project id, some of this would get a lot faster, but again, not an expert and not positive i'm using the correct vocabulary.

Looking in Dynamo, it doesn't look like there's an index on the projectId on the issues table.

On the challenge resource table, there's a defined range and hash key to partition the table to make the queries faster:

https://github.com/topcoder-platform/resources-api/blob/develop/src/models/Resource.js#L10

It seems like there should be a hash key or range key on projectId

[jmgasper]

@afrisalyp - Please see above. Note that we use dynamoose 2.3 in this API (which is being deprecated): https://github.com/topcoder-platform/evernym-api