Support bulk work periods payments

[maxceem]

We should be able to make bulk payments for multiple work periods up to around 2500 at a time or more.

• We should make sure that we don't hit any API rate limitations.

[maxceem]

It's being implemented by 2 challenges:

• Topcoder TaaS API - Payments - Batch Endpoints https://www.topcoder.com/challenges/aa2758b2-7a53-4662-9331-e1e9f55066d3?tab=details
• Topcoder TaaS API - Batch Payments - Part 1 - Scheduler https://www.topcoder.com/challenges/3dbe253c-d6f0-456a-bd21-8e3e211143fc?tab=details

[sandhiyakavi]

@maxceem @nkumar-topcoder I verified the Payment scheduler and below are some of the issues I found,

Steps to Reproduce:

1. Created 7 RB's using api with Date as 2022-12-18 to 2023-02-28
   ProjectId: 17769, job id: 1f431663-8111-40e0-9108-f1909ed14163

   RB's : dc861c89-6446-44b8-a571-738a03fea4a3, d069a851-f454-4fdd-bf6e-a694b759bda2, 6e5d4c57-ba2e-4ec9-9c44-ac932e327008, 486ed95b-006e-4100-b47a-0e8da6933726, 38e8dd03-b041-48dd-9d00-a1d48e72f2fa, 374d1efd-e6ab-4bdc-b80e-d06bc186a675, 20bd6f91-2bff-4a20-a2ae-0012381460b9

2. Created WorkPeriod payment with query {"query": { "workPeriods.paymentStatus": "pending", "workPeriods.startDate": "2022-12-18" } }. (All 7 success)

Actual result:
6 records were failed at step1: "create-challenge" and one record is still "in-progress" status.

[
    {
        "userHandle": "Leilani_Fahey35",
        "updatedBy": null,
        "endDate": "2022-12-24",
        "resourceBookingId": "d069a851-f454-4fdd-bf6e-a694b759bda2",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T08:46:07.490Z",
                "challengeId": null,
                "amount": 13.23,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": null,
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "create-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "69134b36-1f4b-4479-b428-09c509d2c919",
                "workPeriodId": "2aee7175-3cb5-4d1c-81a0-2ba13cfd1b8e",
                "status": "failed",
                "updatedAt": "2021-06-14T08:50:00.728Z"
            }
        ],
        "createdAt": "2021-06-14T08:29:00.060Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "2aee7175-3cb5-4d1c-81a0-2ba13cfd1b8e",
        "projectId": 17769,
        "startDate": "2022-12-18",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T08:29:00.060Z"
    },
    {
        "userHandle": "Sylvan_Gorczany",
        "updatedBy": null,
        "endDate": "2022-12-24",
        "resourceBookingId": "38e8dd03-b041-48dd-9d00-a1d48e72f2fa",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T08:46:07.735Z",
                "challengeId": null,
                "amount": 13.23,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": null,
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "create-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "ed4b976b-9330-4668-b1fa-9243918f4c23",
                "workPeriodId": "211c8fee-d424-4cff-b0ef-58b34b9de0a9",
                "status": "failed",
                "updatedAt": "2021-06-14T08:50:02.107Z"
            }
        ],
        "createdAt": "2021-06-14T08:28:46.351Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "211c8fee-d424-4cff-b0ef-58b34b9de0a9",
        "projectId": 17769,
        "startDate": "2022-12-18",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T08:28:46.351Z"
    },
    {
        "userHandle": "TCConnCopilot",
        "updatedBy": null,
        "endDate": "2022-12-24",
        "resourceBookingId": "dc861c89-6446-44b8-a571-738a03fea4a3",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T08:46:06.226Z",
                "challengeId": null,
                "amount": 13.23,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": null,
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "create-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "eed39aca-3124-4c66-b52f-3602c7c72bf0",
                "workPeriodId": "9544e15f-6fb8-4d42-a1b8-1abe97e34fbd",
                "status": "failed",
                "updatedAt": "2021-06-14T08:50:00.428Z"
            }
        ],
        "createdAt": "2021-06-14T08:29:13.738Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "9544e15f-6fb8-4d42-a1b8-1abe97e34fbd",
        "projectId": 17769,
        "startDate": "2022-12-18",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T08:29:13.738Z"
    },
    {
        "userHandle": "dan_developer",
        "updatedBy": null,
        "endDate": "2022-12-24",
        "resourceBookingId": "20bd6f91-2bff-4a20-a2ae-0012381460b9",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T08:46:07.879Z",
                "challengeId": null,
                "amount": 13.23,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": null,
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "create-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "f3492521-e6ad-4e2b-be08-7704d2e2d57b",
                "workPeriodId": "396dca7c-90eb-48e3-9999-e944db6a6db8",
                "status": "failed",
                "updatedAt": "2021-06-14T08:50:02.732Z"
            }
        ],
        "createdAt": "2021-06-14T08:28:05.706Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "396dca7c-90eb-48e3-9999-e944db6a6db8",
        "projectId": 17769,
        "startDate": "2022-12-18",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T08:28:05.706Z"
    },
    {
        "userHandle": "pshah_manager",
        "updatedBy": null,
        "endDate": "2022-12-24",
        "resourceBookingId": "6e5d4c57-ba2e-4ec9-9c44-ac932e327008",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T08:46:07.534Z",
                "challengeId": null,
                "amount": 13.23,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": null,
                "billingAccountId": 80000068,
                "id": "51afb7c1-986c-4249-b477-b7b27bdcf934",
                "workPeriodId": "6e188730-64ef-4280-ad63-89e31b204bec",
                "status": "in-progress",
                "updatedAt": "2021-06-14T08:50:00.923Z"
            }
        ],
        "createdAt": "2021-06-14T08:27:26.154Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "6e188730-64ef-4280-ad63-89e31b204bec",
        "projectId": 17769,
        "startDate": "2022-12-18",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T08:27:26.154Z"
    },
    {
        "userHandle": "sachin-kumar",
        "updatedBy": null,
        "endDate": "2022-12-24",
        "resourceBookingId": "486ed95b-006e-4100-b47a-0e8da6933726",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T08:46:07.590Z",
                "challengeId": null,
                "amount": 13.23,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": null,
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "create-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "b7613b6d-0df4-4630-9138-e103c70fcd1d",
                "workPeriodId": "812f0158-76da-4de7-842d-efb722abb85b",
                "status": "failed",
                "updatedAt": "2021-06-14T08:50:01.736Z"
            }
        ],
        "createdAt": "2021-06-14T08:28:30.894Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "812f0158-76da-4de7-842d-efb722abb85b",
        "projectId": 17769,
        "startDate": "2022-12-18",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T08:28:30.894Z"
    },
    {
        "userHandle": "tester1234",
        "updatedBy": null,
        "endDate": "2022-12-24",
        "resourceBookingId": "5a7473e5-bc9a-4862-9bd4-6c40cb0eb514",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T09:30:32.410Z",
                "challengeId": null,
                "amount": 111,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": "e087a001-665c-4a08-afe3-b34ca2fa3abf",
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "activate-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "8dcfdef7-7a02-4fd6-b7b0-c173f134079b",
                "workPeriodId": "286e1397-725c-47e6-96ed-3f87ac8911c0",
                "status": "failed",
                "updatedAt": "2021-06-14T09:35:03.079Z"
            }
        ],
        "createdAt": "2021-06-14T09:24:28.514Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "286e1397-725c-47e6-96ed-3f87ac8911c0",
        "projectId": 17769,
        "startDate": "2022-12-18",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T09:24:28.514Z"
    },
    {
        "userHandle": "topcoderaccone",
        "updatedBy": null,
        "endDate": "2022-12-24",
        "resourceBookingId": "374d1efd-e6ab-4bdc-b80e-d06bc186a675",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T08:46:07.833Z",
                "challengeId": null,
                "amount": 13.23,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": null,
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "create-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "a029ce34-7411-4502-b50b-5c2e11a0b1ed",
                "workPeriodId": "65525619-6961-4498-a066-145c7fb6e9d4",
                "status": "failed",
                "updatedAt": "2021-06-14T08:50:02.335Z"
            }
        ],
        "createdAt": "2021-06-14T08:29:23.562Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "65525619-6961-4498-a066-145c7fb6e9d4",
        "projectId": 17769,
        "startDate": "2022-12-18",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T08:29:23.562Z"
    }
]

[sandhiyakavi]

@maxceem @nkumar-topcoder Verified the Payment scheduler with another 6 RBs created through rcrm again but here 4 were failed at Step 3: "Activate challenge" and two records are still in "in-progress" state.

Steps to Reproduce:

1. Created 6 RB's using api with Date as 2021-12-19 to 2022-01-16
   ProjectId: 17769,job id: 0da20d59-c3c4-4930-9abd-fe93b4de0d97
   RB's : fa0a7d82-cb4c-4faa-bc78-6b102b79f6b4, 889488bb-9b33-45fb-a00c-1bd75e9524db, 61c5ddf8-e154-46ea-b789-d5dc3ecb2fa9, 484f059d-fbff-445b-aefc-b738bb71b8b6, 225dc7eb-cb68-42f2-a11b-767813034bf7, 0d2af87e-2741-4baf-9d16-9f785f5a6d6e

2. Created WorkPeriod payment with query {"query": { "workPeriods.paymentStatus": "pending", "workPeriods.startDate": "2021-12-19", "projectId": 17769 } }. (All 6 success)

[
    {
        "userHandle": "GunaK-TopCoder",
        "updatedBy": null,
        "endDate": "2021-12-25",
        "resourceBookingId": "0d2af87e-2741-4baf-9d16-9f785f5a6d6e",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T10:41:41.568Z",
                "challengeId": null,
                "amount": 222,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": null,
                "billingAccountId": 80000068,
                "id": "3d8a26c7-ec44-458b-b554-9bd7a8d94fe2",
                "workPeriodId": "2e2d3728-d3c1-44b7-80f4-7dbfe7d22e21",
                "status": "in-progress",
                "updatedAt": "2021-06-14T10:45:11.019Z"
            }
        ],
        "createdAt": "2021-06-14T10:38:30.016Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "2e2d3728-d3c1-44b7-80f4-7dbfe7d22e21",
        "projectId": 17769,
        "startDate": "2021-12-19",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T10:38:30.016Z"
    },
    {
        "userHandle": "TCConnCopilot",
        "updatedBy": null,
        "endDate": "2021-12-25",
        "resourceBookingId": "889488bb-9b33-45fb-a00c-1bd75e9524db",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T10:41:39.977Z",
                "challengeId": null,
                "amount": 222,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": "0a1e0aa4-8a0e-48a0-b0d2-cbeeb5337745",
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "activate-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "8762141a-211f-406b-b404-70e16ee1ef09",
                "workPeriodId": "ee517d8b-bb3b-47af-8517-130e5a76dc56",
                "status": "failed",
                "updatedAt": "2021-06-14T10:45:04.766Z"
            }
        ],
        "createdAt": "2021-06-14T10:37:36.975Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "ee517d8b-bb3b-47af-8517-130e5a76dc56",
        "projectId": 17769,
        "startDate": "2021-12-19",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T10:37:36.975Z"
    },
    {
        "userHandle": "nkumar1",
        "updatedBy": null,
        "endDate": "2021-12-25",
        "resourceBookingId": "fa0a7d82-cb4c-4faa-bc78-6b102b79f6b4",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T10:41:38.886Z",
                "challengeId": null,
                "amount": 222,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": "ff486941-ca26-4e36-a458-622837f05f49",
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "activate-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "85379165-456d-4f1d-972e-18b3c8b97aec",
                "workPeriodId": "fd4c1a02-86d9-42ad-bb79-f36b59d52a73",
                "status": "failed",
                "updatedAt": "2021-06-14T10:45:02.254Z"
            }
        ],
        "createdAt": "2021-06-14T10:37:21.268Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "fd4c1a02-86d9-42ad-bb79-f36b59d52a73",
        "projectId": 17769,
        "startDate": "2021-12-19",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T10:37:21.268Z"
    },
    {
        "userHandle": "pshah_manager",
        "updatedBy": null,
        "endDate": "2021-12-25",
        "resourceBookingId": "61c5ddf8-e154-46ea-b789-d5dc3ecb2fa9",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T10:41:40.776Z",
                "challengeId": null,
                "amount": 222,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": "7dd1403f-6700-4c68-b8d2-3c37505e8776",
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "activate-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "c0600ffb-b6bc-4727-bd11-4a48b6be6468",
                "workPeriodId": "7e97332c-4695-4d65-bec5-7943e708fc4e",
                "status": "failed",
                "updatedAt": "2021-06-14T10:45:06.753Z"
            }
        ],
        "createdAt": "2021-06-14T10:37:51.833Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "7e97332c-4695-4d65-bec5-7943e708fc4e",
        "projectId": 17769,
        "startDate": "2021-12-19",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T10:37:51.833Z"
    },
    {
        "userHandle": "tester1234",
        "updatedBy": null,
        "endDate": "2021-12-25",
        "resourceBookingId": "225dc7eb-cb68-42f2-a11b-767813034bf7",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T10:41:41.408Z",
                "challengeId": null,
                "amount": 222,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": {
                    "challengeId": "b583ea79-34f9-4547-8252-511fabcac040",
                    "errorMessage": "Bad Request",
                    "errorCode": 400,
                    "step": "activate-challenge",
                    "retry": 1
                },
                "billingAccountId": 80000068,
                "id": "016c7874-b0e8-43cb-8e29-1157845b2b5a",
                "workPeriodId": "7eb468ab-4120-47ba-97f7-e41e73eb87df",
                "status": "failed",
                "updatedAt": "2021-06-14T10:45:10.980Z"
            }
        ],
        "createdAt": "2021-06-14T10:38:20.146Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "7eb468ab-4120-47ba-97f7-e41e73eb87df",
        "projectId": 17769,
        "startDate": "2021-12-19",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T10:38:20.146Z"
    },
    {
        "userHandle": "topacc_three",
        "updatedBy": null,
        "endDate": "2021-12-25",
        "resourceBookingId": "484f059d-fbff-445b-aefc-b738bb71b8b6",
        "daysWorked": null,
        "payments": [
            {
                "createdAt": "2021-06-14T10:41:41.284Z",
                "challengeId": null,
                "amount": 222,
                "updatedBy": null,
                "createdBy": "57646ff9-1cd3-4d3c-88ba-eb09a395366c",
                "statusDetails": null,
                "billingAccountId": 80000068,
                "id": "9163dccb-6f0d-4f60-93c8-bb6e8fda6d20",
                "workPeriodId": "3adfe3ba-64b2-47c7-983c-f725b69ceb27",
                "status": "in-progress",
                "updatedAt": "2021-06-14T10:45:06.813Z"
            }
        ],
        "createdAt": "2021-06-14T10:38:09.330Z",
        "memberRate": null,
        "createdBy": "00000000-0000-0000-0000-000000000000",
        "customerRate": null,
        "id": "3adfe3ba-64b2-47c7-983c-f725b69ceb27",
        "projectId": 17769,
        "startDate": "2021-12-19",
        "paymentStatus": "pending",
        "updatedAt": "2021-06-14T10:38:09.330Z"
    }
]

[sandhiyakavi]

@maxceem @nkumar-topcoder While trying for multiple payment, I observed the below issues

samples: https://platform.topcoder-dev.com/taas-admin/work-periods?startDate=2023-06-11&by=start_date&order=asc&perPage=10&page=1
https://platform.topcoder-dev.com/taas-admin/work-periods?startDate=2023-06-04&by=start_date&order=asc&perPage=10&page=1

1. If we remove the BA of the project and try to schedule payment, it's moving to in-progress and for some resource it's getting stuck at 'in-progress' status and we are able to do another payment irrespective of status still being in 'in-progress'
   

2. Also for some resource, even though the status is still 'in-progress' the WP status is changed to 'Completed' (with 2 payments in completed status) with daysworked=3

Note: If there are no failed status, all the payments for multiple payment scenario are getting completed successfully.
sample: https://platform.topcoder-dev.com/taas-admin/work-periods?startDate=2023-05-28&by=start_date&order=asc&perPage=10&page=1