Create a way to test nightly locally, or as part of a PR that fixes nightly

[Dr-Irv]

Right now, if we get a PR that fixes an issue for the nightly tests, the only way to test it is to approve the PR and merge it, and see what happens as a result of the merge.

We should figure out a way that we can run the nightly tests on demand, either locally, or by doing something in the PR that signals that the nightly tests should be run as part of the PR.

Maybe @twoertwein has an idea here.

[loicdiridollou]

@twoertwein I looked into it and why you were having troubles creating that.
Here is what happened:

• The workflows you see in the Actions tabs are the ones that gets triggered when you push to the branch.
• When you use a comment, you actually use a different trigger (the on section of the yaml workflow).
• So even if you commented and it actually triggered the workflow you were hoping for, it would not display in the Checks tab of the PR.

Here is a workaround that we could use (I actually tried it out, PR on my own repo is below since you need to have the workflow file on main branch to have it applied in a PR):

• You create a comment command like @twoertwein you did on your PR.
• The Action will get triggered and run.
• Once the Action is complete you actually create a check run (https://docs.github.com/en/rest/checks/runs?apiVersion=2022-11-28) where you will list if your test ran without error or not.
• From there, this will be displayed a little differently than a typical Actions but you will still get most of the information:

- We can decide what we want to add in the messages there.

Of course the content of the message will be a little different than if we were using a more standard Actions that is triggered on push but that is a good workaround not to have to run the nightly checks on merge to main branch (mostly to save CI credits and have the status check appearing in green).
PR I used is there loicdiridollou#2, feel free to trigger the workflow with /nightly, I have not implemented the other commands yet but wanted to get a good proof-of-concept to share first and discuss if this is a route that you would like to take.
The code for the workflow is there https://github.com/loicdiridollou/pandas-stubs/blob/main/.github/workflows/comment_commands.yml it is a little tricky as I did not comment anything but we can discuss what is going on there if you need (only focus on the nightly job since I did not implement the other ones yet).

Please let me know what you think and I can spend a little bit more time working on improvements/changes.

[twoertwein]

Great, thank you @loicdiridollou for figuring it out!

Feel free to open a PR!