feat: add pytest-testmon plug in

[bnkc]

The test suite is pretty large (824 items). This pytest plug-in dynamically selects and re-executes only tests affected by recent changes which makes it easier to pinpoint issues within the test suite.

Example usage:

sh bin/test --testmon
-----------------------------------------------
collected 24 items / 22 deselected / 2 selected                                                                                              
tests/api_resources/test_completions.py ..    

=== 2 passed, 22 deselected in 0.37s ===

The --testmon flag only ran the tests my recent changes affected. If no changes impact the test suite, no tests will run. You always have the option to run the full test suite by omitting the flag.

• I understand that this repository is auto-generated and my pull request may not be merged

Additional context & links

• pytest-testmon
• Determining affected tests

[rattrayalex]

Thanks for the PR! How slow is the current test suite for you now, and how fast is it after this change? Will it slow down CI at all?

cc @RobertCraigie

[bnkc]

Hi @rattrayalex thanks for the response!

Full test suite:

======== 784 passed, 40 skipped in 7.24s ========

The current test suite is pretty fast. After including --testmon The test suite can vary in speed (can only run faster than full test suite unless your changes affect every single test). In the initial example provided it took 0.37s to run two tests impacted by changes. This can be incredibly helpful to speed up development as you can quickly iterate on changes.

I do still believe it's necessary to run the full test suite upon commit, maybe introducing a pre-commit hook.

[rattrayalex]

Great, thanks! We'll consider incorporating this on the codegen side of things, which may take a bit, but I'm optimistic :)

[bnkc]

Thankyou!

[slyapustin]

@bnkc, I believe it's risky to overlook certain test cases, even if they seem indirectly related. Addressing all test scenarios ensures comprehensive coverage.

@rattrayalex, if execution time is a concern, I recommend two approaches: either optimize the slower tests for efficiency or consider using tools like pytest-xdist, which enables parallel test execution, thus reducing overall time.

[bnkc]

@slyapustin Thanks for your interest in this PR. If you read my proposal carefully, you'd see that I had never suggested to overlook non-tangentially related tests. It's important to run the full test suite upon commit. It's NOT important and actually very annoying to run the test suite every time you are making small tweaks in development. This can speed up development significantly by simply doing a thorough check of your test suite at the end of your changes.

Hope this helps.

[rattrayalex]

Thanks; we do intend to incorporate this change! Thanks for your patience.