test: adds scheduled smoke-tests

[AriPerkkio]

What:

See discussion from #97.
This PR will add weekly scheduled smoke tests to be run on every Sunday at 00:00. These tests will verify that rules of recommended set do not cause linter crashes.

The test set contains 646 repositories. My initial repositories.json contained ~150 repositories so I included ~500 more by querying libraries.io API. These repositories had @testing-library/react as dependency.

Contents of each of these repositories are run against this ESLint plugin's production build, the ./dist directory.

Example run on CI: https://github.com/AriPerkkio/eslint-plugin-jest-dom/actions/runs/390036475

Why:

Unit tests cover only cases which are expected. The AST produced by Javascript and Typescript can cause very unexpected conditions. These often lead into null pointers and can crash the linter.

ESLint rules should never crash. A crashing rule causes linter to lose all valid linting problems.

The current master branch seems to contain two bugs which were found while setting up this PR:

Crash reports

Repository: roginfarrer/react-collapsed

Rule: prefer-to-have-style

Message: Cannot read property 'replace' of undefined Occurred while linting :74
Path: roginfarrer/react-collapsed/src/__tests__/index.test.tsx
Link

  );
  const collapse = getByTestId('collapse');
  expect(collapse).toHaveAttribute('id');
  expect(collapse).toHaveAttribute('aria-hidden', 'false');
  expect(collapse.style).not.toContain(
    expect.objectContaining({
      display: 'none',
      height: '0px',
    })
  );

Repository: FormidableLabs/spectacle

Rule: prefer-to-have-style

Message: Cannot read property 'range' of undefined Occurred while linting :37
Path: FormidableLabs/spectacle/src/components/deck/deck.test.js
Link

    );
    expect(tree.find('AnimatedDeckDiv').props().style).toHaveProperty(
      'transform'
    );
    expect(tree.find('AnimatedDeckDiv').props().style).not.toHaveProperty(
      'opacity'
    );
  });

  it('should pass the transition effect to children slides', () => {

How:

Smoke tests are run on CI. The Github action will build the production build, link local dependency to point at this directory and scan the given repositories. The process will exit with error code in case of errors.
The job can also be ran manually, workflow_dispatch.

Checklist:

• Documentation
• Tests
• Ready to be merged
  

As discussed on the issue earlier, it would be great if Github Issues could be created automatically. I have not implemented this but I've added a feature which allows such integration.
See the onComplete property of configuration file.

I had to push this commit with --no-verify option. The kcd-scripts froze on the test step, kcd-scripts test --findRelatedTests.

All comments are welcome!

[codecov]

Codecov Report

Merging #108 (9a6e1ba) into master (878cb0e) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master      #108   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           12        12           
  Lines          218       218           
  Branches        27        27           
=========================================
  Hits           218       218           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 878cb0e...9a6e1ba. Read the comment docs.

[benmonro]

@AriPerkkio is it possible to run non recommended rules as well? For example, we recently added a new rule prefer-in-document which we want to vet before putting in the recommended config.

[AriPerkkio]

prefer-in-document is now added.

[AriPerkkio]

@benmonro FYI, just realized this after the merge. I'm not sure but this should probably have been prefixed with plugin name:

'jest-dom/prefer-in-document': 'error'

It's currently now showing any errors but the rule might not have been loaded by ESLint.

[benmonro]

Oh good catch. Actually I might add another config with all the rules set to error anyway so we can just switch to that

[benmonro]

this is so cool! 🔥

[benmonro]

@all-contributors please add @AriPerkkio for code, bugs, tests

[allcontributors]

@benmonro

I've put up a pull request to add @AriPerkkio! 🎉

[benmonro]

@AriPerkkio is it possible to have a manual trigger? if so how does that work?

[AriPerkkio]

is it possible to have a manual trigger? if so how does that work?

The workflow_dispatch is manual trigger in Github Actions. https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#manual-events

on:
  schedule:
    - cron: '0 0 * * SUN'
  workflow_dispatch:

Note that the syntax above requires that additional : since schedule has properties. It looks weird but is valid yml.

The button for triggering manual event is found from Actions tab:

[github-actions]

🎉 This PR is included in version 3.4.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀