Not working with Storybook 7

[jfairley]

Logs and screenshots

click for logs...

[storybook-with-cypress] yarn test
yarn run v1.22.17
$ DEBUG=code-coverage cypress run
[62033:0519/165728.394350:ERROR:node_bindings.cc(279)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[62033:0519/165728.394372:ERROR:node_bindings.cc(279)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[62033:0519/165728.394376:ERROR:node_bindings.cc(279)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.

DevTools listening on ws://127.0.0.1:54544/devtools/browser/b64f05d0-47fb-4b88-a3c7-35082697cdab
  code-coverage combined NYC options { 'report-dir': './coverage', reporter: [ 'lcov', 'clover', 'json', 'json-summary' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: false, all: true, include: [ 'stories/**/*' ] } +0ms

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        12.12.0                                                                        │
  │ Browser:        Electron 106 (headless)                                                        │
  │ Node Version:   v18.15.0 (/Users/jfairley/n/bin/node)                                          │
  │ Specs:          1 found (button.cy.js)                                                         │
  │ Searched:       cypress/e2e/**/*.cy.{js,jsx,ts,tsx}                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  button.cy.js                                                                    (1 of 1)


  Button
  code-coverage NYC file /Users/jfairley/dev/code/storybook-with-cypress/.nyc_output/out.json has 10 key(s) +3s
  code-coverage 1 key /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js file path /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js +0ms
  code-coverage 2 key /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js file path /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js +0ms
  code-coverage 3 key /Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js file path /Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js +0ms
  code-coverage in file /Users/jfairley/dev/code/storybook-with-cypress/.nyc_output/out.json all files are not found? false +0ms
  code-coverage NYC file /Users/jfairley/dev/code/storybook-with-cypress/.nyc_output/out.json has 10 key(s) +0ms
  code-coverage nyc needs to report on all included files +0ms
  code-coverage include all files options: { all: true, include: [ 'stories/**/*' ], exclude: undefined, extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ] } +1ms
  code-coverage searching files to include using patterns [ 'stories/**/*', '!**/node_modules/**' ] +0ms
  code-coverage found 3 file(s) +4ms
/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.jsx
/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.stories.js
/Users/jfairley/dev/code/storybook-with-cypress/stories/button.css
  code-coverage coverage has 10 record(s) +1ms
coverage has the following first paths
/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js
/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js
/Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js
/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.stories.js
  code-coverage calling NYC reporter with options { 'report-dir': '/Users/jfairley/dev/code/storybook-with-cypress/coverage', reporter: [ 'lcov', 'clover', 'json', 'json-summary' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: false, all: true, include: [ 'stories/**/*' ], 'temp-dir': '/Users/jfairley/dev/code/storybook-with-cypress/.nyc_output', tempDir: '/Users/jfairley/dev/code/storybook-with-cypress/.nyc_output', reportDir: '/Users/jfairley/dev/code/storybook-with-cypress/coverage' } +6ms
  code-coverage current working directory is /Users/jfairley/dev/code/storybook-with-cypress +0ms
  code-coverage after reporting, returning the report folder name /Users/jfairley/dev/code/storybook-with-cypress/coverage +51ms
  code-coverage Final coverage in /Users/jfairley/dev/code/storybook-with-cypress/coverage/coverage-final.json +0ms
  code-coverage There are 10 key(s) in /Users/jfairley/dev/code/storybook-with-cypress/coverage/coverage-final.json +0ms
  code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js statements covered 0/0 +0ms
  code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/block-navigation.js statements covered 0/0 +0ms
  code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/prettify.js statements covered 0/0 +0ms
  code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/sorter.js statements covered 0/0 +0ms
  code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/cypress/e2e/button.cy.js statements covered 0/0 +0ms
  code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js statements covered 0/0 +0ms
  code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js statements covered 0/0 +0ms
  code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/stories/Button.jsx statements covered 0/0 +0ms
  code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/stories/Button.stories.js statements covered 0/0 +0ms
  code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/stories/button.css statements covered 0/0 +0ms
    ✓ should render (329ms)


  1 passing (1s)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 second                                                                         │
  │ Spec Ran:     button.cy.js                                                                     │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Video)

  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: 0 seconds                                                  

  -  Video output: /Users/jfairley/dev/code/storybook-with-cypress/cypress/videos/button.cy.js.mp4


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  button.cy.js                             00:01        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:01        1        1        -        -        -  

✨  Done in 6.19s.

Versions

• What is this plugin's version? If this is NOT the latest released version can you try the latest version, please?
  • 3.10.4
• If the plugin worked before in version X, but stopped after upgrading to version Y, please try the released versions between X and Y to see where the breaking change was.
  • The break occurs when upgrading storybook to v7.
• What is Cypress version?
  • 12.12.0
• What is your operating system?
  • MacOS Ventura 13.1
• What is the shell?
  • zsh
• What is the Node version?
  • 18.15.0
• What is the NPM version?
  • 9.5.0
• How do you instrument your application? Cypress does not instrument web application code, so you need to do it yourself.
  • I have two methods applied, both of which I suspect would work without the other, but I'm trying to cover my bases.
    • "@storybook/addon-coverage"
    • "babel-plugin-istanbul" registered in .babelrc.json
• When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? Can you paste a screenshot?
  • yes
  • 
• Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Can you paste at least part of it so we can see the keys and file paths?

  • Not fully empty, but no coverage reported in it.

  • see `.nyc_output/out.json`

    {
      "/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js": {
        "path": "/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js",
        "statementMap": {},
        "fnMap": {},
        "branchMap": {},
        "s": {},
        "f": {},
        "b": {},
        "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
        "hash": "b100d0ceb86dc20ac064a45298efa0bdb436b8da"
      },
      "/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js": {
        "path": "/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js",
        "statementMap": {},
        "fnMap": {},
        "branchMap": {},
        "s": {},
        "f": {},
        "b": {},
        "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
        "hash": "3c1c42d7c6490def901876d78c647964f3422f2a"
      },
      "/Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js": {
        "path": "/Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js",
        "statementMap": {},
        "fnMap": {},
        "branchMap": {},
        "s": {},
        "f": {},
        "b": {}
      },
      "/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.stories.js": {
        "path": "/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.stories.js",
        "statementMap": {},
        "fnMap": {},
        "branchMap": {},
        "s": {},
        "f": {},
        "b": {}
      },
      "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/block-navigation.js": {
        "path": "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/block-navigation.js",
        "statementMap": {},
        "fnMap": {},
        "branchMap": {},
        "s": {},
        "f": {},
        "b": {}
      },
      "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/prettify.js": {
        "path": "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/prettify.js",
        "statementMap": {},
        "fnMap": {},
        "branchMap": {},
        "s": {},
        "f": {},
        "b": {}
      },
      "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/sorter.js": {
        "path": "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/sorter.js",
        "statementMap": {},
        "fnMap": {},
        "branchMap": {},
        "s": {},
        "f": {},
        "b": {}
      },
      "/Users/jfairley/dev/code/storybook-with-cypress/cypress/e2e/button.cy.js": {
        "path": "/Users/jfairley/dev/code/storybook-with-cypress/cypress/e2e/button.cy.js",
        "statementMap": {},
        "fnMap": {},
        "branchMap": {},
        "s": {},
        "f": {},
        "b": {}
      },
      "/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.jsx": {
        "path": "/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.jsx",
        "statementMap": {},
        "fnMap": {},
        "branchMap": {},
        "s": {},
        "f": {},
        "b": {}
      },
      "/Users/jfairley/dev/code/storybook-with-cypress/stories/button.css": {
        "path": "/Users/jfairley/dev/code/storybook-with-cypress/stories/button.css",
        "statementMap": {},
        "fnMap": {},
        "branchMap": {},
        "s": {},
        "f": {},
        "b": {}
      }
    }

• Do you have any custom NYC settings in package.json (nyc object) or in other NYC config files
  • open `.nycrc`

    {
      "all": true,
      "include": ["stories/**/*"]
    }

• Do you run Cypress tests in a Docker container?
  • no

Describe the bug

I run Cypress e2e tests against my instrumented Storybook stories for React.

After upgrading to Storybook 7, when I run Cypress, I see window.__coverage__ as expected, but .nyc_output/out.json never shows coverage within it.

This worked with Storybook 6, but only upgrading to Storybook 7 breaks cypress coverage.

I assume the problem might be with conflicting transitive dependencies, but I don't know.

Link to the repo

I have created a new repository to showcase the issue. My application repositories are private.

https://github.com/jfairley/storybook-with-cypress

[jfairley]

Through local debugging, I've verified that win.__coverage__ is not available when the coverage plugin is running.

It always returns here:

code-coverage/support.js

Lines 70 to 73 in 1468c58

	const applicationSourceCoverage = win.__coverage__
	if (!applicationSourceCoverage) {
	return
	}

Yet... I've added a test step to show that the coverage object can be obtained within the test function.

    cy.window().then((win) => {
      cy.log("coverage object", win.__coverage__);
    });

(source)

[jfairley]

The best I can tell, __coverage__ just isn't available yet when called from beforeEach, but I have found that it is available during an afterEach.

My tests are working correctly with coverage when I change the beforeEach to an afterEach.

code-coverage/support.js

Line 63 in 1468c58

beforeEach(() => {

- beforeEach(() => {
+ afterEach(() => {

I don't know what the impacts are for this change. Maybe there can be both a beforeEach and an afterEach. It looks like you already have code that dedupes coverage numbers.

FWIW, I also have this problem when I render Storybook with the Vite builder. (I'm using the default Webpack 5 builder.)

[jfairley]

For the short term, I've added the following monkey patch to my repos to get me rolling again. 😭

package.json : "scripts"

"postinstall": "perl -i -pe's/beforeEach/afterEach/g' node_modules/@cypress/code-coverage/support.js",

[chrisbreiding]

I replicated this issue and will forward it to the appropriate team. They will evaluate the priority of this ticket and consider their capacity to pick it up. Please note that this does not guarantee that this issue will be resolved. The ticket will indicate status changes during evaluation, so we ask that you please refrain from asking for updates. Thanks!

[tommyc38]

Are you producing source maps?

[jfairley]

Are you producing source maps?

Yes. Coverage works with afterEach. This is a timing issue where __coverage__ isn't yet defined when the storybook beforeEach runs.

[lephuongbg]

For SPA app in general, window.__coverage__ will most likely not exist at the time of window's load event, because instrumented js is loaded asynchronously.

https://github.com/cypress-io/code-coverage/blob/d169f7bed6c55a13425b7a764f3d5baf3f0bc24e/support.js#L92C45-L92C45

Changing the timing to afterEach instead should be the way to go, unless there is a good reason why it was done with window's load event in the first place.