Test is 6x slower in 3.3.2 with cy:open

[bahmutov]

@jfaissolle commented on Fri Jun 28 2019

Current behavior:

With cypress open, running a single spec:
Running in 3.3.1 takes 21 seconds.
Running in 3.3.2 takes 117 seconds.

Most CPU is consumed by the hasBinary function.

Desired behavior:
Perfomance at least as good as 3.3.1 with cypress open.

Versions
Cypress 3.3.1 on Ubuntu 18.04.2, Chrome 75.0.3770.100

---

@flotwig commented on Fri Jun 28 2019

Hey @jfaissolle, could you share the spec code that is taking longer in 3.3.2? We made performance improvements in 3.3.2, but it looks like you're hitting a new edge case caused by #4407.

---

@jfaissolle commented on Mon Jul 01 2019

I have found that the slowdown happens with @cypress/code-coverage on. If I disable the plugin, tests perform as fast as in 3.3.1.

Here is an example of test.

const continueButton = '[data-ft="panel-footer"] button:contains("Continue")';
const backButton = '[data-ft="panel-footer"] button:contains("Back")';

const surveyCategory = '[data-ft="survey-category"]';

context('Stepper', function () {
  beforeEach(function () {
    cy.visit('iframe.html?id=survey--survey-stepper');
  });

  it('Looks good', function () {
    cy.wait(500);
    cy.matchImageSnapshot();
  });
});


context('Actions', function () {
  beforeEach(function () {
    cy.visit('iframe.html?id=survey--survey-wizard');
    cy.get('[data-ft="survey-card"]:first()').click();
  });

  it('Displays 3 steps', function () {
    cy.get('[data-ft="survey-stepper"] button').should('have.length', 3);
  });

  it('Shows first category in survey', function () {
    cy.get(surveyCategory).should('contain', 'Actionnariat');
  });

  it('Clicking on second category should display second category', function () {
    cy.get('[data-ft="survey-stepper"] span:contains("Evolution organisation"):last()').click();
    cy.get(surveyCategory).should('contain', 'Evolution organisation');
  });

  it('Clicking on Continue / Back should navigate in categories', function () {
    cy.get(continueButton).click();

    cy.get(surveyCategory).should('contain', 'Evolution organisation');

    cy.get(continueButton).click();

    cy.get(surveyCategory).should('contain', 'Effectif');

    cy.get(continueButton).should('not.exist');

    cy.get(backButton).click();

    cy.get(surveyCategory).should('contain', 'Evolution organisation');
  });

  it('Close with close button', function () {
    cy.get('[data-ft="panel-footer"] button:contains("Save")').click();
  });
});

---

@bahmutov commented on Mon Jul 01 2019

@jfaissolle yes, it can quite possibly be due to code-coverage plugin, because it combines the code coverage after each test with previous results - which includes loading a JSON file, merging it, saving updated JSON file. These are slow file operations and can be super slow for large applications where the coverage object is large. I wonder if hasBinary gets triggered when we use cy.task to send coverage info from the plugin to the Node process here https://github.com/cypress-io/code-coverage/blob/master/support.js and if we can pass a flag to NOT verify it (we know the object to be good for example)

[flotwig]

hasBinary recurses through all the keys on the object that's sent through the websocket, so one potential optimization would be to serialize the object (ie, with JSON.stringify) before sending it through the websocket if you already know it's just a plain JavaScript object.

[bahmutov]

Some ideas to improve performance:

• measure it first
• send serialized coverage as a string (see comment above)
• merge the object in the browser and send it once after the tests finish (right now it is sent after each test)

Meanwhile, need to add a warning to this plugin that it WILL affect the test performance due to instrumentation and saving the data

[jfaissolle]

I just want to add that in 3.3.1, there is a performance impact but it is more like 25%. Now, it is like 500%...

[fsmaia]

Any updates on 3.4.0?

[bahmutov]

I don't think 3.4.0 would make any difference

[lukeapage]

I have my own code coverage plugin with cypress and just hit this issue.

with our code base the task takes way over 60 seconds just to try to send the data. I'm not sure thats a great performance metric on the cypress side.

using JSON.stringify and JSON.parse to get into the task resolved the issue - that one thing makes it from timing out after 60 seconds to negligable timing.

[flotwig]

The slowdown comes from the changes introduced in cypress-io/cypress#4407, which allows circular references to be sent over the websocket. There is probably room for optimization in the normalization steps to improve this. Created an issue: cypress-io/cypress#4713

In the meantime, serializing the message manually (with JSON.stringify, for example) will get around this since it won't be checked for circular references.

[bautistaaa]

The slowdown comes from the changes introduced in cypress-io/cypress#4407, which allows circular references to be sent over the websocket. There is probably room for optimization in the normalization steps to improve this. Created an issue: cypress-io/cypress#4713

In the meantime, serializing the message manually (with JSON.stringify, for example) will get around this since it won't be checked for circular references.

Where exactly does JSON.stringify go? Need to workaround this!

[42tg]

The slowdown comes from the changes introduced in cypress-io/cypress#4407, which allows circular references to be sent over the websocket. There is probably room for optimization in the normalization steps to improve this. Created an issue: cypress-io/cypress#4713
In the meantime, serializing the message manually (with JSON.stringify, for example) will get around this since it won't be checked for circular references.

Where exactly does JSON.stringify go? Need to workaround this!

You have to replace the
cy.task('combineCoverage', applicationSourceCoverage)
with
cy.task('combineCoverage', JSON.stringify(applicationSourceCoverage));

and adjust the concrete task accordingly (to parse the now given string to json) this fixed my issues with the performance in 3.4.0.