Performance test plan

[miroslavstastny]

Perf measuring

This document describes workflow used to measure (and avoid regressions in) performance / bundle size.

Scenarios

Scenario	Setup	Question it answers
Minimal	Only required props	What is the base overhead cost of a component?
Slots	All slot props	What is the cost of slot prop logic?
InlineStyles	Inline style overrides	What is the cost of inline style overrides?
InlineVariables	Inline variable overrides	What is the cost of inline variable overrides?
NoTheme	No theming system applied	What is the cost of the theming system?

• Add Minimal scenarios perf: add base perf examples #2164
• Add Slots scenarios
• Add InlineStyles scenarios
• Add InlineVariables scenarios
• Add NoTheme scenarios

Workflow overview

1. run perf tests as part of CI for every PR build,
2. store the results to external storage,
3. in every PR, compare current results with results from a fork point,
4. (optional), with every release, get current results from external storage, push them to git,
5. show numbers (with history and charts) in docsite.

Individual steps

Run perf tests

• yarn build:stats to measure bundle size locally
• run bundle size tests in CI
• yarn perf to measure performance locally
• run perf tests in CI (will the results be stable enough?)
• add perf examples for all components perf: add base perf examples #2164
• add perf examples for compound pages (See Semantic-UI/React layouts)
• add conformance test for perf tests

External storage

• spin-up external storage

Let's start with a nosql database, with following keys:

• commit hash
  • to map 1:1 from git commit to the result
• git branch
  • to be able to display stats for a branch without git
  • may not work for branches other than master as branch names can be reused
  • but is it required for a branch?
  • works for master
• ci build number
  • to order results on a single branch
  • to distinguish multiple builds on a single commit

Comparing results in every PR

• save current results to external storage
• for PR builds, compare current results with fork-point results, add the diff to the PR

As part of each PR build, perf tests are executed, results stored to the external storage and compared with master.

To find correct master commit:

git merge-base --fork-point master

Note: This does not work for PR to other branches than master, not tested for PRs from forks!

We will use DangerJS to compare the results and add them as a comment to the PR.

In the first version it will just log the difference without enforcing any policy.

Master builds

Master builds will just store the results to the external storage without performing any check.

Commit current results to git with every public release

This is an optional step.

We plan to display performance charts in docsite. But that adds dependency on the external storage. To avoid that dependency, as part of the release we can get appropriate results from external storage and commit them to main repo.

Display results in docsite.

• add perf page with summary results and history (See bundlephobia.com for inspiration)
• for each component, add bundle size to the component page
• for each component, add perf results to the component page

[layershifter]

My proposal is to simplify this:

• we have CIRCLE_PULL_REQUEST in every build, just store results for PR, https://circleci.com/docs/2.0/env-vars/#built-in-environment-variables
  I don't see any reasons to keep multiple results for one PR
• on CIRCLE_BRANCH==master && CIRCLE_TAG store results for a release

[levithomason]

@miroslavstastny we should work toward closing this PR in our perf workstream. Let's keep it up to date as we move forward.