Skip to content

Add perf testing #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Sep 24, 2020
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .github/workflows/perf.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: Performance

on: [pull_request]

jobs:
perf:
runs-on: Ubuntu-18.04
steps:
- name: Checkout
uses: actions/checkout@master
with:
fetch-depth: 1
- name: Run Benchmark
run: |
git clone https://github.com/kylef/swiftenv.git ~/.swiftenv
export SWIFTENV_ROOT="$HOME/.swiftenv"
export PATH="$SWIFTENV_ROOT/bin:$PATH"
eval "$(swiftenv init -)"
swiftenv install $TOOLCHAIN_DOWNLOAD
node ci/perf-tester/index.js
env:
TOOLCHAIN_DOWNLOAD: https://github.com/swiftwasm/swift/releases/download/swift-wasm-5.3-SNAPSHOT-2020-08-10-a/swift-wasm-5.3-SNAPSHOT-2020-08-10-a-linux.tar.gz
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,7 @@ benchmark_setup:
.PHONY: run_benchmark
run_benchmark:
cd IntegrationTests && make -s run_benchmark

.PHONY: perf-tester
perf-tester:
cd ci/perf-tester && npm install && npm run build
15 changes: 15 additions & 0 deletions ci/perf-tester/index.js

Large diffs are not rendered by default.

4,823 changes: 4,823 additions & 0 deletions ci/perf-tester/package-lock.json

Large diffs are not rendered by default.

15 changes: 15 additions & 0 deletions ci/perf-tester/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"private": true,
"main": "index.js",
"scripts": {
"build": "microbundle -f cjs --define 'navigator={}' --compress --no-sourcemap --target node src/index.js",
"test": "jest"
},
"license": "MIT",
"devDependencies": {
"@actions/core": "^1.2.2",
"@actions/exec": "^1.0.3",
"@actions/github": "^2.0.1",
"microbundle": "^0.12.0-next.8"
}
}
203 changes: 203 additions & 0 deletions ci/perf-tester/src/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
import { setFailed, startGroup, endGroup, debug } from '@actions/core';
import { GitHub, context } from '@actions/github';
import { exec } from '@actions/exec';
import { getInput, runBenchmark, averageBenchmarks, toDiff, diffTable, toBool } from './utils.js';

async function run(octokit, context, token) {
const { number: pull_number } = context.issue;

const pr = context.payload.pull_request;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we have a .prettierrc file added to the repository and make formatting consistent here with the rest of the .ts and .js files that use 4 spaces for indentation?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Do you think it makes sense to add a package.json to the root of the project so the various JS parts (IntegrationTests, Runtime, perf-tester) each use the same version of Prettier?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a clear preference here. @kateinoigakukun WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's reasonable to put package.json on the project root.

try {
debug('pr' + JSON.stringify(pr, null, 2));
} catch (e) { }
if (!pr) {
throw Error('Could not retrieve PR information. Only "pull_request" triggered workflows are currently supported.');
}

console.log(`PR #${pull_number} is targetted at ${pr.base.ref} (${pr.base.sha})`);

const buildScript = getInput('build-script');
startGroup(`[current] Build using '${buildScript}'`);
await exec(buildScript);
endGroup();

startGroup(`[current] Running benchmark`);
const newBenchmarks = await Promise.all([runBenchmark(), runBenchmark()]).then(averageBenchmarks);
endGroup();

startGroup(`[base] Checkout target branch`);
let baseRef;
try {
baseRef = context.payload.base.ref;
if (!baseRef) throw Error('missing context.payload.pull_request.base.ref');
await exec(`git fetch -n origin ${context.payload.pull_request.base.ref}`);
console.log('successfully fetched base.ref');
} catch (e) {
console.log('fetching base.ref failed', e.message);
try {
await exec(`git fetch -n origin ${pr.base.sha}`);
console.log('successfully fetched base.sha');
} catch (e) {
console.log('fetching base.sha failed', e.message);
try {
await exec(`git fetch -n`);
} catch (e) {
console.log('fetch failed', e.message);
}
}
}

console.log('checking out and building base commit');
try {
if (!baseRef) throw Error('missing context.payload.base.ref');
await exec(`git reset --hard ${baseRef}`);
}
catch (e) {
await exec(`git reset --hard ${pr.base.sha}`);
}
endGroup();

startGroup(`[base] Build using '${buildScript}'`);
await exec(buildScript);
endGroup();

startGroup(`[base] Running benchmark`);
const oldBenchmarks = await Promise.all([runBenchmark(), runBenchmark()]).then(averageBenchmarks);
endGroup();

const diff = toDiff(oldBenchmarks, newBenchmarks);

const markdownDiff = diffTable(diff, {
collapseUnchanged: true,
omitUnchanged: false,
showTotal: true,
minimumChangeThreshold: parseInt(getInput('minimum-change-threshold'), 10)
});

let outputRawMarkdown = false;

const commentInfo = {
...context.repo,
issue_number: pull_number
};

const comment = {
...commentInfo,
body: markdownDiff + '\n\n<a href="https://github.com/j-f1/performance-action"><sub>performance-action</sub></a>'
};

if (toBool(getInput('use-check'))) {
if (token) {
const finish = await createCheck(octokit, context);
await finish({
conclusion: 'success',
output: {
title: `Compressed Size Action`,
summary: markdownDiff
}
});
}
else {
outputRawMarkdown = true;
}
}
else {
startGroup(`Updating stats PR comment`);
let commentId;
try {
const comments = (await octokit.issues.listComments(commentInfo)).data;
for (let i = comments.length; i--;) {
const c = comments[i];
if (c.user.type === 'Bot' && /<sub>[\s\n]*performance-action/.test(c.body)) {
commentId = c.id;
break;
}
}
}
catch (e) {
console.log('Error checking for previous comments: ' + e.message);
}

if (commentId) {
console.log(`Updating previous comment #${commentId}`)
try {
await octokit.issues.updateComment({
...context.repo,
comment_id: commentId,
body: comment.body
});
}
catch (e) {
console.log('Error editing previous comment: ' + e.message);
commentId = null;
}
}

// no previous or edit failed
if (!commentId) {
console.log('Creating new comment');
try {
await octokit.issues.createComment(comment);
} catch (e) {
console.log(`Error creating comment: ${e.message}`);
console.log(`Submitting a PR review comment instead...`);
try {
const issue = context.issue || pr;
await octokit.pulls.createReview({
owner: issue.owner,
repo: issue.repo,
pull_number: issue.number,
event: 'COMMENT',
body: comment.body
});
} catch (e) {
console.log('Error creating PR review.');
outputRawMarkdown = true;
}
}
}
endGroup();
}

if (outputRawMarkdown) {
console.log(`
Error: performance-action was unable to comment on your PR.
This can happen for PR's originating from a fork without write permissions.
You can copy the size table directly into a comment using the markdown below:
\n\n${comment.body}\n\n
`.replace(/^(\t| )+/gm, ''));
}

console.log('All done!');
}


// create a check and return a function that updates (completes) it
async function createCheck(octokit, context) {
const check = await octokit.checks.create({
...context.repo,
name: 'Compressed Size',
head_sha: context.payload.pull_request.head.sha,
status: 'in_progress',
});

return async details => {
await octokit.checks.update({
...context.repo,
check_run_id: check.data.id,
completed_at: new Date().toISOString(),
status: 'completed',
...details
});
};
}

(async () => {
try {
const token = getInput('repo-token', { required: true });
const octokit = new GitHub(token);
await run(octokit, context, token);
} catch (e) {
setFailed(e.message);
}
})();
Loading