Create benchmark for fluent.runtime (fixes #85) #87
Conversation
This has a benchmark for imports, one for bundle generation, and one for a pseudo template. The data here should inform down-stream consumers what to cache.
|
Luke, I guess you're the best to review this? I intentionally didn't try to solve for the case that your benchmarks served, i.e., compare fluent in a hot loop to gettext in a hot loop. I'd rather try to see what we intend to look at, and then see if there are equivalents in gettext. Also, we should run these in automation, but I'm not yet sure how to best do that. |
|
This all looks good to me, and I was able to run the benchmarks following your instructions. Regarding running in automation, it is relatively easy to add things to the Travis matrix without adding all combinations - https://docs.travis-ci.com/user/customizing-the-build/#explicitly-including-jobs But I have no idea what should be added - we presumably want something that will track the performance of these things over time? Is that even possible to do that reliably using cloud services? I found an article on benchmarking in the cloud which suggests not. I found this issue - travis-ci/travis-ci#352 - but it didn't turn up anything immediately useful. |
|
Thanks for the review. For automation, I think I want to wait on #70 and then add running the benchmarks as single jobs. Once that works, we can change the automation on PRs to run the benchmark on both master and PR, and compare the two. I think running the benchmark twice on the same machine is the only plausible way to get numbers to compare. |
I haven't figured out how to actually run this on travis.
I'd love if this would be extra jobs, maybe just on py2.7, py3.7, pypy2, pypy3?
Reminds me, we're not testing py3.7 at all yet, we need to. Which is partly #70.