Give the ember tests some idiomatic love.. #7
Conversation
- Utilize ember's run-loop as it was meant to be utilized. - Use minified ember build, as this is closer to production. Please note, this is a quite unnatural test. Forcing typically temporally distinct operations (such as user clicks) to be synchronous, grinds against some frameworks architecture. I do not believe these tests are trying explicitly to mis-represent, but I fear this approach is flawed. This commit allows ember's run-loop to do what it was Intended todo, which helps dramatically. I would love to find a more natural method of testing these types of scenarios.
|
one example run: seems to be between 500ms -> 750ms depending on what appears to be common jitter. |
|
Hey Stefan, thanks a lot for the suggestion. This is really impressive perf gain, but here's my thoughts: To my understanding, the benchmark intends to simulate user actions. Admitted, firing all the events in a synchronous for loop is unlikely to happen for a real user. But in my opinion, the point of the for loop is not stressing the runtime but rather increasing the sample size to counter fluctuation (potentially caused by GC) and browser timing accuracy issues. Logically, each iteration should be considered a separate event. What we actually want to measure is the time needed for the framework to fully propagate the changes and manipulate the DOM for just one iteration. In that sense, the for loop is a test fixture and all framework specific code should happen inside, and that is the case for every other framework in the test. Allowing Ember to wrap the entire loop in its own run loop seems to be giving it unfair advantage. |
|
I believe we should strive for an accurate and healthy benchmark or strategy of benchmarks. I would love to brain storm with you if you have time. Ideas:
I wonder if using microtime, and running interactions at the healthy multiple of the rate users would natural interact with an application would yield much more valuable results. This should result in realistic GC pressure, and DOM pressure. Additionally, the N we should increase, is the cost PER user interaction, rather then increasing the number of interactions. An example would be. A User triggering an action causing:
We would then run with a series of distinct and separate tests were we grow N from x -> y [1...1000] Allowing for healthy pauses, which should mimic user interactions, puttying realistic stresses on the run-time. Another aspect is, data entering the app via another strategy, such as websocket, or xhr. For this we likely need N + Y, N is the amount of change, and Y is the frequency. I believe this theme is more aligned with reality. Likely can be thought of as the difference between using AB to test a webserver, and an httperf script that mimics reality. |
|
Also, if you are interested. Forcing a full flush after each event resulted in some slow down, but not nearly as much as the original Averaging around: EmberJS-TodoMVC : Adding100Items : Sync: 310.23699999786913 ms Ranging from 800ms -> 1100ms |
|
Another perspective would be, what if we tested file-system writes with 0 buffers, forcing sync to disk for each byte written. This would be entirely un-realistic and if made faster, would only result in degraded performance for end-users. |
|
For measuring each iteration, perhaps something like https://github.com/btford/zone.js/ may come in handy? |
|
@duckbox interesting, ya it may. |
|
Closing this because the original PR no longer applies. Still open for discussion regarding a more natural benchmark - although that's probably out of scope for this repo. |
Add RTL support ..
Please note, this is a quite unnatural test. Forcing typically temporally distinct operations (such as user clicks) to be synchronous, grinds against some frameworks architecture. I do not believe these tests are trying explicitly to mis-represent, but
I fear this approach is flawed. This commit allows ember's run-loop to do what it was
Intended todo, which helps dramatically.
I would love to find a more natural method of testing these types of scenarios.