You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Q]: What is the best way to integrate ember-cli-code-coverage with ember-exam #80
We are currently running ember-cli-code-coverage along with ember-exam parallel runner.
This works great, however, the downside is, that coverage is generated on a console for each ember-exam partition separately.
Is there a way, to merge the reports to obtain one coverage report?
It is definitely possible to coalesce multiple coverage reports together, but we want to make sure we find a good interface to support this across the board (including potential future code coverage solutions). As of right now, I don't have many more thoughts than that.
The trick is that each addon needs knowledge from the other. Ember-exam would need to know that coverage was being run and what to call to combine reports OR this addon would need to know that the test suite had been split and be configured to write coverage to different locations for each run.
An early plan that I think could work:
Add a command to ember-cli-code-coverage that combines several coverage reports
Document how to configure coverage to write to a different folder for each ember-exam run
Add a callback/hook to ember-exam to be able to call a command after the parallel runs with some info about the runs.
Document how to call the combine code coverage command from that hook. Bonus: Make a ember-cli-code-coverage-and-exam addon that does this configuration for the user.
I don't think this addon needs to know about ember-exam at all. Maybe I'm missing part of the picture, but I think it should be sufficient to just do the following:
Expose an option to let ember-cli-code-coverage know there will be more than one coverage report.
When POSTing to /write-coverage, include identifiable information in the payload, such as current test url and maybe browser UA.
Given 1 and 2, have the middleware write each coverage report out to a separate location.
On process exit, merge the multiple reports into a "final" report.
In my opinion, it could be sufficient to have 4 be a separate command that consumers run themselves (since it may be difficult to hook into the process "exit" event).
I also want to note that ember-exam doesn't really do anything special as far as the actual "test process" is concerned; it runs tests in parallel simply by configuring Testem and modifying TestLoader at runtime. This is why I don't think ember-exam should actually have any knowledge of this process, as it is really just sugar on configuration and doesn't do anything with the process output.
I've just been working on this as part of trying to speed up our product build time, should have a PR soon for a merge-coverage command that creates a consolidated report - just needs a bit of clean up.
Activity
kategengler commentedon Nov 30, 2016
From some quick googling it looks like there are some utilities out there for Istanbul, which is the coverage utility that this addon uses.
To get combined coverage, you'll likely have to run an additional command on the results of each partition.
rwjblue commentedon Nov 30, 2016
ya, that makes sense
We should track down the required commands / tools and figure out how to make them play nicely together. We have a few options there:
/cc @trentmwillis for his thoughts/opinions
trentmwillis commentedon Nov 30, 2016
There is an open issue on ember-exam to explore this as well: ember-cli/ember-exam#18. There's not much there, but a conversation with @stefanpenner about that issue led to this RFC: ember-cli/rfcs#57.
It is definitely possible to coalesce multiple coverage reports together, but we want to make sure we find a good interface to support this across the board (including potential future code coverage solutions). As of right now, I don't have many more thoughts than that.
kategengler commentedon Nov 30, 2016
The trick is that each addon needs knowledge from the other. Ember-exam would need to know that coverage was being run and what to call to combine reports OR this addon would need to know that the test suite had been split and be configured to write coverage to different locations for each run.
An early plan that I think could work:
trentmwillis commentedon Dec 1, 2016
I don't think this addon needs to know about ember-exam at all. Maybe I'm missing part of the picture, but I think it should be sufficient to just do the following:
POSTing to/write-coverage, include identifiable information in the payload, such as current test url and maybe browser UA.In my opinion, it could be sufficient to have 4 be a separate command that consumers run themselves (since it may be difficult to hook into the process "exit" event).
I also want to note that ember-exam doesn't really do anything special as far as the actual "test process" is concerned; it runs tests in parallel simply by configuring Testem and modifying
TestLoaderat runtime. This is why I don't think ember-exam should actually have any knowledge of this process, as it is really just sugar on configuration and doesn't do anything with the process output.kategengler commentedon Dec 1, 2016
I had wild misconceptions about how ember-exam worked; your plan sounds doable.
jdenly commentedon Dec 1, 2016
I've just been working on this as part of trying to speed up our product build time, should have a PR soon for a merge-coverage command that creates a consolidated report - just needs a bit of clean up.
jdenly commentedon Dec 2, 2016
There is a PR available here:
#83
kategengler commentedon Jan 5, 2017
Should be solved by #83