Skip to content

[Q]: What is the best way to integrate ember-cli-code-coverage with ember-exam #80

Closed

Description

@gustaff-weldon

We are currently running ember-cli-code-coverage along with ember-exam parallel runner.
This works great, however, the downside is, that coverage is generated on a console for each ember-exam partition separately.

Is there a way, to merge the reports to obtain one coverage report?

Activity

  1. kategengler commented on Nov 30, 2016

    @kategengler
    Collaborator

    From some quick googling it looks like there are some utilities out there for Istanbul, which is the coverage utility that this addon uses.

    To get combined coverage, you'll likely have to run an additional command on the results of each partition.

  2. rwjblue commented on Nov 30, 2016

    @rwjblue
    Collaborator

    ya, that makes sense

    We should track down the required commands / tools and figure out how to make them play nicely together. We have a few options there:

    • Make ember-cli-code-coverage detect ember-exam and do the right thing automatically
    • Make ember-exam detect ember-cli-code-coverage and do something...
    • Document in both the ember-exam and ember-cli-code-coverage README's what the solution is

    /cc @trentmwillis for his thoughts/opinions

  3. trentmwillis commented on Nov 30, 2016

    @trentmwillis

    There is an open issue on ember-exam to explore this as well: ember-cli/ember-exam#18. There's not much there, but a conversation with @stefanpenner about that issue led to this RFC: ember-cli/rfcs#57.

    It is definitely possible to coalesce multiple coverage reports together, but we want to make sure we find a good interface to support this across the board (including potential future code coverage solutions). As of right now, I don't have many more thoughts than that.

  4. kategengler commented on Nov 30, 2016

    @kategengler
    Collaborator

    The trick is that each addon needs knowledge from the other. Ember-exam would need to know that coverage was being run and what to call to combine reports OR this addon would need to know that the test suite had been split and be configured to write coverage to different locations for each run.

    An early plan that I think could work:

    • Add a command to ember-cli-code-coverage that combines several coverage reports
    • Document how to configure coverage to write to a different folder for each ember-exam run
    • Add a callback/hook to ember-exam to be able to call a command after the parallel runs with some info about the runs.
    • Document how to call the combine code coverage command from that hook. Bonus: Make a ember-cli-code-coverage-and-exam addon that does this configuration for the user.
  5. trentmwillis commented on Dec 1, 2016

    @trentmwillis

    I don't think this addon needs to know about ember-exam at all. Maybe I'm missing part of the picture, but I think it should be sufficient to just do the following:

    1. Expose an option to let ember-cli-code-coverage know there will be more than one coverage report.
    2. When POSTing to /write-coverage, include identifiable information in the payload, such as current test url and maybe browser UA.
    3. Given 1 and 2, have the middleware write each coverage report out to a separate location.
    4. On process exit, merge the multiple reports into a "final" report.

    In my opinion, it could be sufficient to have 4 be a separate command that consumers run themselves (since it may be difficult to hook into the process "exit" event).


    I also want to note that ember-exam doesn't really do anything special as far as the actual "test process" is concerned; it runs tests in parallel simply by configuring Testem and modifying TestLoader at runtime. This is why I don't think ember-exam should actually have any knowledge of this process, as it is really just sugar on configuration and doesn't do anything with the process output.

  6. kategengler commented on Dec 1, 2016

    @kategengler
    Collaborator

    I had wild misconceptions about how ember-exam worked; your plan sounds doable.

  7. jdenly commented on Dec 1, 2016

    @jdenly
    Contributor

    I've just been working on this as part of trying to speed up our product build time, should have a PR soon for a merge-coverage command that creates a consolidated report - just needs a bit of clean up.

  8. jdenly commented on Dec 2, 2016

    @jdenly
    Contributor

    There is a PR available here:

    #83

  9. kategengler commented on Jan 5, 2017

    @kategengler
    Collaborator

    Should be solved by #83

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions