Skip to content

Specifying --path option when generating coverage results in partial/truncated reports. #309

Open

Description

@alexabreu

In order to lower our CI costs, we're building once and sharing that build across multiple partition test runs (via ember exam). All seems to work fine, but the lcov.info reports being generated across each test run are both identical and incomplete.

Simply removing the path option, therefore forcing a fresh build prior to running tests, remedies this problem and results in correct coverage reports. This, however, has the unwanted side effect of each test run worker being billed for the added build time.

Did I miss a configuration option to handle this specific use case?

Incomplete lcov.info
TN:
SF:app/formats.js
FNF:0
FNH:0
LF:0
LH:0
BRF:0
BRH:0
end_of_record
TN:
SF:app/resolver.js
FNF:0
FNH:0
LF:0
LH:0
BRF:0
BRH:0
end_of_record
TN:
SF:app/components/esri-icon.js
FNF:0
FNH:0
LF:0
LH:0
BRF:0
BRH:0
end_of_record
Incomplete coverage-summary.json
{"total": {"lines":{"total":0,"covered":0,"skipped":0,"pct":100},"statements":{"total":0,"covered":0,"skipped":0,"pct":100},"functions":{"total":0,"covered":0,"skipped":0,"pct":100},"branches":{"total":0,"covered":0,"skipped":0,"pct":100}}
,"app/formats.js": {"lines":{"total":0,"covered":0,"skipped":0,"pct":0},"functions":{"total":0,"covered":0,"skipped":0,"pct":0},"statements":{"total":0,"covered":0,"skipped":0,"pct":0},"branches":{"total":0,"covered":0,"skipped":0,"pct":0}}
,"app/resolver.js": {"lines":{"total":0,"covered":0,"skipped":0,"pct":0},"functions":{"total":0,"covered":0,"skipped":0,"pct":0},"statements":{"total":0,"covered":0,"skipped":0,"pct":0},"branches":{"total":0,"covered":0,"skipped":0,"pct":0}}
,"app/components/esri-icon.js": {"lines":{"total":0,"covered":0,"skipped":0,"pct":0},"functions":{"total":0,"covered":0,"skipped":0,"pct":0},"statements":{"total":0,"covered":0,"skipped":0,"pct":0},"branches":{"total":0,"covered":0,"skipped":0,"pct":0}}
}

Activity

  1. kategengler commented on Apr 7, 2021

    @kategengler
    Collaborator

    Can you confirm:

  2. barryofguilder commented on Sep 1, 2021

    @barryofguilder

    @kategengler We just ran into this in our Ember App and I can confirm that we've done all 3 of those steps. I'm not sure what all information you need, but here's a snippet of the npm scripts we're using in our GitHub Action workflow:

    # Builds the app, which all of the exam partions use
    # Since this is done in a separate GitHub Action job than ember-exam, I'm not specifying the output-path as
    # that's being handle within our action using upload-artifact/download-artifact.
    COVERAGE=true ember build --environment=test
    
    #  exam script:
    COVERAGE=true ember exam --query=nolint --split=4 --parallel=1
    
    # Called like this in the GitHub Action (done 4 times to cover all partitions):
    npm run exam -- --partition=1 --path=dist
    
    # Called in our SonarCloud step
    npm run coverage-merge
    
  3. kategengler commented on Sep 1, 2021

    @kategengler
    Collaborator

    @barryofguilder
    If you follow the same steps locally, does it work? Before you run coverage-merge, you should see several coverage123abc directories.

    Could you add a test app here https://github.com/kategengler/ember-cli-code-coverage/tree/master/test-packages with a reproduction?

  4. barryofguilder commented on Sep 1, 2021

    @barryofguilder

    @kategengler

    I was able to reproduce this locally. I do see several coverage_123abc directories (one for each ember-exam run), but they barely contain any coverage info. Way less than what we had in each partition before we added the --path=dist param.

    I will try and get a test app uploaded to the repo tomorrow.

  5. barryofguilder commented on Sep 7, 2021

    @barryofguilder

    I'm still trying to get a test app together to show the issue. I'm struggling to have ember exam partition the tests when there's not many tests. It seems to be running all the tests in each partition.

  6. ndekeister-us commented on Sep 14, 2021

    @ndekeister-us

    @barryofguilder may i ask you what versions of ember-cli-code-coverage + ember-exam are you using? Tried to reproduce your issue with latest versions (ember-cli-code-coverage@1.0.3 and ember-exam@6.1.0) in an Ember app having a large test suite (1.5k+ tests) and i'm not able to reproduce, did following steps

    // exam script -> COVERAGE=true ember exam --query=nolint --split=4 --parallel=1
    COVERAGE=true ember build --environment=test
    npm run exam -- --partition=1 --path=dist
    npm run exam -- --partition=2 --path=dist
    npm run exam -- --partition=3 --path=dist
    npm run exam -- --partition=4 --path=dist
    npm run coverage-merge
    

    Resulting of having same coverage data as running directly COVERAGE=true ember exam --query=nolint

  7. barryofguilder commented on Sep 14, 2021

    @barryofguilder

    @ndekeister-us sure, here are the versions we are using:

    • ember-exam: "^6.1.0"
    • ember-cli-code-coverage: "^1.0.3"
    • ember-cli: "~3.25.2"
    • ember-source: "3.25.1"
  8. ndekeister-us commented on Sep 15, 2021

    @ndekeister-us

    Thanks, made some try in multiple apps but can't success to reproduce even if we have exact same packages & versions, hope you will success to reproduce it in a test app 🤞

  9. alexabreu commented on Feb 7, 2022

    @alexabreu
    Author

    Happy to report that the root cause of this bug is due to this TypeScript issue and was indeed resolved by upgrading to v2.0.0.beta-4.

  10. barryofguilder commented on Feb 7, 2022

    @barryofguilder

    Happy to report that the root cause of this bug is due to this TypeScript issue and was indeed resolved by upgrading to v2.0.0.beta-4.

    Ohhh, that would explain why I was never able to reproduce. I wasn't using TypeScript in any of the test apps.

  11. alexabreu commented on Feb 11, 2022

    @alexabreu
    Author

    So I may have spoken too soon... While it does seem beta-4 improves the situation, we're seeing empty coverage reports for a subset of our partitions. For instance, we'll get 7/10 valid coverage reports and 3 empty files. Strangely, it seems the number of valid reports changes across builds. Does this sound like it's related to #349?

    I've tried:

    • Eliminating the use of the load-balance flag
    • Increasing/decreasing the value used for split flag

    What else might I try?

    This is what our script calls look like:

    coverage.js

    module.exports = {
      parallel: true,
    }
    
    

    package.json

    ...
    "test:ci": "ember exam --parallel=2 --load-balance --split=10 --test-port 0",
    "posttest:ci": "npx ember coverage-merge",
    ...
    

    COVERAGE=true npm run test:ci -- --partition=7 --path='test-dist'

  12. LanceStasinski commented on Jun 30, 2022

    @LanceStasinski

    Hi @kategengler, has there been any update regarding this issue? I'm working with the same configuration as @alexabreu and seeing empty coverage reports as described in the previous comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions