Repository navigation
Pull Request Tests with [feature] tag in commits should report failed in CI when "Feature" tests fail #4641
Description
I have had 2 pull requests merged where I missed the failing tests due to Travis CI reporting the build passing when tests actually failed. It is partially my fault for not being completely attentive to the test results, but in my defense there are a very large number of feature tests run and it is easy to lose track in the sea of results.
IMO, the CI should report the build failed when when the [feature] tag is present in the commit and any "Feature" test fails. This will make it easier to spot when something goes wrong.
Example PR #4494 Travis CI Passing with failed test https://travis-ci.org/PowerShell/PowerShell/jobs/266123025
Reacted by Ilya
Activity
[-]Pull Request Tests with [feature] tag should report failed in CI when "Feature" tests fail[/-][+]Pull Request Tests with [feature] tag in commits should report failed in CI when "Feature" tests fail[/+]TravisEz13 commentedon Aug 22, 2017
This shouldn't be specific to the feature tag. It indicates there is some bug in
Test-PSPesterResultsin build.psm1. I tried a simple repro locally, and couldn't repro the issue.markekraus commentedon Aug 22, 2017
I had a really fun one #4622. https://travis-ci.org/PowerShell/PowerShell/jobs/266586350#L8280 That one was an unhandled exception in a Job, that killed the calling process, and the pester process, and the test "passed". That's when I started looking very closely at the test results.
TravisEz13 commentedon Aug 22, 2017
This is interesting... pester must write the result as it is going because
Test-PSPesterResultswill fail if the results file is missing.I think we will have to switch to running
Invoke-Pesterwith-PassThruand use those results to determine if the tests passed or failed.TravisEz13 commentedon Aug 22, 2017
I found one issue...
Start-PSPesteris not checking the exit code from powershell.