Jenkins › Jenkins users

MSTest Plugin doesn't fail the job on failing test.

Classic

List

Threaded

6 messages Options

barrett

MSTest Plugin doesn't fail the job on failing test.

I would like the MSTest plugin to fail the job when a test fails. Is this feasible ?

Mark Waite

Re: MSTest Plugin doesn't fail the job on failing test.

You might consider the comments in this thread before you mark a job as failed due to test failures:

http://jenkins.361315.n4.nabble.com/plugin-A-build-with-failed-tests-should-be-marked-as-Unstable-or-Failure-td3585663.html

If that guidance doesn't meet your needs and you decide that you still want to fail the build on test failures rather than mark the build "Unstable" on test failures, then I think the easiest way is to invoke mstest.exe directly (instead of using the MSTest plugin to launch it) and then have the MSTest batch file exit with a non zero exit code if any test fails.

Mark Waite

From: bearrito <[hidden email]>
To: [hidden email]
Sent: Thursday, September 6, 2012 10:04 AM
Subject: MSTest Plugin doesn't fail the job on failing test.

I would like the MSTest plugin to fail the job when a test fails. Is this feasible ?

William Soula

RE: MSTest Plugin doesn't fail the job on failing test.

Is there anything in the log to denote a failure? You could use the log parser to fail if it finds test failures. But I agree with the guy below they should be marked unstable, which can also be done with the log parser plugin.

Will

From: [hidden email] [[hidden email]] on behalf of Mark Waite [[hidden email]]
Sent: Thursday, September 06, 2012 12:50 PM
To: [hidden email]
Subject: Re: MSTest Plugin doesn't fail the job on failing test.

You might consider the comments in this thread before you mark a job as failed due to test failures:

http://jenkins.361315.n4.nabble.com/plugin-A-build-with-failed-tests-should-be-marked-as-Unstable-or-Failure-td3585663.html

Mark Waite

From: bearrito <[hidden email]>
To: [hidden email]
Sent: Thursday, September 6, 2012 10:04 AM
Subject: MSTest Plugin doesn't fail the job on failing test.

I would like the MSTest plugin to fail the job when a test fails. Is this feasible ?

barrett

Re: MSTest Plugin doesn't fail the job on failing test.

In reply to this post by Mark Waite

Not marking a build as failed with failing unit tests doesn't make any sense to me. We fail the build if a compile fails so why not with tests?

What if I was working in a dynamic/interpreted language and so couldn't rely on compiler/type system to find what would be guaranteed run-time errors?

I'd prefer to use the MSTest plugin because the MSTest plugin performs the conversion from the .trx test output to the junit format that jenkins understands.

-barrett

On Thursday, September 6, 2012 1:51:00 PM UTC-4, Mark Waite wrote:

You might consider the comments in this thread before you mark a job as failed due to test failures:

http://jenkins.361315.n4.nabble.com/plugin-A-build-with-failed-tests-should-be-marked-as-Unstable-or-Failure-td3585663.html

If that guidance doesn't meet your needs and you decide that you still want to fail the build on test failures rather than mark the build "Unstable" on test failures, then I think the easiest way is to invoke mstest.exe directly (instead of using the MSTest plugin to launch it) and then have the MSTest batch file exit with a non zero exit code if any test fails.

Mark Waite

From: bearrito <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="ervc_NoUAHcJ">j.barrett...@...>
To: <a href="javascript:" target="_blank" gdf-obfuscated-mailto="ervc_NoUAHcJ">jenkins...@googlegroups.com
Sent: Thursday, September 6, 2012 10:04 AM
Subject: MSTest Plugin doesn't fail the job on failing test.

I would like the MSTest plugin to fail the job when a test fails. Is this feasible ?

Mark Waite

Re: MSTest Plugin doesn't fail the job on failing test.

If that's what you want, then that's a good thing to do. I prefer having "Unstable" mean "code compiled, one or more tests failed" so I gained more meaning from the "red" / "yellow" / "blue" states of the build. If you gain more meaning by having two states, "red" and "blue" for your results, that's OK too.

I had not considered the case where you truly have no compilation phase at all, since all the projects I use involve a mixture of compiled and dynamic languages, with the dynamic languages typically not being invoked until after the compilation phase is complete.

The cases where I've needed to run MSTest (a few years ago now) also required that I be able to process more than one MSTest output file at a time, and that capability was not available at that time (added in version 0.7). I see that the capability to process multiple files with wildcards has been available for over a year, so I'm glad it meets your needs. Using the MSTest plugin seems like a good way to do it.

I think the idea of causing your MSTest invocation to return a non-zero exit code is still a reasonable approach to give you the failure you're seeking when a test fails. You might also consider using the MSTestRunner plugin to launch MSTest ( https://wiki.jenkins-ci.org/display/JENKINS/MSTestRunner+Plugin ) to see if it has a way to mark the job as failed when a test fails. Yet another alternative would be to investigate the xUnit plugin to see if it will give you the flexibility you want.

Mark Waite

From: bearrito <[hidden email]>
To: [hidden email]
Cc: Mark Waite <[hidden email]>
Sent: Friday, September 7, 2012 11:35 AM
Subject: Re: MSTest Plugin doesn't fail the job on failing test.

Not marking a build as failed with failing unit tests doesn't make any sense to me. We fail the build if a compile fails so why not with tests?
What if I was working in a dynamic/interpreted language and so couldn't rely on compiler/type system to find what would be guaranteed run-time errors?

I'd prefer to use the MSTest plugin because the MSTest plugin performs the conversion from the .trx test output to the junit format that jenkins understands.

-barrett

On Thursday, September 6, 2012 1:51:00 PM UTC-4, Mark Waite wrote:
You might consider the comments in this thread before you mark a job as failed due to test failures:

http://jenkins.361315.n4. nabble.com/plugin-A-build- with-failed-tests-should-be- marked-as-Unstable-or-Failure- td3585663.html

If that guidance doesn't meet your needs and you decide that you still want to fail the build on test failures rather than mark the build "Unstable" on test failures, then I think the easiest way is to invoke mstest.exe directly (instead of using the MSTest plugin to launch it) and then have the MSTest batch file exit with a non zero exit code if any test fails.

Mark Waite

From: bearrito <j.barrett...@... >
To: jenkins...@googlegroups. com
Sent: Thursday, September 6, 2012 10:04 AM
Subject: MSTest Plugin doesn't fail the job on failing test.

I would like the MSTest plugin to fail the job when a test fails. Is this feasible ?

barrett

Re: MSTest Plugin doesn't fail the job on failing test.

Obviously, the choice is a bit ideological and context dependent. I wasn't trying to be contentious.

I'll have a look at some of the options you mentioned.

On Friday, September 7, 2012 5:10:56 PM UTC-4, Mark Waite wrote:

If that's what you want, then that's a good thing to do. I prefer having "Unstable" mean "code compiled, one or more tests failed" so I gained more meaning from the "red" / "yellow" / "blue" states of the build. If you gain more meaning by having two states, "red" and "blue" for your results, that's OK too.

I had not considered the case where you truly have no compilation phase at all, since all the projects I use involve a mixture of compiled and dynamic languages, with the dynamic languages typically not being invoked until after the compilation phase is complete.

The cases where I've needed to run MSTest (a few years ago now) also required that I be able to process more than one MSTest output file at a time, and that capability was not available at that time (added in version 0.7). I see that the capability to process multiple files with wildcards has been available for over a year, so I'm glad it meets your needs. Using the MSTest plugin seems like a good way to do it.

I think the idea of causing your MSTest invocation to return a non-zero exit code is still a reasonable approach to give you the failure you're seeking when a test fails. You might also consider using the MSTestRunner plugin to launch MSTest ( https://wiki.jenkins-ci.org/display/JENKINS/MSTestRunner+Plugin ) to see if it has a way to mark the job as failed when a test fails. Yet another alternative would be to investigate the xUnit plugin to see if it will give you the flexibility you want.

Mark Waite

From: bearrito <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="oRsYG9c7qYAJ">j.barrett...@...>
To: <a href="javascript:" target="_blank" gdf-obfuscated-mailto="oRsYG9c7qYAJ">jenkins...@googlegroups.com
Cc: Mark Waite <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="oRsYG9c7qYAJ">mark...@...>
Sent: Friday, September 7, 2012 11:35 AM
Subject: Re: MSTest Plugin doesn't fail the job on failing test.

Not marking a build as failed with failing unit tests doesn't make any sense to me. We fail the build if a compile fails so why not with tests?
What if I was working in a dynamic/interpreted language and so couldn't rely on compiler/type system to find what would be guaranteed run-time errors?

I'd prefer to use the MSTest plugin because the MSTest plugin performs the conversion from the .trx test output to the junit format that jenkins understands.

-barrett

On Thursday, September 6, 2012 1:51:00 PM UTC-4, Mark Waite wrote:
You might consider the comments in this thread before you mark a job as failed due to test failures:

http://jenkins.361315.n4. nabble.com/plugin-A-build- with-failed-tests-should-be- marked-as-Unstable-or-Failure- td3585663.html

If that guidance doesn't meet your needs and you decide that you still want to fail the build on test failures rather than mark the build "Unstable" on test failures, then I think the easiest way is to invoke mstest.exe directly (instead of using the MSTest plugin to launch it) and then have the MSTest batch file exit with a non zero exit code if any test fails.

Mark Waite

From: bearrito <[hidden email] >
To: jenkins...@googlegroups. com
Sent: Thursday, September 6, 2012 10:04 AM
Subject: MSTest Plugin doesn't fail the job on failing test.

I would like the MSTest plugin to fail the job when a test fails. Is this feasible ?