[JIRA] (JENKINS-12616) Tests failes but the job is still OK

Thomas Fürer created JENKINS-12616:
--------------------------------------

             Summary: Tests failes but the job is still OK
                 Key: JENKINS-12616
                 URL: https://issues.jenkins-ci.org/browse/JENKINS-12616
             Project: Jenkins
          Issue Type: Bug
          Components: xunit
         Environment: jenkins 1.445
xunit 1.34

Ubuntu server 11.11
            Reporter: Thomas Fürer
            Assignee: gbois
            Priority: Blocker
         Attachments: boost-config.PNG, failedButOK.PNG, TEST-failed.xml

just use the attached result file with BOOST

the error is listed correct in the test report, but the job is still OK but need to be at least UNSTABLE.
As you can see in the screenshot I do not have any thresholds set.

I do not have any example at the moment, but I think we have this problem also with NUnit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

    [ https://issues.jenkins-ci.org/browse/JENKINS-12616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=159303#comment-159303 ]

Vladimir Zak commented on JENKINS-12616:
----------------------------------------

yes, the same setting and the same behavior ;) this is really problem
               

> Tests failes but the job is still OK
> ------------------------------------
>
>                 Key: JENKINS-12616
>                 URL: https://issues.jenkins-ci.org/browse/JENKINS-12616
>             Project: Jenkins
>          Issue Type: Bug
>          Components: xunit
>         Environment: jenkins 1.445
> xunit 1.34
> Ubuntu server 11.11
>            Reporter: Thomas Fürer
>            Assignee: gbois
>            Priority: Blocker
>         Attachments: boost-config.PNG, failedButOK.PNG, TEST-failed.xml
>
>
> just use the attached result file with BOOST
> the error is listed correct in the test report, but the job is still OK but need to be at least UNSTABLE.
> As you can see in the screenshot I do not have any thresholds set.
> I do not have any example at the moment, but I think we have this problem also with NUnit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

    [ https://issues.jenkins-ci.org/browse/JENKINS-12616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=159303#comment-159303 ]

Vladimir Zak edited comment on JENKINS-12616 at 2/20/12 6:49 PM:
-----------------------------------------------------------------

yes, the same setting and the same behavior - this is problem with cppunit too.
               
      was (Author: zakyn):
    yes, the same setting and the same behavior ;) this is really problem
                 

> Tests failes but the job is still OK
> ------------------------------------
>
>                 Key: JENKINS-12616
>                 URL: https://issues.jenkins-ci.org/browse/JENKINS-12616
>             Project: Jenkins
>          Issue Type: Bug
>          Components: xunit
>         Environment: jenkins 1.445
> xunit 1.34
> Ubuntu server 11.11
>            Reporter: Thomas Fürer
>            Assignee: gbois
>            Priority: Blocker
>         Attachments: boost-config.PNG, failedButOK.PNG, TEST-failed.xml
>
>
> just use the attached result file with BOOST
> the error is listed correct in the test report, but the job is still OK but need to be at least UNSTABLE.
> As you can see in the screenshot I do not have any thresholds set.
> I do not have any example at the moment, but I think we have this problem also with NUnit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

    [ https://issues.jenkins-ci.org/browse/JENKINS-12616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=159320#comment-159320 ]

gbois commented on JENKINS-12616:
---------------------------------

If you don't have any threshold set, the expected build result is always set to SUCCESS.

               

> Tests failes but the job is still OK
> ------------------------------------
>
>                 Key: JENKINS-12616
>                 URL: https://issues.jenkins-ci.org/browse/JENKINS-12616
>             Project: Jenkins
>          Issue Type: Bug
>          Components: xunit
>         Environment: jenkins 1.445
> xunit 1.34
> Ubuntu server 11.11
>            Reporter: Thomas Fürer
>            Assignee: gbois
>            Priority: Blocker
>         Attachments: boost-config.PNG, failedButOK.PNG, TEST-failed.xml
>
>
> just use the attached result file with BOOST
> the error is listed correct in the test report, but the job is still OK but need to be at least UNSTABLE.
> As you can see in the screenshot I do not have any thresholds set.
> I do not have any example at the moment, but I think we have this problem also with NUnit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

    [ https://issues.jenkins-ci.org/browse/JENKINS-12616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=159326#comment-159326 ]

Thomas Fürer commented on JENKINS-12616:
----------------------------------------

it is the same behavior if I set a threshold of
Build Status Total New Total New
Thresholds: 1       1       10      10

I'm using the file attached which has only one failing test.
               

> Tests failes but the job is still OK
> ------------------------------------
>
>                 Key: JENKINS-12616
>                 URL: https://issues.jenkins-ci.org/browse/JENKINS-12616
>             Project: Jenkins
>          Issue Type: Bug
>          Components: xunit
>         Environment: jenkins 1.445
> xunit 1.34
> Ubuntu server 11.11
>            Reporter: Thomas Fürer
>            Assignee: gbois
>            Priority: Blocker
>         Attachments: boost-config.PNG, failedButOK.PNG, TEST-failed.xml
>
>
> just use the attached result file with BOOST
> the error is listed correct in the test report, but the job is still OK but need to be at least UNSTABLE.
> As you can see in the screenshot I do not have any thresholds set.
> I do not have any example at the moment, but I think we have this problem also with NUnit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

    [ https://issues.jenkins-ci.org/browse/JENKINS-12616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=159368#comment-159368 ]

gbois commented on JENKINS-12616:
---------------------------------

If no values are set in the threshold section, threshold process is not activated.

In your case, you have one failed test and if you set the value '0' in 'Total' column, the build should be set to UNSTABLE (the number of test must exceed strict the value).
It's OK in my environment, could you confirm?

               

> Tests failes but the job is still OK
> ------------------------------------
>
>                 Key: JENKINS-12616
>                 URL: https://issues.jenkins-ci.org/browse/JENKINS-12616
>             Project: Jenkins
>          Issue Type: Bug
>          Components: xunit
>         Environment: jenkins 1.445
> xunit 1.34
> Ubuntu server 11.11
>            Reporter: Thomas Fürer
>            Assignee: gbois
>            Priority: Blocker
>         Attachments: boost-config.PNG, failedButOK.PNG, TEST-failed.xml
>
>
> just use the attached result file with BOOST
> the error is listed correct in the test report, but the job is still OK but need to be at least UNSTABLE.
> As you can see in the screenshot I do not have any thresholds set.
> I do not have any example at the moment, but I think we have this problem also with NUnit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

     [ https://issues.jenkins-ci.org/browse/JENKINS-12616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Fürer updated JENKINS-12616:
-----------------------------------

    Priority: Minor  (was: Blocker)

setting threshold values solves the problem
               

> Tests failes but the job is still OK
> ------------------------------------
>
>                 Key: JENKINS-12616
>                 URL: https://issues.jenkins-ci.org/browse/JENKINS-12616
>             Project: Jenkins
>          Issue Type: Bug
>          Components: xunit
>         Environment: jenkins 1.445
> xunit 1.34
> Ubuntu server 11.11
>            Reporter: Thomas Fürer
>            Assignee: gbois
>            Priority: Minor
>         Attachments: boost-config.PNG, failedButOK.PNG, TEST-failed.xml
>
>
> just use the attached result file with BOOST
> the error is listed correct in the test report, but the job is still OK but need to be at least UNSTABLE.
> As you can see in the screenshot I do not have any thresholds set.
> I do not have any example at the moment, but I think we have this problem also with NUnit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

    [ https://issues.jenkins-ci.org/browse/JENKINS-12616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=159391#comment-159391 ]

Thomas Fürer commented on JENKINS-12616:
----------------------------------------

because this is a change of the default behaivor I suggest to set default thresholds so set the default behavior is set.
               

> Tests failes but the job is still OK
> ------------------------------------
>
>                 Key: JENKINS-12616
>                 URL: https://issues.jenkins-ci.org/browse/JENKINS-12616
>             Project: Jenkins
>          Issue Type: Bug
>          Components: xunit
>         Environment: jenkins 1.445
> xunit 1.34
> Ubuntu server 11.11
>            Reporter: Thomas Fürer
>            Assignee: gbois
>            Priority: Minor
>         Attachments: boost-config.PNG, failedButOK.PNG, TEST-failed.xml
>
>
> just use the attached result file with BOOST
> the error is listed correct in the test report, but the job is still OK but need to be at least UNSTABLE.
> As you can see in the screenshot I do not have any thresholds set.
> I do not have any example at the moment, but I think we have this problem also with NUnit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

     [ https://issues.jenkins-ci.org/browse/JENKINS-12616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

gbois closed JENKINS-12616.
---------------------------

    Resolution: Won't Fix

Giving default values for threshold is difficult: failed test  or skipped test; unstable or fail.

The response depends on the context.

My choice: if no values are set, no threshold is processed.
               

> Tests failes but the job is still OK
> ------------------------------------
>
>                 Key: JENKINS-12616
>                 URL: https://issues.jenkins-ci.org/browse/JENKINS-12616
>             Project: Jenkins
>          Issue Type: Bug
>          Components: xunit
>         Environment: jenkins 1.445
> xunit 1.34
> Ubuntu server 11.11
>            Reporter: Thomas Fürer
>            Assignee: gbois
>            Priority: Minor
>         Attachments: boost-config.PNG, failedButOK.PNG, TEST-failed.xml
>
>
> just use the attached result file with BOOST
> the error is listed correct in the test report, but the job is still OK but need to be at least UNSTABLE.
> As you can see in the screenshot I do not have any thresholds set.
> I do not have any example at the moment, but I think we have this problem also with NUnit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira