Quantcast

retry build upon failure?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

retry build upon failure?

edward.young
Is it possible to retry a build upon a failure? Maybe with a retry count?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: retry build upon failure?

nicolas de loof-2
see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin

2012/9/14 Ed Young <[hidden email]>
Is it possible to retry a build upon a failure? Maybe with a retry count?

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

RE: retry build upon failure?

William Soula

Or <a href="https://wiki.jenkins-ci.org/display/JENKINS/Retry&#43;Failed&#43;Builds&#43;Plugin">https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of nicolas de loof
Sent: Friday, September 14, 2012 6:16 AM
To: [hidden email]
Subject: Re: retry build upon failure?

 

see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin

2012/9/14 Ed Young <[hidden email]>

Is it possible to retry a build upon a failure? Maybe with a retry count?

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: retry build upon failure?

edward.young
I installed the Naginator Plugin (v1.8) because the retry failed build
plugin indicated that it was deprecated in favor of the Naginatore
plugin.

It doesn't seem to be working the way I expect it to however. I
configured it to simply

Retry build after failure up to 3 times, with all other defaults, but
after a failure there is no retry. I configured it for a  fixed delay
and it still did not retry.

It seems like a very simple configuration and the build is a hard
failure (test failures) so I can't tell where I'm going wrong.
Using Jenkins 1.436.

 
       
 
               
 
       
               
               


On Fri, Sep 14, 2012 at 5:59 AM, William Soula
<[hidden email]> wrote:

> Or https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin
>
>
>
>
>
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of nicolas de loof
> Sent: Friday, September 14, 2012 6:16 AM
> To: [hidden email]
> Subject: Re: retry build upon failure?
>
>
>
> see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
>
> 2012/9/14 Ed Young <[hidden email]>
>
> Is it possible to retry a build upon a failure? Maybe with a retry count?
>
>



--
- Ed
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: retry build upon failure?

Sami Tikka
Could you post exactly what you see in the console log of the failed build? Also, it might help if you could post your job config.xml or put in pastebin/gist so we could take a look.

-- Sami

Ed Young <[hidden email]> kirjoitti 14.9.2012 kello 22.38:

> I installed the Naginator Plugin (v1.8) because the retry failed build
> plugin indicated that it was deprecated in favor of the Naginatore
> plugin.
>
> It doesn't seem to be working the way I expect it to however. I
> configured it to simply
>
> Retry build after failure up to 3 times, with all other defaults, but
> after a failure there is no retry. I configured it for a  fixed delay
> and it still did not retry.
>
> It seems like a very simple configuration and the build is a hard
> failure (test failures) so I can't tell where I'm going wrong.
> Using Jenkins 1.436.
>
>    
>    
>    
>        
>    
>    
>        
>        
>
>
> On Fri, Sep 14, 2012 at 5:59 AM, William Soula
> <[hidden email]> wrote:
>> Or https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin
>>
>>
>>
>>
>>
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of nicolas de loof
>> Sent: Friday, September 14, 2012 6:16 AM
>> To: [hidden email]
>> Subject: Re: retry build upon failure?
>>
>>
>>
>> see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
>>
>> 2012/9/14 Ed Young <[hidden email]>
>>
>> Is it possible to retry a build upon a failure? Maybe with a retry count?
>>
>>
>
>
>
> --
> - Ed
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: retry build upon failure?

Mark Waite
When Ed says the build is a"hard failure", I assumed that meant the build status was "Failed" and would be marked red.  However, in parentheses he says "(test failures)".  Typically, test failures mark the build as "unstable", not as "failed".  If the build is unstable, I would not expect a "retry on fail" to be invoked, since "unstable" is not "failed".

Can you clarify?  Was the build red (failed) or yellow (unstable)?

Mark Waite


From: Sami Tikka <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Sunday, September 16, 2012 11:08 AM
Subject: Re: retry build upon failure?

Could you post exactly what you see in the console log of the failed build? Also, it might help if you could post your job config.xml or put in pastebin/gist so we could take a look.

-- Sami

Ed Young <[hidden email]> kirjoitti 14.9.2012 kello 22.38:

> I installed the Naginator Plugin (v1.8) because the retry failed build
> plugin indicated that it was deprecated in favor of the Naginatore
> plugin.
>
> It doesn't seem to be working the way I expect it to however. I
> configured it to simply
>
> Retry build after failure up to 3 times, with all other defaults, but
> after a failure there is no retry. I configured it for a  fixed delay
> and it still did not retry.
>
> It seems like a very simple configuration and the build is a hard
> failure (test failures) so I can't tell where I'm going wrong.
> Using Jenkins 1.436.
>
>   
>   
>   
>       
>   
>   
>       
>       
>
>
> On Fri, Sep 14, 2012 at 5:59 AM, William Soula
> <[hidden email]> wrote:
>> Or https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin
>>
>>
>>
>>
>>
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of nicolas de loof
>> Sent: Friday, September 14, 2012 6:16 AM
>> To: [hidden email]
>> Subject: Re: retry build upon failure?
>>
>>
>>
>> see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
>>
>> 2012/9/14 Ed Young <[hidden email]>
>>
>> Is it possible to retry a build upon a failure? Maybe with a retry count?
>>
>>
>
>
>
> --
> - Ed


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: retry build upon failure?

edward.young
Sounds like we have some terminology differences. These are test
failures, which I consider a failed builds. I've experimented with the
configuration in many different ways and it never results in a retry,
even when I set it to retry after "unstable" builds (test failure I
guess)

If the build passes all tests, it's a pass and "blue". If it fails any
tests at all it's "red" which I call a "hard failure", but others may
not, I reckon. In any case, our policy is that if it fails with test
failures, it's a failed build.

So I set it to retry upon "unstable" builds and it still does not retry.

These are grails builds, and we're using the spock test framework.

Here's the relevant sections of the build config:
  <com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>false</rerunIfUnstable>   <-- original setting
      <checkRegexp>false</checkRegexp>
      <delay class="com.chikli.hudson.plugin.naginator.FixedDelay">
        <delay>1</delay>
      </delay>
      <maxSchedule>3</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

--------

 <com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>true</rerunIfUnstable>  <---- retry on unstable
      <checkRegexp>false</checkRegexp>
      <maxSchedule>2</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

---------

In no cases does it retry.

Here is the end of an example build that fails due to test failures.

...
Running 342 spock tests... 188 of 342
| Completed 342 spock tests, 3 failed in 865846ms
| Compiling 1 source files
| Compiling 1 source files.
| Compiling 1 source files..
| Tests FAILED  - view reports in
/opt/comcast/data/jenkins/jobs/chimps-geb-grails-lb-test/workspace/target/test-reports
Build step 'Build With Grails' marked build as failure

Still didn't retry even after specifying retry after an "unstable" build.




On Sun, Sep 16, 2012 at 2:56 PM, Mark Waite <[hidden email]> wrote:

> When Ed says the build is a"hard failure", I assumed that meant the build
> status was "Failed" and would be marked red.  However, in parentheses he
> says "(test failures)".  Typically, test failures mark the build as
> "unstable", not as "failed".  If the build is unstable, I would not expect a
> "retry on fail" to be invoked, since "unstable" is not "failed".
>
> Can you clarify?  Was the build red (failed) or yellow (unstable)?
>
> Mark Waite
>
> ________________________________
> From: Sami Tikka <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Sunday, September 16, 2012 11:08 AM
>
> Subject: Re: retry build upon failure?
>
> Could you post exactly what you see in the console log of the failed build?
> Also, it might help if you could post your job config.xml or put in
> pastebin/gist so we could take a look.
>
> -- Sami
>
> Ed Young <[hidden email]> kirjoitti 14.9.2012 kello 22.38:
>
>> I installed the Naginator Plugin (v1.8) because the retry failed build
>> plugin indicated that it was deprecated in favor of the Naginatore
>> plugin.
>>
>> It doesn't seem to be working the way I expect it to however. I
>> configured it to simply
>>
>> Retry build after failure up to 3 times, with all other defaults, but
>> after a failure there is no retry. I configured it for a  fixed delay
>> and it still did not retry.
>>
>> It seems like a very simple configuration and the build is a hard
>> failure (test failures) so I can't tell where I'm going wrong.
>> Using Jenkins 1.436.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Sep 14, 2012 at 5:59 AM, William Soula
>> <[hidden email]> wrote:
>>> Or https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin
>>>
>>>
>>>
>>>
>>>
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of nicolas de loof
>>> Sent: Friday, September 14, 2012 6:16 AM
>>> To: [hidden email]
>>> Subject: Re: retry build upon failure?
>>>
>>>
>>>
>>> see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
>>>
>>> 2012/9/14 Ed Young <[hidden email]>
>>>
>>> Is it possible to retry a build upon a failure? Maybe with a retry count?
>>>
>>>
>>
>>
>>
>> --
>> - Ed
>
>



--
- Ed
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: retry build upon failure?

Mark Waite
I think you've found a bug.  At least, my initial experiments seem to confirm the same results you reported.

I had never experimented with the Naginator plugin before.  I installed it into a Jenkins 1.466.2 installation and created a new free style project with a single build step, a shell with the one line command "exit 1" and a single post build action, retry failed build.

I configured the retry post build action to rebuild up to two times.

The first time that job ran, the jenkins log file contained the following report:

INFO: Failure-Test #1 main build action completed: FAILURE
Sep 16, 2012 9:56:54 PM hudson.model.Executor run
SEVERE: Executor threw an exception
java.lang.NullPointerException
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:65)
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:20)
        at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:178)
        at hudson.model.Run.run(Run.java:1488)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:239)

The job did not retry.

Can you submit a bug against the Naginator plugin?

Thanks,
Mark Waite


From: Ed Young <[hidden email]>
To: [hidden email]
Sent: Sunday, September 16, 2012 8:56 PM
Subject: Re: retry build upon failure?

Sounds like we have some terminology differences. These are test
failures, which I consider a failed builds. I've experimented with the
configuration in many different ways and it never results in a retry,
even when I set it to retry after "unstable" builds (test failure I
guess)

If the build passes all tests, it's a pass and "blue". If it fails any
tests at all it's "red" which I call a "hard failure", but others may
not, I reckon. In any case, our policy is that if it fails with test
failures, it's a failed build.

So I set it to retry upon "unstable" builds and it still does not retry.

These are grails builds, and we're using the spock test framework.

Here's the relevant sections of the build config:
  <com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>false</rerunIfUnstable>  <-- original setting
      <checkRegexp>false</checkRegexp>
      <delay class="com.chikli.hudson.plugin.naginator.FixedDelay">
        <delay>1</delay>
      </delay>
      <maxSchedule>3</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

--------

<com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>true</rerunIfUnstable>  <---- retry on unstable
      <checkRegexp>false</checkRegexp>
      <maxSchedule>2</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

---------

In no cases does it retry.

Here is the end of an example build that fails due to test failures.

...
Running 342 spock tests... 188 of 342
| Completed 342 spock tests, 3 failed in 865846ms
| Compiling 1 source files
| Compiling 1 source files.
| Compiling 1 source files..
| Tests FAILED  - view reports in
/opt/comcast/data/jenkins/jobs/chimps-geb-grails-lb-test/workspace/target/test-reports
Build step 'Build With Grails' marked build as failure

Still didn't retry even after specifying retry after an "unstable" build.




On Sun, Sep 16, 2012 at 2:56 PM, Mark Waite <[hidden email]> wrote:

> When Ed says the build is a"hard failure", I assumed that meant the build
> status was "Failed" and would be marked red.  However, in parentheses he
> says "(test failures)".  Typically, test failures mark the build as
> "unstable", not as "failed".  If the build is unstable, I would not expect a
> "retry on fail" to be invoked, since "unstable" is not "failed".
>
> Can you clarify?  Was the build red (failed) or yellow (unstable)?
>
> Mark Waite
>
> ________________________________
> From: Sami Tikka <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Sunday, September 16, 2012 11:08 AM
>
> Subject: Re: retry build upon failure?
>
> Could you post exactly what you see in the console log of the failed build?
> Also, it might help if you could post your job config.xml or put in
> pastebin/gist so we could take a look.
>
> -- Sami
>
> Ed Young <[hidden email]> kirjoitti 14.9.2012 kello 22.38:
>
>> I installed the Naginator Plugin (v1.8) because the retry failed build
>> plugin indicated that it was deprecated in favor of the Naginatore
>> plugin.
>>
>> It doesn't seem to be working the way I expect it to however. I
>> configured it to simply
>>
>> Retry build after failure up to 3 times, with all other defaults, but
>> after a failure there is no retry. I configured it for a  fixed delay
>> and it still did not retry.
>>
>> It seems like a very simple configuration and the build is a hard
>> failure (test failures) so I can't tell where I'm going wrong.
>> Using Jenkins 1.436.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Sep 14, 2012 at 5:59 AM, William Soula
>> <[hidden email]> wrote:
>>> Or https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin
>>>
>>>
>>>
>>>
>>>
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of nicolas de loof
>>> Sent: Friday, September 14, 2012 6:16 AM
>>> To: [hidden email]
>>> Subject: Re: retry build upon failure?
>>>
>>>
>>>
>>> see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
>>>
>>> 2012/9/14 Ed Young <[hidden email]>
>>>
>>> Is it possible to retry a build upon a failure? Maybe with a retry count?
>>>
>>>
>>
>>
>>
>> --
>> - Ed
>
>



--
- Ed


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: retry build upon failure?

Mark Waite
I further simplified my configuration by removing Jenkins and its configuration files completely from a test computer, then I installed Jenkins 1.466.2, installed the Naginator plugin, and confirmed it was the only plugin installed.  I created a new Jenkins job with a shell command "exit 1" and set it to retry up to twice on failed builds.

The /var/log/jenkins.log file again included the following stack trace:

Sep 16, 2012 10:15:51 PM hudson.model.Run run
INFO: Naginator-Retry #1 main build action completed: FAILURE
Sep 16, 2012 10:15:51 PM hudson.model.Executor run
SEVERE: Executor threw an exception
java.lang.NullPointerException
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:65)
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:20)
        at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:178)
        at hudson.model.Run.run(Run.java:1488)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:239)

Looking at the source code, it appears the plugin wants a delay value.  I had not set a delay value, since it was optional in the user interface.  When I set the delay value to a fixed 2 second delay, then the job retried after first failure (and no null pointer exception in the log file).  I was able to confirm that it would retry up to 4 times after a failed build.

You might consider setting a delay value to see if that helps your case like it helped mine.

You might also consider watching the log file to see if there are any further hints in what you're doing.

I'll file the bug report with the results of my investigation.

Mark Waite


From: Mark Waite <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Sunday, September 16, 2012 10:09 PM
Subject: Re: retry build upon failure?

I think you've found a bug.  At least, my initial experiments seem to confirm the same results you reported.

I had never experimented with the Naginator plugin before.  I installed it into a Jenkins 1.466.2 installation and created a new free style project with a single build step, a shell with the one line command "exit 1" and a single post build action, retry failed build.

I configured the retry post build action to rebuild up to two times.

The first time that job ran, the jenkins log file contained the following report:

INFO: Failure-Test #1 main build action completed: FAILURE
Sep 16, 2012 9:56:54 PM hudson.model.Executor run
SEVERE: Executor threw an exception
java.lang.NullPointerException
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:65)
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:20)
        at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:178)
        at hudson.model.Run.run(Run.java:1488)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:239)

The job did not retry.

Can you submit a bug against the Naginator plugin?

Thanks,
Mark Waite


From: Ed Young <[hidden email]>
To: [hidden email]
Sent: Sunday, September 16, 2012 8:56 PM
Subject: Re: retry build upon failure?

Sounds like we have some terminology differences. These are test
failures, which I consider a failed builds. I've experimented with the
configuration in many different ways and it never results in a retry,
even when I set it to retry after "unstable" builds (test failure I
guess)

If the build passes all tests, it's a pass and "blue". If it fails any
tests at all it's "red" which I call a "hard failure", but others may
not, I reckon. In any case, our policy is that if it fails with test
failures, it's a failed build.

So I set it to retry upon "unstable" builds and it still does not retry.

These are grails builds, and we're using the spock test framework.

Here's the relevant sections of the build config:
  <com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>false</rerunIfUnstable>  <-- original setting
      <checkRegexp>false</checkRegexp>
      <delay class="com.chikli.hudson.plugin.naginator.FixedDelay">
        <delay>1</delay>
      </delay>
      <maxSchedule>3</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

--------

<com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>true</rerunIfUnstable>  <---- retry on unstable
      <checkRegexp>false</checkRegexp>
      <maxSchedule>2</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

---------

In no cases does it retry.

Here is the end of an example build that fails due to test failures.

...
Running 342 spock tests... 188 of 342
| Completed 342 spock tests, 3 failed in 865846ms
| Compiling 1 source files
| Compiling 1 source files.
| Compiling 1 source files..
| Tests FAILED  - view reports in
/opt/comcast/data/jenkins/jobs/chimps-geb-grails-lb-test/workspace/target/test-reports
Build step 'Build With Grails' marked build as failure

Still didn't retry even after specifying retry after an "unstable" build.




On Sun, Sep 16, 2012 at 2:56 PM, Mark Waite <[hidden email]> wrote:

> When Ed says the build is a"hard failure", I assumed that meant the build
> status was "Failed" and would be marked red.  However, in parentheses he
> says "(test failures)".  Typically, test failures mark the build as
> "unstable", not as "failed".  If the build is unstable, I would not expect a
> "retry on fail" to be invoked, since "unstable" is not "failed".
>
> Can you clarify?  Was the build red (failed) or yellow (unstable)?
>
> Mark Waite
>
> ________________________________
> From: Sami Tikka <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Sunday, September 16, 2012 11:08 AM
>
> Subject: Re: retry build upon failure?
>
> Could you post exactly what you see in the console log of the failed build?
> Also, it might help if you could post your job config.xml or put in
> pastebin/gist so we could take a look.
>
> -- Sami
>
> Ed Young <[hidden email]> kirjoitti 14.9.2012 kello 22.38:
>
>> I installed the Naginator Plugin (v1.8) because the retry failed build
>> plugin indicated that it was deprecated in favor of the Naginatore
>> plugin.
>>
>> It doesn't seem to be working the way I expect it to however. I
>> configured it to simply
>>
>> Retry build after failure up to 3 times, with all other defaults, but
>> after a failure there is no retry. I configured it for a  fixed delay
>> and it still did not retry.
>>
>> It seems like a very simple configuration and the build is a hard
>> failure (test failures) so I can't tell where I'm going wrong.
>> Using Jenkins 1.436.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Sep 14, 2012 at 5:59 AM, William Soula
>> <[hidden email]> wrote:
>>> Or https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin
>>>
>>>
>>>
>>>
>>>
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of nicolas de loof
>>> Sent: Friday, September 14, 2012 6:16 AM
>>> To: [hidden email]
>>> Subject: Re: retry build upon failure?
>>>
>>>
>>>
>>> see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
>>>
>>> 2012/9/14 Ed Young <[hidden email]>
>>>
>>> Is it possible to retry a build upon a failure? Maybe with a retry count?
>>>
>>>
>>
>>
>>
>> --
>> - Ed
>
>



--
- Ed




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: retry build upon failure?

edward.young
In reply to this post by Mark Waite
done:
https://issues.jenkins-ci.org/browse/JENKINS-15184

I'll do some more investigating, as well.

Thanks,

Ed


On Sun, Sep 16, 2012 at 10:09 PM, Mark Waite <[hidden email]> wrote:
I think you've found a bug.  At least, my initial experiments seem to confirm the same results you reported.

I had never experimented with the Naginator plugin before.  I installed it into a Jenkins 1.466.2 installation and created a new free style project with a single build step, a shell with the one line command "exit 1" and a single post build action, retry failed build.

I configured the retry post build action to rebuild up to two times.

The first time that job ran, the jenkins log file contained the following report:

INFO: Failure-Test #1 main build action completed: FAILURE
Sep 16, 2012 9:56:54 PM hudson.model.Executor run
SEVERE: Executor threw an exception
java.lang.NullPointerException
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:65)
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:20)
        at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:178)
        at hudson.model.Run.run(Run.java:1488)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:239)

The job did not retry.

Can you submit a bug against the Naginator plugin?

Thanks,
Mark Waite


From: Ed Young <[hidden email]>
To: [hidden email]
Sent: Sunday, September 16, 2012 8:56 PM

Subject: Re: retry build upon failure?

Sounds like we have some terminology differences. These are test
failures, which I consider a failed builds. I've experimented with the
configuration in many different ways and it never results in a retry,
even when I set it to retry after "unstable" builds (test failure I
guess)

If the build passes all tests, it's a pass and "blue". If it fails any
tests at all it's "red" which I call a "hard failure", but others may
not, I reckon. In any case, our policy is that if it fails with test
failures, it's a failed build.

So I set it to retry upon "unstable" builds and it still does not retry.

These are grails builds, and we're using the spock test framework.

Here's the relevant sections of the build config:
  <com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>false</rerunIfUnstable>  <-- original setting
      <checkRegexp>false</checkRegexp>
      <delay class="com.chikli.hudson.plugin.naginator.FixedDelay">
        <delay>1</delay>
      </delay>
      <maxSchedule>3</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

--------

<com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>true</rerunIfUnstable>  <---- retry on unstable
      <checkRegexp>false</checkRegexp>
      <maxSchedule>2</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

---------

In no cases does it retry.

Here is the end of an example build that fails due to test failures.

...
Running 342 spock tests... 188 of 342
| Completed 342 spock tests, 3 failed in 865846ms
| Compiling 1 source files
| Compiling 1 source files.
| Compiling 1 source files..
| Tests FAILED  - view reports in
/opt/comcast/data/jenkins/jobs/chimps-geb-grails-lb-test/workspace/target/test-reports
Build step 'Build With Grails' marked build as failure

Still didn't retry even after specifying retry after an "unstable" build.




On Sun, Sep 16, 2012 at 2:56 PM, Mark Waite <[hidden email]> wrote:
> When Ed says the build is a"hard failure", I assumed that meant the build
> status was "Failed" and would be marked red.  However, in parentheses he
> says "(test failures)".  Typically, test failures mark the build as
> "unstable", not as "failed".  If the build is unstable, I would not expect a
> "retry on fail" to be invoked, since "unstable" is not "failed".
>
> Can you clarify?  Was the build red (failed) or yellow (unstable)?
>
> Mark Waite
>
> ________________________________
> From: Sami Tikka <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Sunday, September 16, 2012 11:08 AM
>
> Subject: Re: retry build upon failure?
>
> Could you post exactly what you see in the console log of the failed build?
> Also, it might help if you could post your job config.xml or put in
> pastebin/gist so we could take a look.
>
> -- Sami
>
> Ed Young <[hidden email]> kirjoitti 14.9.2012 kello 22.38:
>
>> I installed the Naginator Plugin (v1.8) because the retry failed build
>> plugin indicated that it was deprecated in favor of the Naginatore
>> plugin.
>>
>> It doesn't seem to be working the way I expect it to however. I
>> configured it to simply
>>
>> Retry build after failure up to 3 times, with all other defaults, but
>> after a failure there is no retry. I configured it for a  fixed delay
>> and it still did not retry.
>>
>> It seems like a very simple configuration and the build is a hard
>> failure (test failures) so I can't tell where I'm going wrong.
>> Using Jenkins 1.436.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Sep 14, 2012 at 5:59 AM, William Soula
>> <[hidden email]> wrote:
>>> Or https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin
>>>
>>>
>>>
>>>
>>>
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of nicolas de loof
>>> Sent: Friday, September 14, 2012 6:16 AM
>>> To: [hidden email]
>>> Subject: Re: retry build upon failure?
>>>
>>>
>>>
>>> see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
>>>

>>> 2012/9/14 Ed Young <[hidden email]>
>>>
>>> Is it possible to retry a build upon a failure? Maybe with a retry count?
>>>
>>>
>>
>>
>>
>> --
>> - Ed
>
>



--
- Ed





--
- Ed
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: retry build upon failure?

Mark Waite
In reply to this post by Mark Waite
I see that you've already submitted Jenkins-15184.  Can you investigate if your failure mode is the same failure mode as the one I describe below?  

If it is the same, then you can attach this information to the bug report (and you can use it as a work around).

If it is not the same, then I can submit a bug report for this problem in addition to your problem.

Thanks,
Mark Waite


From: Mark Waite <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Sunday, September 16, 2012 10:26 PM
Subject: Re: retry build upon failure?

I further simplified my configuration by removing Jenkins and its configuration files completely from a test computer, then I installed Jenkins 1.466.2, installed the Naginator plugin, and confirmed it was the only plugin installed.  I created a new Jenkins job with a shell command "exit 1" and set it to retry up to twice on failed builds.

The /var/log/jenkins.log file again included the following stack trace:

Sep 16, 2012 10:15:51 PM hudson.model.Run run
INFO: Naginator-Retry #1 main build action completed: FAILURE
Sep 16, 2012 10:15:51 PM hudson.model.Executor run
SEVERE: Executor threw an exception
java.lang.NullPointerException
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:65)
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:20)
        at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:178)
        at hudson.model.Run.run(Run.java:1488)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:239)

Looking at the source code, it appears the plugin wants a delay value.  I had not set a delay value, since it was optional in the user interface.  When I set the delay value to a fixed 2 second delay, then the job retried after first failure (and no null pointer exception in the log file).  I was able to confirm that it would retry up to 4 times after a failed build.

You might consider setting a delay value to see if that helps your case like it helped mine.

You might also consider watching the log file to see if there are any further hints in what you're doing.

I'll file the bug report with the results of my investigation.

Mark Waite


From: Mark Waite <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Sunday, September 16, 2012 10:09 PM
Subject: Re: retry build upon failure?

I think you've found a bug.  At least, my initial experiments seem to confirm the same results you reported.

I had never experimented with the Naginator plugin before.  I installed it into a Jenkins 1.466.2 installation and created a new free style project with a single build step, a shell with the one line command "exit 1" and a single post build action, retry failed build.

I configured the retry post build action to rebuild up to two times.

The first time that job ran, the jenkins log file contained the following report:

INFO: Failure-Test #1 main build action completed: FAILURE
Sep 16, 2012 9:56:54 PM hudson.model.Executor run
SEVERE: Executor threw an exception
java.lang.NullPointerException
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:65)
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:20)
        at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:178)
        at hudson.model.Run.run(Run.java:1488)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:239)

The job did not retry.

Can you submit a bug against the Naginator plugin?

Thanks,
Mark Waite


From: Ed Young <[hidden email]>
To: [hidden email]
Sent: Sunday, September 16, 2012 8:56 PM
Subject: Re: retry build upon failure?

Sounds like we have some terminology differences. These are test
failures, which I consider a failed builds. I've experimented with the
configuration in many different ways and it never results in a retry,
even when I set it to retry after "unstable" builds (test failure I
guess)

If the build passes all tests, it's a pass and "blue". If it fails any
tests at all it's "red" which I call a "hard failure", but others may
not, I reckon. In any case, our policy is that if it fails with test
failures, it's a failed build.

So I set it to retry upon "unstable" builds and it still does not retry.

These are grails builds, and we're using the spock test framework.

Here's the relevant sections of the build config:
  <com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>false</rerunIfUnstable>  <-- original setting
      <checkRegexp>false</checkRegexp>
      <delay class="com.chikli.hudson.plugin.naginator.FixedDelay">
        <delay>1</delay>
      </delay>
      <maxSchedule>3</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

--------

<com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>true</rerunIfUnstable>  <---- retry on unstable
      <checkRegexp>false</checkRegexp>
      <maxSchedule>2</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

---------

In no cases does it retry.

Here is the end of an example build that fails due to test failures.

...
Running 342 spock tests... 188 of 342
| Completed 342 spock tests, 3 failed in 865846ms
| Compiling 1 source files
| Compiling 1 source files.
| Compiling 1 source files..
| Tests FAILED  - view reports in
/opt/comcast/data/jenkins/jobs/chimps-geb-grails-lb-test/workspace/target/test-reports
Build step 'Build With Grails' marked build as failure

Still didn't retry even after specifying retry after an "unstable" build.




On Sun, Sep 16, 2012 at 2:56 PM, Mark Waite <[hidden email]> wrote:

> When Ed says the build is a"hard failure", I assumed that meant the build
> status was "Failed" and would be marked red.  However, in parentheses he
> says "(test failures)".  Typically, test failures mark the build as
> "unstable", not as "failed".  If the build is unstable, I would not expect a
> "retry on fail" to be invoked, since "unstable" is not "failed".
>
> Can you clarify?  Was the build red (failed) or yellow (unstable)?
>
> Mark Waite
>
> ________________________________
> From: Sami Tikka <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Sunday, September 16, 2012 11:08 AM
>
> Subject: Re: retry build upon failure?
>
> Could you post exactly what you see in the console log of the failed build?
> Also, it might help if you could post your job config.xml or put in
> pastebin/gist so we could take a look.
>
> -- Sami
>
> Ed Young <[hidden email]> kirjoitti 14.9.2012 kello 22.38:
>
>> I installed the Naginator Plugin (v1.8) because the retry failed build
>> plugin indicated that it was deprecated in favor of the Naginatore
>> plugin.
>>
>> It doesn't seem to be working the way I expect it to however. I
>> configured it to simply
>>
>> Retry build after failure up to 3 times, with all other defaults, but
>> after a failure there is no retry. I configured it for a  fixed delay
>> and it still did not retry.
>>
>> It seems like a very simple configuration and the build is a hard
>> failure (test failures) so I can't tell where I'm going wrong.
>> Using Jenkins 1.436.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Sep 14, 2012 at 5:59 AM, William Soula
>> <[hidden email]> wrote:
>>> Or https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin
>>>
>>>
>>>
>>>
>>>
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of nicolas de loof
>>> Sent: Friday, September 14, 2012 6:16 AM
>>> To: [hidden email]
>>> Subject: Re: retry build upon failure?
>>>
>>>
>>>
>>> see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
>>>
>>> 2012/9/14 Ed Young <[hidden email]>
>>>
>>> Is it possible to retry a build upon a failure? Maybe with a retry count?
>>>
>>>
>>
>>
>>
>> --
>> - Ed
>
>



--
- Ed






Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: retry build upon failure?

edward.young
I can, but not till tomorrow. Thanks for looking into it with me. -Ed

On Sun, Sep 16, 2012 at 10:29 PM, Mark Waite <[hidden email]> wrote:
I see that you've already submitted Jenkins-15184.  Can you investigate if your failure mode is the same failure mode as the one I describe below?  

If it is the same, then you can attach this information to the bug report (and you can use it as a work around).

If it is not the same, then I can submit a bug report for this problem in addition to your problem.

Thanks,
Mark Waite


From: Mark Waite <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Sunday, September 16, 2012 10:26 PM

Subject: Re: retry build upon failure?

I further simplified my configuration by removing Jenkins and its configuration files completely from a test computer, then I installed Jenkins 1.466.2, installed the Naginator plugin, and confirmed it was the only plugin installed.  I created a new Jenkins job with a shell command "exit 1" and set it to retry up to twice on failed builds.

The /var/log/jenkins.log file again included the following stack trace:

Sep 16, 2012 10:15:51 PM hudson.model.Run run
INFO: Naginator-Retry #1 main build action completed: FAILURE
Sep 16, 2012 10:15:51 PM hudson.model.Executor run
SEVERE: Executor threw an exception
java.lang.NullPointerException
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:65)
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:20)
        at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:178)
        at hudson.model.Run.run(Run.java:1488)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:239)

Looking at the source code, it appears the plugin wants a delay value.  I had not set a delay value, since it was optional in the user interface.  When I set the delay value to a fixed 2 second delay, then the job retried after first failure (and no null pointer exception in the log file).  I was able to confirm that it would retry up to 4 times after a failed build.

You might consider setting a delay value to see if that helps your case like it helped mine.

You might also consider watching the log file to see if there are any further hints in what you're doing.

I'll file the bug report with the results of my investigation.

Mark Waite


From: Mark Waite <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Sunday, September 16, 2012 10:09 PM
Subject: Re: retry build upon failure?

I think you've found a bug.  At least, my initial experiments seem to confirm the same results you reported.

I had never experimented with the Naginator plugin before.  I installed it into a Jenkins 1.466.2 installation and created a new free style project with a single build step, a shell with the one line command "exit 1" and a single post build action, retry failed build.

I configured the retry post build action to rebuild up to two times.

The first time that job ran, the jenkins log file contained the following report:

INFO: Failure-Test #1 main build action completed: FAILURE
Sep 16, 2012 9:56:54 PM hudson.model.Executor run
SEVERE: Executor threw an exception
java.lang.NullPointerException
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:65)
        at com.chikli.hudson.plugin.naginator.NaginatorListener.onCompleted(NaginatorListener.java:20)
        at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:178)
        at hudson.model.Run.run(Run.java:1488)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:239)

The job did not retry.

Can you submit a bug against the Naginator plugin?

Thanks,
Mark Waite


From: Ed Young <[hidden email]>
To: [hidden email]
Sent: Sunday, September 16, 2012 8:56 PM
Subject: Re: retry build upon failure?

Sounds like we have some terminology differences. These are test
failures, which I consider a failed builds. I've experimented with the
configuration in many different ways and it never results in a retry,
even when I set it to retry after "unstable" builds (test failure I
guess)

If the build passes all tests, it's a pass and "blue". If it fails any
tests at all it's "red" which I call a "hard failure", but others may
not, I reckon. In any case, our policy is that if it fails with test
failures, it's a failed build.

So I set it to retry upon "unstable" builds and it still does not retry.

These are grails builds, and we're using the spock test framework.

Here's the relevant sections of the build config:
  <com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>false</rerunIfUnstable>  <-- original setting
      <checkRegexp>false</checkRegexp>
      <delay class="com.chikli.hudson.plugin.naginator.FixedDelay">
        <delay>1</delay>
      </delay>
      <maxSchedule>3</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

--------

<com.chikli.hudson.plugin.naginator.NaginatorPublisher>
      <regexpForRerun></regexpForRerun>
      <rerunIfUnstable>true</rerunIfUnstable>  <---- retry on unstable
      <checkRegexp>false</checkRegexp>
      <maxSchedule>2</maxSchedule>
    </com.chikli.hudson.plugin.naginator.NaginatorPublisher>

---------

In no cases does it retry.

Here is the end of an example build that fails due to test failures.

...
Running 342 spock tests... 188 of 342
| Completed 342 spock tests, 3 failed in 865846ms
| Compiling 1 source files
| Compiling 1 source files.
| Compiling 1 source files..
| Tests FAILED  - view reports in
/opt/comcast/data/jenkins/jobs/chimps-geb-grails-lb-test/workspace/target/test-reports
Build step 'Build With Grails' marked build as failure

Still didn't retry even after specifying retry after an "unstable" build.




On Sun, Sep 16, 2012 at 2:56 PM, Mark Waite <[hidden email]> wrote:
> When Ed says the build is a"hard failure", I assumed that meant the build
> status was "Failed" and would be marked red.  However, in parentheses he
> says "(test failures)".  Typically, test failures mark the build as
> "unstable", not as "failed".  If the build is unstable, I would not expect a
> "retry on fail" to be invoked, since "unstable" is not "failed".
>
> Can you clarify?  Was the build red (failed) or yellow (unstable)?
>
> Mark Waite
>
> ________________________________
> From: Sami Tikka <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Sunday, September 16, 2012 11:08 AM
>
> Subject: Re: retry build upon failure?
>
> Could you post exactly what you see in the console log of the failed build?
> Also, it might help if you could post your job config.xml or put in
> pastebin/gist so we could take a look.
>
> -- Sami
>
> Ed Young <[hidden email]> kirjoitti 14.9.2012 kello 22.38:
>
>> I installed the Naginator Plugin (v1.8) because the retry failed build
>> plugin indicated that it was deprecated in favor of the Naginatore
>> plugin.
>>
>> It doesn't seem to be working the way I expect it to however. I
>> configured it to simply
>>

>> Retry build after failure up to 3 times, with all other defaults, but
>> after a failure there is no retry. I configured it for a  fixed delay
>> and it still did not retry.
>>
>> It seems like a very simple configuration and the build is a hard
>> failure (test failures) so I can't tell where I'm going wrong.
>> Using Jenkins 1.436.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Sep 14, 2012 at 5:59 AM, William Soula
>> <[hidden email]> wrote:
>>> Or https://wiki.jenkins-ci.org/display/JENKINS/Retry+Failed+Builds+Plugin
>>>
>>>
>>>
>>>
>>>
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of nicolas de loof
>>> Sent: Friday, September 14, 2012 6:16 AM
>>> To: [hidden email]
>>> Subject: Re: retry build upon failure?
>>>
>>>
>>>
>>> see http://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
>>>

>>> 2012/9/14 Ed Young <[hidden email]>
>>>
>>> Is it possible to retry a build upon a failure? Maybe with a retry count?
>>>
>>>
>>
>>
>>
>> --
>> - Ed
>
>



--
- Ed









--
- Ed
Loading...