Immortal phantom build

[Cliffano Subagio]

Hi,

One of my Jenkins jobs is running (the job page showing a build in progress), but the console output page only shows the spinning wheel without anything else
On the dashboard page, none of the executors is running that job, all executors on all slaves are idle.

I got the /threadDump output when it happened, nothing obvious that screams error there.
Any suggestion where else I should look to investigate what's causing this problem?

I can't stop that phantom job in progress, clicked the red x button and the build stayed in progress.

This is blocking the next build for the same job, which in turn blocks my delivery pipeline.

This happened few times before for the past few months, the workaround was to restart the master, but it's obviously not our preference since it takes a while to start back up.


Cheers,
Cliff.

[Richard Bywater-2]

I think the last few times that's happened to me its been memory
related (e.g. out of PermGen). Think it spat some errors out to the
Winstone wrapper.log.

Possibly not related to your issue but just in case...

Cheers
Richard.

On Fri, Aug 10, 2012 at 11:44 AM, Cliffano Subagio <[hidden email]> wrote:

> Hi,
>
> One of my Jenkins jobs is running (the job page showing a build in
> progress), but the console output page only shows the spinning wheel without
> anything else
> On the dashboard page, none of the executors is running that job, all
> executors on all slaves are idle.
>
> I got the /threadDump output when it happened, nothing obvious that screams
> error there.
> Any suggestion where else I should look to investigate what's causing this
> problem?
>
> I can't stop that phantom job in progress, clicked the red x button and the
> build stayed in progress.
>
> This is blocking the next build for the same job, which in turn blocks my
> delivery pipeline.
> This happened few times before for the past few months, the workaround was
> to restart the master, but it's obviously not our preference since it takes
> a while to start back up.
>
>
> Cheers,
> Cliff.

[William Soula]

Is it possible to just delete the build?  I've seen builds from other jobs show up in a different job's list and since the build number was much higher it hid the test failures in the real build because it was the lastSuccessfulBuild due to the build number being so big.  I solved that problem by just deleting the offending build and it didn't affect the job the phantom build came from.  But you might not have the delete button since the phantom job is running.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Richard Bywater
Sent: Friday, August 10, 2012 12:36 AM
To: [hidden email]
Subject: Re: Immortal phantom build

I think the last few times that's happened to me its been memory related (e.g. out of PermGen). Think it spat some errors out to the Winstone wrapper.log.

Possibly not related to your issue but just in case...

Cheers
Richard.

On Fri, Aug 10, 2012 at 11:44 AM, Cliffano Subagio <[hidden email]> wrote:

> Hi,
>
> One of my Jenkins jobs is running (the job page showing a build in
> progress), but the console output page only shows the spinning wheel
> without anything else On the dashboard page, none of the executors is
> running that job, all executors on all slaves are idle.
>
> I got the /threadDump output when it happened, nothing obvious that
> screams error there.
> Any suggestion where else I should look to investigate what's causing
> this problem?
>
> I can't stop that phantom job in progress, clicked the red x button
> and the build stayed in progress.
>
> This is blocking the next build for the same job, which in turn blocks
> my delivery pipeline.
> This happened few times before for the past few months, the workaround
> was to restart the master, but it's obviously not our preference since
> it takes a while to start back up.
>
>
> Cheers,
> Cliff.

[Cliffano Subagio]

The weird thing is that all slaves were still working and I could build other jobs, so if it's memory related issue, then it might affect that particular phantom build.
The other problem is on identifying which slave is having the problem. I checked the logs of each one, and didn't find any error.
Thanks for pointing it though, I'll keep an eye on memory usage.

Cheers,
Cliff.

On Fri, Aug 10, 2012 at 3:36 PM, Richard Bywater <[hidden email]> wrote:

I think the last few times that's happened to me its been memory
related (e.g. out of PermGen). Think it spat some errors out to the
Winstone wrapper.log.

Possibly not related to your issue but just in case...

Cheers
Richard.

On Fri, Aug 10, 2012 at 11:44 AM, Cliffano Subagio <[hidden email]> wrote:
> Hi,
>
> One of my Jenkins jobs is running (the job page showing a build in
> progress), but the console output page only shows the spinning wheel without
> anything else
> On the dashboard page, none of the executors is running that job, all
> executors on all slaves are idle.
>
> I got the /threadDump output when it happened, nothing obvious that screams
> error there.
> Any suggestion where else I should look to investigate what's causing this
> problem?
>
> I can't stop that phantom job in progress, clicked the red x button and the
> build stayed in progress.
>
> This is blocking the next build for the same job, which in turn blocks my
> delivery pipeline.
> This happened few times before for the past few months, the workaround was
> to restart the master, but it's obviously not our preference since it takes
> a while to start back up.
>
>
> Cheers,
> Cliff.

[Cliffano Subagio]

Do you mean kill the build by deleting something directly on the filesystem?
I tried stopping the build (the red x icon next to the progress bar), but it didn't do anything, phantom build kept going.

I wonder what's causing the job to think that there's a build running (job page showing a running phantom build) when Jenkins itself doesn't (dashboard page shows all executors being idle). Surely there's an endpoint that displays the build status, perhaps the dashboard page is retrieving the build status from a different endpoint to the job page's.

Cheers,
Cliff.

On Fri, Aug 10, 2012 at 11:54 PM, William Soula <[hidden email]> wrote:

Is it possible to just delete the build?  I've seen builds from other jobs show up in a different job's list and since the build number was much higher it hid the test failures in the real build because it was the lastSuccessfulBuild due to the build number being so big.  I solved that problem by just deleting the offending build and it didn't affect the job the phantom build came from.  But you might not have the delete button since the phantom job is running.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Richard Bywater
Sent: Friday, August 10, 2012 12:36 AM
To: [hidden email]
Subject: Re: Immortal phantom build

I think the last few times that's happened to me its been memory related (e.g. out of PermGen). Think it spat some errors out to the Winstone wrapper.log.

Possibly not related to your issue but just in case...

Cheers
Richard.

On Fri, Aug 10, 2012 at 11:44 AM, Cliffano Subagio <[hidden email]> wrote:
> Hi,
>
> One of my Jenkins jobs is running (the job page showing a build in
> progress), but the console output page only shows the spinning wheel
> without anything else On the dashboard page, none of the executors is
> running that job, all executors on all slaves are idle.
>
> I got the /threadDump output when it happened, nothing obvious that
> screams error there.
> Any suggestion where else I should look to investigate what's causing
> this problem?
>
> I can't stop that phantom job in progress, clicked the red x button
> and the build stayed in progress.
>
> This is blocking the next build for the same job, which in turn blocks
> my delivery pipeline.
> This happened few times before for the past few months, the workaround
> was to restart the master, but it's obviously not our preference since
> it takes a while to start back up.
>
>
> Cheers,
> Cliff.