Jenkins › Jenkins users

Is every build kept in the JVM?

Classic

List

Threaded

7 messages Options

Mandeville, Rob

Is every build kept in the JVM?

I’m getting OOM exceptions left and right in my Jenkins instance. It’s a fairly large installation, with over 100 slave nodes, and I’m running in Java 6 HotSpot. I generated a heap dump (great feature to do that via the Web page, BTW) and finding something that was surprising to me.

It appears that every build that Jenkins “remembers” is kept in the JVM itself. That is, when I’m keeping the last 400 runs of a given job, I have the metadata (though not the logs, I hope…) of all 400 runs in the JVM. Is this in fact the case? Is there a way to store build information historically without keeping it in core? Is this a problem for other users?

Thanks in advance,

--Rob

The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.

barrett

Re: Is every build kept in the JVM?

I wonder if that is intentional or a memory leak.

Great to know this by the way. Does it load all the metadata on service startup or does it slowly accumulate?

-barrett

On Monday, September 10, 2012 8:58:44 AM UTC-4, Mandeville, Rob wrote:

I’m getting OOM exceptions left and right in my Jenkins instance. It’s a fairly large installation, with over 100 slave nodes, and I’m running in Java 6 HotSpot. I generated a heap dump (great feature to do that via the Web page, BTW) and finding something that was surprising to me.

It appears that every build that Jenkins “remembers” is kept in the JVM itself. That is, when I’m keeping the last 400 runs of a given job, I have the metadata (though not the logs, I hope…) of all 400 runs in the JVM. Is this in fact the case? Is there a way to store build information historically without keeping it in core? Is this a problem for other users?

Thanks in advance,

--Rob

The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.

Mark Waite

Re: Is every build kept in the JVM?

I believe that is intentional. The technique I've commonly used is to use the Jenkins setting to trim the history of jobs to a smaller set (in my case, a week's worth or less).

If you have a use case which requires that you keep so long a history, you will probably find that Jenkins starts more and more slowly as your history grows.

Mark Waite

From: bearrito <[hidden email]>
To: [hidden email]
Cc: [hidden email]
Sent: Monday, September 10, 2012 11:08 AM
Subject: Re: Is every build kept in the JVM?

I wonder if that is intentional or a memory leak.

Great to know this by the way. Does it load all the metadata on service startup or does it slowly accumulate?

-barrett

On Monday, September 10, 2012 8:58:44 AM UTC-4, Mandeville, Rob wrote:

I’m getting OOM exceptions left and right in my Jenkins instance. It’s a fairly large installation, with over 100 slave nodes, and I’m running in Java 6 HotSpot. I generated a heap dump (great feature to do that via the Web page, BTW) and finding something that was surprising to me.

It appears that every build that Jenkins “remembers” is kept in the JVM itself. That is, when I’m keeping the last 400 runs of a given job, I have the metadata (though not the logs, I hope…) of all 400 runs in the JVM. Is this in fact the case? Is there a way to store build information historically without keeping it in core? Is this a problem for other users?

Thanks in advance,

--Rob

The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.

Christoph Kutzinski

Re: Is every build kept in the JVM?

In reply to this post by barrett

I'd say it's intentionally AND a memory leak ;-)
This is IMO one of the major pain points with Jenkins. We keep 'only' 3
weeks worth of builds and still it takes >40 minutes for a Jenkins
startup - spending the majority of the time in loading the builds.

Unfortunately, it doesn't seem to be easy to fix this without breaking
existing API. I had once tried to figure out an easy way to load old
builds only on demand, but gave up after several hours.

As Mark already said, the currently preferred solution is to remove old
builds.

Christoph

Am 10.09.2012 19:08, schrieb bearrito:

> I wonder if that is intentional or a memory leak.
>
> Great to know this by the way. Does it load all the metadata on service
> startup or does it slowly accumulate?
>
> -barrett
>
> On Monday, September 10, 2012 8:58:44 AM UTC-4, Mandeville, Rob wrote:
>
> I’m getting OOM exceptions left and right in my Jenkins instance.
> It’s a fairly large installation, with over 100 slave nodes, and I’m
> running in Java 6 HotSpot. I generated a heap dump (great feature
> to do that via the Web page, BTW) and finding something that was
> surprising to me.
>
> It appears that every build that Jenkins “remembers” is kept in the
> JVM itself. That is, when I’m keeping the last 400 runs of a given
> job, I have the metadata (though not the logs, I hope…) of all 400
> runs in the JVM. Is this in fact the case? Is there a way to store
> build information historically without keeping it in core? Is this
> a problem for other users?
>
> Thanks in advance,
>
> --Rob
>
> The information in this message is for the intended recipient(s)
> only and may be the proprietary and/or confidential property of
> Litle & Co., LLC, and thus protected from disclosure. If you are not
> the intended recipient(s), or an employee or agent responsible for
> delivering this message to the intended recipient, you are hereby
> notified that any use, dissemination, distribution or copying of
> this communication is prohibited. If you have received this
> communication in error, please notify Litle & Co. immediately by
> replying to this message and then promptly deleting it and your
> reply permanently from your computer.
>

Richard Bywater-2

Re: Is every build kept in the JVM?

Agreed - I think this is one thing that, if fixed/redesigned, would be
a big help! Having said that I'm guessing from all the comments that
it would mean a pretty big effort?

Richard.

On Tue, Sep 11, 2012 at 6:00 AM, Christoph Kutzinski <[hidden email]> wrote:

> I'd say it's intentionally AND a memory leak ;-)
> This is IMO one of the major pain points with Jenkins. We keep 'only' 3
> weeks worth of builds and still it takes >40 minutes for a Jenkins startup -
> spending the majority of the time in loading the builds.
>
>
> Unfortunately, it doesn't seem to be easy to fix this without breaking
> existing API. I had once tried to figure out an easy way to load old builds
> only on demand, but gave up after several hours.
>
> As Mark already said, the currently preferred solution is to remove old
> builds.
>
>
> Christoph
>
> Am 10.09.2012 19:08, schrieb bearrito:
>
>> I wonder if that is intentional or a memory leak.
>>
>> Great to know this by the way. Does it load all the metadata on service
>> startup or does it slowly accumulate?
>>
>> -barrett
>>
>> On Monday, September 10, 2012 8:58:44 AM UTC-4, Mandeville, Rob wrote:
>>
>> I’m getting OOM exceptions left and right in my Jenkins instance.
>> It’s a fairly large installation, with over 100 slave nodes, and I’m
>> running in Java 6 HotSpot. I generated a heap dump (great feature
>> to do that via the Web page, BTW) and finding something that was
>> surprising to me.
>>
>> It appears that every build that Jenkins “remembers” is kept in the
>> JVM itself. That is, when I’m keeping the last 400 runs of a given
>> job, I have the metadata (though not the logs, I hope…) of all 400
>> runs in the JVM. Is this in fact the case? Is there a way to store
>> build information historically without keeping it in core? Is this
>> a problem for other users?
>>
>> Thanks in advance,
>>
>> --Rob
>>
>> The information in this message is for the intended recipient(s)
>> only and may be the proprietary and/or confidential property of
>> Litle & Co., LLC, and thus protected from disclosure. If you are not
>> the intended recipient(s), or an employee or agent responsible for
>> delivering this message to the intended recipient, you are hereby
>> notified that any use, dissemination, distribution or copying of
>> this communication is prohibited. If you have received this
>> communication in error, please notify Litle & Co. immediately by
>> replying to this message and then promptly deleting it and your
>> reply permanently from your computer.
>>
>

barrett

Re: Is every build kept in the JVM?

In reply to this post by Christoph Kutzinski

Can I ask how many jobs you have running? Also, how many jobs do you accrue during a three week period?

My job count is slowly increasing and I'm worried about running into this sort of issue. I don't have any reason to keep old builds so I can keep it fairly trim.

-barrett

On Monday, September 10, 2012 2:00:17 PM UTC-4, kutzi wrote:

I'd say it's intentionally AND a memory leak ;-)
This is IMO one of the major pain points with Jenkins. We keep 'only' 3
weeks worth of builds and still it takes >40 minutes for a Jenkins
startup - spending the majority of the time in loading the builds.

Unfortunately, it doesn't seem to be easy to fix this without breaking
existing API. I had once tried to figure out an easy way to load old
builds only on demand, but gave up after several hours.

As Mark already said, the currently preferred solution is to remove old
builds.

Christoph

Am 10.09.2012 19:08, schrieb bearrito:

> I wonder if that is intentional or a memory leak.
>
> Great to know this by the way. Does it load all the metadata on service
> startup or does it slowly accumulate?
>
> -barrett
>
> On Monday, September 10, 2012 8:58:44 AM UTC-4, Mandeville, Rob wrote:
>
> I’m getting OOM exceptions left and right in my Jenkins instance.
> It’s a fairly large installation, with over 100 slave nodes, and I’m
> running in Java 6 HotSpot. I generated a heap dump (great feature
> to do that via the Web page, BTW) and finding something that was
> surprising to me.
>
> It appears that every build that Jenkins “remembers” is kept in the
> JVM itself. That is, when I’m keeping the last 400 runs of a given
> job, I have the metadata (though not the logs, I hope…) of all 400
> runs in the JVM. Is this in fact the case? Is there a way to store
> build information historically without keeping it in core? Is this
> a problem for other users?
>
> Thanks in advance,
>
> --Rob
>
> The information in this message is for the intended recipient(s)
> only and may be the proprietary and/or confidential property of
> Litle & Co., LLC, and thus protected from disclosure. If you are not
> the intended recipient(s), or an employee or agent responsible for
> delivering this message to the intended recipient, you are hereby
> notified that any use, dissemination, distribution or copying of
> this communication is prohibited. If you have received this
> communication in error, please notify Litle & Co. immediately by
> replying to this message and then promptly deleting it and your
> reply permanently from your computer.
>

Christopher Orr

Re: Is every build kept in the JVM?

In reply to this post by Christoph Kutzinski

For the benefit of anybody not on the developers' mailing list, Kohsuke
is currently implementing lazy-loading of job information!

This is currently running on ci.jenkins-ci.org and you can beta test the
latest lazy-load build by grabbing jenkins.war from there:
https://ci.jenkins-ci.org/job/jenkins_lazy_load/

The relevant mailing list thread is here:
https://groups.google.com/group/jenkinsci-dev/browse_thread/thread/ae2e44f4b5b6903a

If you have issues with this build, there's a thread on GitHub:
https://github.com/jenkinsci/jenkins/pull/573

Regards,
Chris

On 09/10/2012 08:00 PM, Christoph Kutzinski wrote:

> I'd say it's intentionally AND a memory leak ;-)
> This is IMO one of the major pain points with Jenkins. We keep 'only' 3
> weeks worth of builds and still it takes >40 minutes for a Jenkins
> startup - spending the majority of the time in loading the builds.
>
>
> Unfortunately, it doesn't seem to be easy to fix this without breaking
> existing API. I had once tried to figure out an easy way to load old
> builds only on demand, but gave up after several hours.
>
> As Mark already said, the currently preferred solution is to remove old
> builds.
>
>
> Christoph
>
> Am 10.09.2012 19:08, schrieb bearrito:
>> I wonder if that is intentional or a memory leak.
>>
>> Great to know this by the way. Does it load all the metadata on service
>> startup or does it slowly accumulate?
>>
>> -barrett
>>
>> On Monday, September 10, 2012 8:58:44 AM UTC-4, Mandeville, Rob wrote:
>>
>> I’m getting OOM exceptions left and right in my Jenkins instance.
>> It’s a fairly large installation, with over 100 slave nodes, and I’m
>> running in Java 6 HotSpot. I generated a heap dump (great feature
>> to do that via the Web page, BTW) and finding something that was
>> surprising to me.
>>
>> It appears that every build that Jenkins “remembers” is kept in the
>> JVM itself. That is, when I’m keeping the last 400 runs of a given
>> job, I have the metadata (though not the logs, I hope…) of all 400
>> runs in the JVM. Is this in fact the case? Is there a way to store
>> build information historically without keeping it in core? Is this
>> a problem for other users?
>>
>> Thanks in advance,
>>
>> --Rob
>>
>> The information in this message is for the intended recipient(s)
>> only and may be the proprietary and/or confidential property of
>> Litle & Co., LLC, and thus protected from disclosure. If you are not
>> the intended recipient(s), or an employee or agent responsible for
>> delivering this message to the intended recipient, you are hereby
>> notified that any use, dissemination, distribution or copying of
>> this communication is prohibited. If you have received this
>> communication in error, please notify Litle & Co. immediately by
>> replying to this message and then promptly deleting it and your
>> reply permanently from your computer.
>>
>