Server usage pegged at 99%
Locked
12
12
Server usage pegged at 99%
 
|
Hello,
For some reason, if I watch top on my master while I browse around on the web interface, the java process stays pegged at 99-100% and it takes several seconds to render. Is there a common reason for that? I have 2Gb allocated to jenkins and the res is only 750MB, so I don't think it's GC churn. Thanks, andrew -- -- Andrew Melo |
Re: Server usage pegged at 99%
|
On Tue, Jul 31, 2012 at 1:46 PM, Andrew Melo <[hidden email]> wrote:
> Hello, > > For some reason, if I watch top on my master while I browse around on > the web interface, the java process stays pegged at 99-100% and it > takes several seconds to render. Is there a common reason for that? I > have 2Gb allocated to jenkins and the res is only 750MB, so I don't > think it's GC churn. Is it running on a linux host that has been up since at least June 30th? If so it could be the leap-second bug. If so, resetting the time will fix it. -- Les Mikesell [hidden email] |
Re: Server usage pegged at 99%
|
|
On Tue, Jul 31, 2012 at 2:16 PM, Les Mikesell <[hidden email]> wrote:
> On Tue, Jul 31, 2012 at 1:46 PM, Andrew Melo <[hidden email]> wrote: >> Hello, >> >> For some reason, if I watch top on my master while I browse around on >> the web interface, the java process stays pegged at 99-100% and it >> takes several seconds to render. Is there a common reason for that? I >> have 2Gb allocated to jenkins and the res is only 750MB, so I don't >> think it's GC churn. > > Is it running on a linux host that has been up since at least June > 30th? If so it could be the leap-second bug. If so, resetting the > time will fix it. Resetting the time (ntp?) or the jenkins server? Either way, it's been an issue since before then, but I'll give whichever one a try Thanks, Andrew > > -- > Les Mikesell > [hidden email] -- -- Andrew Melo |
Re: Server usage pegged at 99%
|
|
On Tue, Jul 31, 2012 at 2:17 PM, Andrew Melo <[hidden email]> wrote:
> >> >>> For some reason, if I watch top on my master while I browse around on >>> the web interface, the java process stays pegged at 99-100% and it >>> takes several seconds to render. Is there a common reason for that? I >>> have 2Gb allocated to jenkins and the res is only 750MB, so I don't >>> think it's GC churn. >> >> Is it running on a linux host that has been up since at least June >> 30th? If so it could be the leap-second bug. If so, resetting the >> time will fix it. > > Resetting the time (ntp?) or the jenkins server? > > Either way, it's been an issue since before then, but I'll give > whichever one a try It is a linux kernel bug triggered by ntp on the day of a leap second. Resetting the system time any way other than ntp will fix it. For example: date -s "`date`" But if it happened before June 30th or the system has been rebooted since, this is not the problem. -- Les Mikesell [hidden email] |
Re: Server usage pegged at 99%
|
|
On Tue, Jul 31, 2012 at 2:36 PM, Les Mikesell <[hidden email]> wrote:
> On Tue, Jul 31, 2012 at 2:17 PM, Andrew Melo <[hidden email]> wrote: >> >> >>>> For some reason, if I watch top on my master while I browse around on >>>> the web interface, the java process stays pegged at 99-100% and it >>>> takes several seconds to render. Is there a common reason for that? I >>>> have 2Gb allocated to jenkins and the res is only 750MB, so I don't >>>> think it's GC churn. >>> >>> Is it running on a linux host that has been up since at least June >>> 30th? If so it could be the leap-second bug. If so, resetting the >>> time will fix it. >> >> Resetting the time (ntp?) or the jenkins server? >> >> Either way, it's been an issue since before then, but I'll give >> whichever one a try > > It is a linux kernel bug triggered by ntp on the day of a leap second. > Resetting the system time any way other than ntp will fix it. > For example: > date -s "`date`" > > But if it happened before June 30th or the system has been rebooted > since, this is not the problem. Well, and it's only when i'm using the web interface (or if background stuff is happening) > > -- > Les Mikesell > [hidden email] -- -- Andrew Melo |
Re: Server usage pegged at 99%
|
|
On Tue, Jul 31, 2012 at 2:38 PM, Andrew Melo <[hidden email]> wrote:
>>> >> But if it happened before June 30th or the system has been rebooted >> since, this is not the problem. > > Well, and it's only when i'm using the web interface (or if background > stuff is happening) > It affects the linux futex() system call that is used mostly in threaded applications (so you see it in java). And I think it is sort of a race condition where the extra CPU use happens at random. -- Les Mikesell [hidden email] |
Re: Server usage pegged at 99%
|
|
On Tue, Jul 31, 2012 at 2:54 PM, Les Mikesell <[hidden email]> wrote:
> On Tue, Jul 31, 2012 at 2:38 PM, Andrew Melo <[hidden email]> wrote: >>>> >>> But if it happened before June 30th or the system has been rebooted >>> since, this is not the problem. >> >> Well, and it's only when i'm using the web interface (or if background >> stuff is happening) >> > > It affects the linux futex() system call that is used mostly in > threaded applications (so you see it in java). And I think it is > sort of a race condition where the extra CPU use happens at random. Well, I restarted it and reset the date and it didn't seem to help. I'm pretty helpless when it comes to java, but is there some sort of way I can attach a profiler to the process and see what it spins on? Thanks > > -- > Les Mikesell > [hidden email] -- -- Andrew Melo |
Re: Re: Server usage pegged at 99%
|
quick way how to look what the thread consuming CPU is doing is to do thread
dump (e.g. using jstack $PID) and use top with threads on (H option) and then look up, see e.g. http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- consuming-most-cpu/ On Wednesday 01 August 2012 09:25:08 Andrew Melo wrote: > On Tue, Jul 31, 2012 at 2:54 PM, Les Mikesell <[hidden email]> wrote: > > On Tue, Jul 31, 2012 at 2:38 PM, Andrew Melo <[hidden email]> wrote: > >>> But if it happened before June 30th or the system has been rebooted > >>> since, this is not the problem. > >> > >> Well, and it's only when i'm using the web interface (or if background > >> stuff is happening) > > > > It affects the linux futex() system call that is used mostly in > > threaded applications (so you see it in java). And I think it is > > sort of a race condition where the extra CPU use happens at random. > > Well, I restarted it and reset the date and it didn't seem to help. > I'm pretty helpless when it comes to java, but is there some sort of > way I can attach a profiler to the process and see what it spins on? > > Thanks > > > -- > > > > Les Mikesell > > > > [hidden email] |
Re: Re: Server usage pegged at 99%
|
|
On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> wrote:
> quick way how to look what the thread consuming CPU is doing is to do thread > dump (e.g. using jstack $PID) and use top with threads on (H option) and then > look up, see e.g. > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- > consuming-most-cpu/ I see. I apparently don't have jstack on this machine :/. Does it only come with the JDK, or can I find it somewhere on the JRE? Once I find the offending thread, should it be pretty obvious what it does? > > On Wednesday 01 August 2012 09:25:08 Andrew Melo wrote: >> On Tue, Jul 31, 2012 at 2:54 PM, Les Mikesell <[hidden email]> wrote: >> > On Tue, Jul 31, 2012 at 2:38 PM, Andrew Melo <[hidden email]> > wrote: >> >>> But if it happened before June 30th or the system has been rebooted >> >>> since, this is not the problem. >> >> >> >> Well, and it's only when i'm using the web interface (or if background >> >> stuff is happening) >> > >> > It affects the linux futex() system call that is used mostly in >> > threaded applications (so you see it in java). And I think it is >> > sort of a race condition where the extra CPU use happens at random. >> >> Well, I restarted it and reset the date and it didn't seem to help. >> I'm pretty helpless when it comes to java, but is there some sort of >> way I can attach a profiler to the process and see what it spins on? >> >> Thanks >> >> > -- >> > >> > Les Mikesell >> > >> > [hidden email] -- -- Andrew Melo |
Re: Re: Re: Server usage pegged at 99%
|
|
On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote:
> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> wrote: > > quick way how to look what the thread consuming CPU is doing is to do > > thread dump (e.g. using jstack $PID) and use top with threads on (H > > option) and then look up, see e.g. > > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- > > consuming-most-cpu/ > > I see. I apparently don't have jstack on this machine :/. Does it only > come with the JDK, or can I find it somewhere on the JRE? Once I find > the offending thread, should it be pretty obvious what it does? jstack is part of JDK you can see the stack trace via Jenkins UI navigating to $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides thread IDs. Once you identify the offending thread, it should be obvious what it does (but it may not be obvious why it does what it does:-) |
Re: Re: Re: Server usage pegged at 99%
|
|
On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek <[hidden email]> wrote:
> On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> wrote: >> > quick way how to look what the thread consuming CPU is doing is to do >> > thread dump (e.g. using jstack $PID) and use top with threads on (H >> > option) and then look up, see e.g. >> > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >> > consuming-most-cpu/ >> >> I see. I apparently don't have jstack on this machine :/. Does it only >> come with the JDK, or can I find it somewhere on the JRE? Once I find >> the offending thread, should it be pretty obvious what it does? > > jstack is part of JDK > > you can see the stack trace via Jenkins UI navigating to > $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version provides > thread IDs. > > > Once you identify the offending thread, it should be obvious what it does (but > it may not be obvious why it does what it does:-) Okay, I installed the jdk, and I looked some more. Using top, I see one jenkins thread taking the lionsshare of the time: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java Then if I run jstack, I get the following backtrace: https://gist.github.com/3228105 Does that look useful at all? Thanks, Andrew -- -- Andrew Melo |
Re: Re: Re: Re: Server usage pegged at 99%
|
|
Looks like you it does search for user's email:
hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor and spends time parsing changelogs: hudson.scm.SubversionChangeLogParser.parse I guess you have quite large instance, otherwise this operation would be quite fast. If you have some job, which has set up option to send an email to devs who broke the build, if the user hasn't specified an email, Jenkins tries to find it e.g. in git or SVN changelogs and search all projects and builds so if you have large instance with several dozen thousands of builds if can take pretty long time. You can fix it by setting up correct email for the user. If you have installed git plugin, make sure you have 1.1.16 (I hope it was fixed in this version) or higher. Git plugin made this search even if the user has set up email correctly On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: > On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek <[hidden email]> wrote: > > On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: > >> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> wrote: > >> > quick way how to look what the thread consuming CPU is doing is to do > >> > thread dump (e.g. using jstack $PID) and use top with threads on (H > >> > option) and then look up, see e.g. > >> > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- > >> > consuming-most-cpu/ > >> > >> I see. I apparently don't have jstack on this machine :/. Does it only > >> come with the JDK, or can I find it somewhere on the JRE? Once I find > >> the offending thread, should it be pretty obvious what it does? > > > > jstack is part of JDK > > > > you can see the stack trace via Jenkins UI navigating to > > $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version > > provides thread IDs. > > > > > > Once you identify the offending thread, it should be obvious what it does > > (but it may not be obvious why it does what it does:-) > > Okay, I installed the jdk, and I looked some more. > > Using top, I see one jenkins thread taking the lionsshare of the time: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java > 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java > 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java > 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java > 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java > 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java > 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java > 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java > 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java > 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java > 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java > 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java > 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java > 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java > 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java > 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java > 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java > 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java > 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java > 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java > 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java > 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java > 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java > 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java > 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java > 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java > 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java > 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java > > > Then if I run jstack, I get the following backtrace: > > https://gist.github.com/3228105 > > Does that look useful at all? > > Thanks, > Andrew |
|
This is a huge issue with the email-ext plugin as well when it does
email address resolution. Quite a number of people have complained about how long it takes. I have yet to come up with a good solution. The perforce plugin has a similar issue. slide On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek <[hidden email]> wrote: > Looks like you it does search for user's email: > hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor > and spends time parsing changelogs: > hudson.scm.SubversionChangeLogParser.parse > > I guess you have quite large instance, otherwise this operation would be quite > fast. > If you have some job, which has set up option to send an email to devs who > broke the build, if the user hasn't specified an email, Jenkins tries to find it > e.g. in git or SVN changelogs and search all projects and builds so if you > have large instance with several dozen thousands of builds if can take pretty > long time. > > You can fix it by setting up correct email for the user. > If you have installed git plugin, make sure you have 1.1.16 (I hope it was > fixed in this version) or higher. Git plugin made this search even if the user > has set up email correctly > > On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek <[hidden email]> > wrote: >> > On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >> >> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> > wrote: >> >> > quick way how to look what the thread consuming CPU is doing is to do >> >> > thread dump (e.g. using jstack $PID) and use top with threads on (H >> >> > option) and then look up, see e.g. >> >> > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >> >> > consuming-most-cpu/ >> >> >> >> I see. I apparently don't have jstack on this machine :/. Does it only >> >> come with the JDK, or can I find it somewhere on the JRE? Once I find >> >> the offending thread, should it be pretty obvious what it does? >> > >> > jstack is part of JDK >> > >> > you can see the stack trace via Jenkins UI navigating to >> > $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version >> > provides thread IDs. >> > >> > >> > Once you identify the offending thread, it should be obvious what it does >> > (but it may not be obvious why it does what it does:-) >> >> Okay, I installed the jdk, and I looked some more. >> >> Using top, I see one jenkins thread taking the lionsshare of the time: >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >> >> >> Then if I run jstack, I get the following backtrace: >> >> https://gist.github.com/3228105 >> >> Does that look useful at all? >> >> Thanks, >> Andrew -- Website: http://earl-of-code.com |
Re: Re: Re: Re: Server usage pegged at 99%
|
|
In reply to this post by Vojtech Juranek
On Wed, Aug 1, 2012 at 11:17 AM, Vojtech Juranek <[hidden email]> wrote:
> Looks like you it does search for user's email: > hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor > and spends time parsing changelogs: > hudson.scm.SubversionChangeLogParser.parse > > I guess you have quite large instance, otherwise this operation would be quite > fast. > If you have some job, which has set up option to send an email to devs who > broke the build, if the user hasn't specified an email, Jenkins tries to find it > e.g. in git or SVN changelogs and search all projects and builds so if you > have large instance with several dozen thousands of builds if can take pretty > long time. > > You can fix it by setting up correct email for the user. > If you have installed git plugin, make sure you have 1.1.16 (I hope it was > fixed in this version) or higher. Git plugin made this search even if the user > has set up email correctly So, if I understand right, jenkins is looking for email addresses, so I need to make sure that every user that registers has a valid address? > > On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek <[hidden email]> > wrote: >> > On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >> >> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> > wrote: >> >> > quick way how to look what the thread consuming CPU is doing is to do >> >> > thread dump (e.g. using jstack $PID) and use top with threads on (H >> >> > option) and then look up, see e.g. >> >> > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >> >> > consuming-most-cpu/ >> >> >> >> I see. I apparently don't have jstack on this machine :/. Does it only >> >> come with the JDK, or can I find it somewhere on the JRE? Once I find >> >> the offending thread, should it be pretty obvious what it does? >> > >> > jstack is part of JDK >> > >> > you can see the stack trace via Jenkins UI navigating to >> > $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version >> > provides thread IDs. >> > >> > >> > Once you identify the offending thread, it should be obvious what it does >> > (but it may not be obvious why it does what it does:-) >> >> Okay, I installed the jdk, and I looked some more. >> >> Using top, I see one jenkins thread taking the lionsshare of the time: >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >> >> >> Then if I run jstack, I get the following backtrace: >> >> https://gist.github.com/3228105 >> >> Does that look useful at all? >> >> Thanks, >> Andrew -- -- Andrew Melo |
Re: Re: Re: Re: Server usage pegged at 99%
|
|
In reply to this post by slide
On Wed, Aug 1, 2012 at 11:22 AM, Slide <[hidden email]> wrote:
> This is a huge issue with the email-ext plugin as well when it does > email address resolution. Quite a number of people have complained > about how long it takes. I have yet to come up with a good solution. > The perforce plugin has a similar issue. If I just stick a dummy address in every user's profile, will that work? > > slide > > On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek <[hidden email]> wrote: >> Looks like you it does search for user's email: >> hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor >> and spends time parsing changelogs: >> hudson.scm.SubversionChangeLogParser.parse >> >> I guess you have quite large instance, otherwise this operation would be quite >> fast. >> If you have some job, which has set up option to send an email to devs who >> broke the build, if the user hasn't specified an email, Jenkins tries to find it >> e.g. in git or SVN changelogs and search all projects and builds so if you >> have large instance with several dozen thousands of builds if can take pretty >> long time. >> >> You can fix it by setting up correct email for the user. >> If you have installed git plugin, make sure you have 1.1.16 (I hope it was >> fixed in this version) or higher. Git plugin made this search even if the user >> has set up email correctly >> >> On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >>> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek <[hidden email]> >> wrote: >>> > On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >>> >> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> >> wrote: >>> >> > quick way how to look what the thread consuming CPU is doing is to do >>> >> > thread dump (e.g. using jstack $PID) and use top with threads on (H >>> >> > option) and then look up, see e.g. >>> >> > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >>> >> > consuming-most-cpu/ >>> >> >>> >> I see. I apparently don't have jstack on this machine :/. Does it only >>> >> come with the JDK, or can I find it somewhere on the JRE? Once I find >>> >> the offending thread, should it be pretty obvious what it does? >>> > >>> > jstack is part of JDK >>> > >>> > you can see the stack trace via Jenkins UI navigating to >>> > $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version >>> > provides thread IDs. >>> > >>> > >>> > Once you identify the offending thread, it should be obvious what it does >>> > (but it may not be obvious why it does what it does:-) >>> >>> Okay, I installed the jdk, and I looked some more. >>> >>> Using top, I see one jenkins thread taking the lionsshare of the time: >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >>> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >>> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >>> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >>> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >>> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >>> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >>> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >>> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >>> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >>> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >>> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >>> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >>> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >>> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >>> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >>> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >>> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >>> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >>> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >>> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >>> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >>> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >>> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >>> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >>> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >>> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >>> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >>> >>> >>> Then if I run jstack, I get the following backtrace: >>> >>> https://gist.github.com/3228105 >>> >>> Does that look useful at all? >>> >>> Thanks, >>> Andrew > > > > -- > Website: http://earl-of-code.com -- -- Andrew Melo |
|
|
No, because its only looking for the email address because it wants to
send an email to that user. On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo <[hidden email]> wrote: > On Wed, Aug 1, 2012 at 11:22 AM, Slide <[hidden email]> wrote: >> This is a huge issue with the email-ext plugin as well when it does >> email address resolution. Quite a number of people have complained >> about how long it takes. I have yet to come up with a good solution. >> The perforce plugin has a similar issue. > > If I just stick a dummy address in every user's profile, will that work? > >> >> slide >> >> On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek <[hidden email]> wrote: >>> Looks like you it does search for user's email: >>> hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor >>> and spends time parsing changelogs: >>> hudson.scm.SubversionChangeLogParser.parse >>> >>> I guess you have quite large instance, otherwise this operation would be quite >>> fast. >>> If you have some job, which has set up option to send an email to devs who >>> broke the build, if the user hasn't specified an email, Jenkins tries to find it >>> e.g. in git or SVN changelogs and search all projects and builds so if you >>> have large instance with several dozen thousands of builds if can take pretty >>> long time. >>> >>> You can fix it by setting up correct email for the user. >>> If you have installed git plugin, make sure you have 1.1.16 (I hope it was >>> fixed in this version) or higher. Git plugin made this search even if the user >>> has set up email correctly >>> >>> On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >>>> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek <[hidden email]> >>> wrote: >>>> > On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >>>> >> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> >>> wrote: >>>> >> > quick way how to look what the thread consuming CPU is doing is to do >>>> >> > thread dump (e.g. using jstack $PID) and use top with threads on (H >>>> >> > option) and then look up, see e.g. >>>> >> > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >>>> >> > consuming-most-cpu/ >>>> >> >>>> >> I see. I apparently don't have jstack on this machine :/. Does it only >>>> >> come with the JDK, or can I find it somewhere on the JRE? Once I find >>>> >> the offending thread, should it be pretty obvious what it does? >>>> > >>>> > jstack is part of JDK >>>> > >>>> > you can see the stack trace via Jenkins UI navigating to >>>> > $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version >>>> > provides thread IDs. >>>> > >>>> > >>>> > Once you identify the offending thread, it should be obvious what it does >>>> > (but it may not be obvious why it does what it does:-) >>>> >>>> Okay, I installed the jdk, and I looked some more. >>>> >>>> Using top, I see one jenkins thread taking the lionsshare of the time: >>>> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >>>> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >>>> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >>>> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >>>> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >>>> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >>>> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >>>> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >>>> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >>>> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >>>> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >>>> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >>>> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >>>> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >>>> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >>>> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >>>> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >>>> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >>>> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >>>> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >>>> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >>>> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >>>> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >>>> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >>>> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >>>> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >>>> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >>>> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >>>> >>>> >>>> Then if I run jstack, I get the following backtrace: >>>> >>>> https://gist.github.com/3228105 >>>> >>>> Does that look useful at all? >>>> >>>> Thanks, >>>> Andrew >> >> >> >> -- >> Website: http://earl-of-code.com > > > > -- > -- > Andrew Melo -- Website: http://earl-of-code.com |
Re: Re: Re: Re: Server usage pegged at 99%
|
|
On Wed, Aug 1, 2012 at 11:26 AM, Slide <[hidden email]> wrote:
> No, because its only looking for the email address because it wants to > send an email to that user. I don't know who's getting emailed. I don't remember setting it up for anything, and we actually wrote some scripts that turn jenkins success/failures into Github issues, so having jenkins also send emails would be redundant. I don't supposed there's a global flag to disable email? (I don't see one at "manage jenkins") -Andrew > > On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo <[hidden email]> wrote: >> On Wed, Aug 1, 2012 at 11:22 AM, Slide <[hidden email]> wrote: >>> This is a huge issue with the email-ext plugin as well when it does >>> email address resolution. Quite a number of people have complained >>> about how long it takes. I have yet to come up with a good solution. >>> The perforce plugin has a similar issue. >> >> If I just stick a dummy address in every user's profile, will that work? >> >>> >>> slide >>> >>> On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek <[hidden email]> wrote: >>>> Looks like you it does search for user's email: >>>> hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor >>>> and spends time parsing changelogs: >>>> hudson.scm.SubversionChangeLogParser.parse >>>> >>>> I guess you have quite large instance, otherwise this operation would be quite >>>> fast. >>>> If you have some job, which has set up option to send an email to devs who >>>> broke the build, if the user hasn't specified an email, Jenkins tries to find it >>>> e.g. in git or SVN changelogs and search all projects and builds so if you >>>> have large instance with several dozen thousands of builds if can take pretty >>>> long time. >>>> >>>> You can fix it by setting up correct email for the user. >>>> If you have installed git plugin, make sure you have 1.1.16 (I hope it was >>>> fixed in this version) or higher. Git plugin made this search even if the user >>>> has set up email correctly >>>> >>>> On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >>>>> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek <[hidden email]> >>>> wrote: >>>>> > On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >>>>> >> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> >>>> wrote: >>>>> >> > quick way how to look what the thread consuming CPU is doing is to do >>>>> >> > thread dump (e.g. using jstack $PID) and use top with threads on (H >>>>> >> > option) and then look up, see e.g. >>>>> >> > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >>>>> >> > consuming-most-cpu/ >>>>> >> >>>>> >> I see. I apparently don't have jstack on this machine :/. Does it only >>>>> >> come with the JDK, or can I find it somewhere on the JRE? Once I find >>>>> >> the offending thread, should it be pretty obvious what it does? >>>>> > >>>>> > jstack is part of JDK >>>>> > >>>>> > you can see the stack trace via Jenkins UI navigating to >>>>> > $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version >>>>> > provides thread IDs. >>>>> > >>>>> > >>>>> > Once you identify the offending thread, it should be obvious what it does >>>>> > (but it may not be obvious why it does what it does:-) >>>>> >>>>> Okay, I installed the jdk, and I looked some more. >>>>> >>>>> Using top, I see one jenkins thread taking the lionsshare of the time: >>>>> >>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >>>>> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >>>>> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >>>>> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >>>>> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >>>>> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >>>>> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >>>>> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >>>>> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >>>>> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >>>>> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >>>>> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >>>>> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >>>>> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >>>>> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >>>>> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >>>>> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >>>>> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >>>>> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >>>>> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >>>>> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >>>>> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >>>>> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >>>>> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >>>>> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >>>>> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >>>>> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >>>>> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >>>>> >>>>> >>>>> Then if I run jstack, I get the following backtrace: >>>>> >>>>> https://gist.github.com/3228105 >>>>> >>>>> Does that look useful at all? >>>>> >>>>> Thanks, >>>>> Andrew >>> >>> >>> >>> -- >>> Website: http://earl-of-code.com >> >> >> >> -- >> -- >> Andrew Melo > > > > -- > Website: http://earl-of-code.com -- -- Andrew Melo |
|
|
Can you gist your global config.xml and something from one of your
jobs as well? Please remember to sanitize it. On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo <[hidden email]> wrote: > On Wed, Aug 1, 2012 at 11:26 AM, Slide <[hidden email]> wrote: >> No, because its only looking for the email address because it wants to >> send an email to that user. > > I don't know who's getting emailed. I don't remember setting it up for > anything, and we actually wrote some scripts that turn jenkins > success/failures into Github issues, so having jenkins also send > emails would be redundant. > > I don't supposed there's a global flag to disable email? (I don't see > one at "manage jenkins") > > -Andrew >> >> On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo <[hidden email]> wrote: >>> On Wed, Aug 1, 2012 at 11:22 AM, Slide <[hidden email]> wrote: >>>> This is a huge issue with the email-ext plugin as well when it does >>>> email address resolution. Quite a number of people have complained >>>> about how long it takes. I have yet to come up with a good solution. >>>> The perforce plugin has a similar issue. >>> >>> If I just stick a dummy address in every user's profile, will that work? >>> >>>> >>>> slide >>>> >>>> On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek <[hidden email]> wrote: >>>>> Looks like you it does search for user's email: >>>>> hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor >>>>> and spends time parsing changelogs: >>>>> hudson.scm.SubversionChangeLogParser.parse >>>>> >>>>> I guess you have quite large instance, otherwise this operation would be quite >>>>> fast. >>>>> If you have some job, which has set up option to send an email to devs who >>>>> broke the build, if the user hasn't specified an email, Jenkins tries to find it >>>>> e.g. in git or SVN changelogs and search all projects and builds so if you >>>>> have large instance with several dozen thousands of builds if can take pretty >>>>> long time. >>>>> >>>>> You can fix it by setting up correct email for the user. >>>>> If you have installed git plugin, make sure you have 1.1.16 (I hope it was >>>>> fixed in this version) or higher. Git plugin made this search even if the user >>>>> has set up email correctly >>>>> >>>>> On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >>>>>> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek <[hidden email]> >>>>> wrote: >>>>>> > On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >>>>>> >> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> >>>>> wrote: >>>>>> >> > quick way how to look what the thread consuming CPU is doing is to do >>>>>> >> > thread dump (e.g. using jstack $PID) and use top with threads on (H >>>>>> >> > option) and then look up, see e.g. >>>>>> >> > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >>>>>> >> > consuming-most-cpu/ >>>>>> >> >>>>>> >> I see. I apparently don't have jstack on this machine :/. Does it only >>>>>> >> come with the JDK, or can I find it somewhere on the JRE? Once I find >>>>>> >> the offending thread, should it be pretty obvious what it does? >>>>>> > >>>>>> > jstack is part of JDK >>>>>> > >>>>>> > you can see the stack trace via Jenkins UI navigating to >>>>>> > $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version >>>>>> > provides thread IDs. >>>>>> > >>>>>> > >>>>>> > Once you identify the offending thread, it should be obvious what it does >>>>>> > (but it may not be obvious why it does what it does:-) >>>>>> >>>>>> Okay, I installed the jdk, and I looked some more. >>>>>> >>>>>> Using top, I see one jenkins thread taking the lionsshare of the time: >>>>>> >>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>>> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >>>>>> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >>>>>> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >>>>>> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >>>>>> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >>>>>> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >>>>>> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >>>>>> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >>>>>> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >>>>>> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >>>>>> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >>>>>> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >>>>>> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >>>>>> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >>>>>> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >>>>>> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >>>>>> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >>>>>> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >>>>>> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >>>>>> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >>>>>> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >>>>>> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >>>>>> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >>>>>> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >>>>>> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >>>>>> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >>>>>> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >>>>>> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >>>>>> >>>>>> >>>>>> Then if I run jstack, I get the following backtrace: >>>>>> >>>>>> https://gist.github.com/3228105 >>>>>> >>>>>> Does that look useful at all? >>>>>> >>>>>> Thanks, >>>>>> Andrew >>>> >>>> >>>> >>>> -- >>>> Website: http://earl-of-code.com >>> >>> >>> >>> -- >>> -- >>> Andrew Melo >> >> >> >> -- >> Website: http://earl-of-code.com > > > > -- > -- > Andrew Melo -- Website: http://earl-of-code.com |
Re: Re: Re: Re: Server usage pegged at 99%
|
|
On Wed, Aug 1, 2012 at 11:37 AM, Slide <[hidden email]> wrote:
> Can you gist your global config.xml and something from one of your > jobs as well? Please remember to sanitize it. We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ And the following is the gist for the job we run each commit (didn't make it in for some reason...) https://gist.github.com/3228599 > > On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo <[hidden email]> wrote: >> On Wed, Aug 1, 2012 at 11:26 AM, Slide <[hidden email]> wrote: >>> No, because its only looking for the email address because it wants to >>> send an email to that user. >> >> I don't know who's getting emailed. I don't remember setting it up for >> anything, and we actually wrote some scripts that turn jenkins >> success/failures into Github issues, so having jenkins also send >> emails would be redundant. >> >> I don't supposed there's a global flag to disable email? (I don't see >> one at "manage jenkins") >> >> -Andrew >>> >>> On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo <[hidden email]> wrote: >>>> On Wed, Aug 1, 2012 at 11:22 AM, Slide <[hidden email]> wrote: >>>>> This is a huge issue with the email-ext plugin as well when it does >>>>> email address resolution. Quite a number of people have complained >>>>> about how long it takes. I have yet to come up with a good solution. >>>>> The perforce plugin has a similar issue. >>>> >>>> If I just stick a dummy address in every user's profile, will that work? >>>> >>>>> >>>>> slide >>>>> >>>>> On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek <[hidden email]> wrote: >>>>>> Looks like you it does search for user's email: >>>>>> hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor >>>>>> and spends time parsing changelogs: >>>>>> hudson.scm.SubversionChangeLogParser.parse >>>>>> >>>>>> I guess you have quite large instance, otherwise this operation would be quite >>>>>> fast. >>>>>> If you have some job, which has set up option to send an email to devs who >>>>>> broke the build, if the user hasn't specified an email, Jenkins tries to find it >>>>>> e.g. in git or SVN changelogs and search all projects and builds so if you >>>>>> have large instance with several dozen thousands of builds if can take pretty >>>>>> long time. >>>>>> >>>>>> You can fix it by setting up correct email for the user. >>>>>> If you have installed git plugin, make sure you have 1.1.16 (I hope it was >>>>>> fixed in this version) or higher. Git plugin made this search even if the user >>>>>> has set up email correctly >>>>>> >>>>>> On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >>>>>>> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek <[hidden email]> >>>>>> wrote: >>>>>>> > On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >>>>>>> >> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> >>>>>> wrote: >>>>>>> >> > quick way how to look what the thread consuming CPU is doing is to do >>>>>>> >> > thread dump (e.g. using jstack $PID) and use top with threads on (H >>>>>>> >> > option) and then look up, see e.g. >>>>>>> >> > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >>>>>>> >> > consuming-most-cpu/ >>>>>>> >> >>>>>>> >> I see. I apparently don't have jstack on this machine :/. Does it only >>>>>>> >> come with the JDK, or can I find it somewhere on the JRE? Once I find >>>>>>> >> the offending thread, should it be pretty obvious what it does? >>>>>>> > >>>>>>> > jstack is part of JDK >>>>>>> > >>>>>>> > you can see the stack trace via Jenkins UI navigating to >>>>>>> > $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version >>>>>>> > provides thread IDs. >>>>>>> > >>>>>>> > >>>>>>> > Once you identify the offending thread, it should be obvious what it does >>>>>>> > (but it may not be obvious why it does what it does:-) >>>>>>> >>>>>>> Okay, I installed the jdk, and I looked some more. >>>>>>> >>>>>>> Using top, I see one jenkins thread taking the lionsshare of the time: >>>>>>> >>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>>>> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >>>>>>> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >>>>>>> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >>>>>>> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >>>>>>> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >>>>>>> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >>>>>>> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >>>>>>> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >>>>>>> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >>>>>>> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >>>>>>> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >>>>>>> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >>>>>>> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >>>>>>> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >>>>>>> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >>>>>>> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >>>>>>> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >>>>>>> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >>>>>>> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >>>>>>> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >>>>>>> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >>>>>>> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >>>>>>> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >>>>>>> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >>>>>>> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >>>>>>> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >>>>>>> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >>>>>>> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >>>>>>> >>>>>>> >>>>>>> Then if I run jstack, I get the following backtrace: >>>>>>> >>>>>>> https://gist.github.com/3228105 >>>>>>> >>>>>>> Does that look useful at all? >>>>>>> >>>>>>> Thanks, >>>>>>> Andrew >>>>> >>>>> >>>>> >>>>> -- >>>>> Website: http://earl-of-code.com >>>> >>>> >>>> >>>> -- >>>> -- >>>> Andrew Melo >>> >>> >>> >>> -- >>> Website: http://earl-of-code.com >> >> >> >> -- >> -- >> Andrew Melo > > > > -- > Website: http://earl-of-code.com -- -- Andrew Melo |
Re: Re: Re: Re: Server usage pegged at 99%
|
|
Oh, wow, I didn't notice, but jenkins has autopopulated a user for
everyone that ever committed on the project. There's like 30 people, ~4000 commits, so I could see why that would take a while :) On Wed, Aug 1, 2012 at 11:42 AM, Andrew Melo <[hidden email]> wrote: > On Wed, Aug 1, 2012 at 11:37 AM, Slide <[hidden email]> wrote: >> Can you gist your global config.xml and something from one of your >> jobs as well? Please remember to sanitize it. > > We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ > > And the following is the gist for the job we run each commit (didn't > make it in for some reason...) > > https://gist.github.com/3228599 > >> >> On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo <[hidden email]> wrote: >>> On Wed, Aug 1, 2012 at 11:26 AM, Slide <[hidden email]> wrote: >>>> No, because its only looking for the email address because it wants to >>>> send an email to that user. >>> >>> I don't know who's getting emailed. I don't remember setting it up for >>> anything, and we actually wrote some scripts that turn jenkins >>> success/failures into Github issues, so having jenkins also send >>> emails would be redundant. >>> >>> I don't supposed there's a global flag to disable email? (I don't see >>> one at "manage jenkins") >>> >>> -Andrew >>>> >>>> On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo <[hidden email]> wrote: >>>>> On Wed, Aug 1, 2012 at 11:22 AM, Slide <[hidden email]> wrote: >>>>>> This is a huge issue with the email-ext plugin as well when it does >>>>>> email address resolution. Quite a number of people have complained >>>>>> about how long it takes. I have yet to come up with a good solution. >>>>>> The perforce plugin has a similar issue. >>>>> >>>>> If I just stick a dummy address in every user's profile, will that work? >>>>> >>>>>> >>>>>> slide >>>>>> >>>>>> On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek <[hidden email]> wrote: >>>>>>> Looks like you it does search for user's email: >>>>>>> hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor >>>>>>> and spends time parsing changelogs: >>>>>>> hudson.scm.SubversionChangeLogParser.parse >>>>>>> >>>>>>> I guess you have quite large instance, otherwise this operation would be quite >>>>>>> fast. >>>>>>> If you have some job, which has set up option to send an email to devs who >>>>>>> broke the build, if the user hasn't specified an email, Jenkins tries to find it >>>>>>> e.g. in git or SVN changelogs and search all projects and builds so if you >>>>>>> have large instance with several dozen thousands of builds if can take pretty >>>>>>> long time. >>>>>>> >>>>>>> You can fix it by setting up correct email for the user. >>>>>>> If you have installed git plugin, make sure you have 1.1.16 (I hope it was >>>>>>> fixed in this version) or higher. Git plugin made this search even if the user >>>>>>> has set up email correctly >>>>>>> >>>>>>> On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >>>>>>>> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek <[hidden email]> >>>>>>> wrote: >>>>>>>> > On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >>>>>>>> >> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek <[hidden email]> >>>>>>> wrote: >>>>>>>> >> > quick way how to look what the thread consuming CPU is doing is to do >>>>>>>> >> > thread dump (e.g. using jstack $PID) and use top with threads on (H >>>>>>>> >> > option) and then look up, see e.g. >>>>>>>> >> > http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >>>>>>>> >> > consuming-most-cpu/ >>>>>>>> >> >>>>>>>> >> I see. I apparently don't have jstack on this machine :/. Does it only >>>>>>>> >> come with the JDK, or can I find it somewhere on the JRE? Once I find >>>>>>>> >> the offending thread, should it be pretty obvious what it does? >>>>>>>> > >>>>>>>> > jstack is part of JDK >>>>>>>> > >>>>>>>> > you can see the stack trace via Jenkins UI navigating to >>>>>>>> > $JENKINS_URL/threadDump but not sure if your (or any) Jenkins version >>>>>>>> > provides thread IDs. >>>>>>>> > >>>>>>>> > >>>>>>>> > Once you identify the offending thread, it should be obvious what it does >>>>>>>> > (but it may not be obvious why it does what it does:-) >>>>>>>> >>>>>>>> Okay, I installed the jdk, and I looked some more. >>>>>>>> >>>>>>>> Using top, I see one jenkins thread taking the lionsshare of the time: >>>>>>>> >>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>>>>> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >>>>>>>> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >>>>>>>> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >>>>>>>> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >>>>>>>> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >>>>>>>> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >>>>>>>> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >>>>>>>> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >>>>>>>> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >>>>>>>> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >>>>>>>> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >>>>>>>> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >>>>>>>> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >>>>>>>> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >>>>>>>> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >>>>>>>> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >>>>>>>> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >>>>>>>> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >>>>>>>> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >>>>>>>> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >>>>>>>> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >>>>>>>> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >>>>>>>> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >>>>>>>> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >>>>>>>> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >>>>>>>> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >>>>>>>> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >>>>>>>> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >>>>>>>> >>>>>>>> >>>>>>>> Then if I run jstack, I get the following backtrace: >>>>>>>> >>>>>>>> https://gist.github.com/3228105 >>>>>>>> >>>>>>>> Does that look useful at all? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Andrew >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Website: http://earl-of-code.com >>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> Andrew Melo >>>> >>>> >>>> >>>> -- >>>> Website: http://earl-of-code.com >>> >>> >>> >>> -- >>> -- >>> Andrew Melo >> >> >> >> -- >> Website: http://earl-of-code.com > > > > -- > -- > Andrew Melo -- -- Andrew Melo |
«
Return to Jenkins users
|
1 view|%1 views
Loading...
| Powered by Nabble | Edit this page |