Quantcast
Jenkins › Jenkins issues

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

classic Classic list List threaded Threaded
♦
♦
 10 messages Options
JIRA noreply@jenkins-ci.org
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

JIRA noreply@jenkins-ci.org
Issue Type: Bug Bug
Affects Versions: current
Assignee: Unassigned
Components: perforce
Created: 14/Aug/12 9:44 AM
Description:

We have noticed that polling, either unbounded or low-bounded completely cripples the application server. The server stops responding for very long periods of time, until the polling has completed. The reason we realised that polling is causing this, was when the plugin regressed and was causing polling to fail with an NPE, the server was lightning fast. What's interesting is that it seems to cripple the entire Tomcat instance, not just Jenkins. Upon capturing a thread dump we recorded the following 

"SCM polling for hudson.maven.MavenModuleSet@2736af9a[IStarReceiver-12.8]" - Thread t@706
   java.lang.Thread.State: BLOCKED
                at java.lang.ThreadGroup.add(ThreadGroup.java:856)
                - waiting to lock <69115611> (a java.lang.ThreadGroup) owned by "SCM polling for hudson.maven.MavenModuleSet@15405f35[IStarSender-12.6]" t@703
                at java.lang.Thread.start(Thread.java:639)
                - locked <64e981d4> (a java.lang.UNIXProcess$2$1)
                at java.lang.UNIXProcess$2.run(UNIXProcess.java:95)
                at java.security.AccessController.doPrivileged(Native Method)
                at java.lang.UNIXProcess.<init>(UNIXProcess.java:81)
                at java.lang.ProcessImpl.start(ProcessImpl.java:65)
                at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
                at hudson.Proc$LocalProc.<init>(Proc.java:244)
                at hudson.Proc$LocalProc.<init>(Proc.java:216)
                at hudson.Launcher$LocalLauncher.launch(Launcher.java:709)
                at hudson.Launcher$ProcStarter.start(Launcher.java:338)
                at hudson.plugins.perforce.HudsonP4DefaultExecutor.exec(HudsonP4DefaultExecutor.java:79)
                at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:321)
                at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:292)
                at com.tek42.perforce.parse.Changes.getChangelist(Changes.java:71)
                at hudson.plugins.perforce.PerforceSCM.getCurrentDepotRevisionState(PerforceSCM.java:1275)
                at hudson.plugins.perforce.PerforceSCM.compareRemoteRevisionWith(PerforceSCM.java:1155)
                at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356)
                at hudson.scm.SCM.poll(SCM.java:373)
                at hudson.model.AbstractProject._poll(AbstractProject.java:1415)
                at hudson.model.AbstractProject.poll(AbstractProject.java:1335)
                at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420)
                at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449)
                at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
                - locked <5fe91263> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"SCM polling for hudson.model.FreeStyleProject@30d4c8ad[JmsQueueRecovery]" - Thread t@705
   java.lang.Thread.State: RUNNABLE
                at java.lang.UNIXProcess.forkAndExec(Native Method)
                at java.lang.UNIXProcess.<init>(UNIXProcess.java:53)
                at java.lang.ProcessImpl.start(ProcessImpl.java:65)
                at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
                at hudson.Proc$LocalProc.<init>(Proc.java:244)
                at hudson.Proc$LocalProc.<init>(Proc.java:216)
                at hudson.Launcher$LocalLauncher.launch(Launcher.java:709)
                at hudson.Launcher$ProcStarter.start(Launcher.java:338)
                at hudson.plugins.perforce.HudsonP4DefaultExecutor.exec(HudsonP4DefaultExecutor.java:79)
                at com.tek42.perforce.parse.AbstractPerforceTemplate.getRawPerforceResponseLines(AbstractPerforceTemplate.java:425)
                at com.tek42.perforce.parse.Changes.getChangeNumbersInRangeForSinglePath(Changes.java:528)
                at com.tek42.perforce.parse.Changes.getChangeNumbersInRange(Changes.java:467)
                at hudson.plugins.perforce.PerforceSCM.getCurrentDepotRevisionState(PerforceSCM.java:1253)
                at hudson.plugins.perforce.PerforceSCM.compareRemoteRevisionWith(PerforceSCM.java:1155)
                at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356)
                at hudson.scm.SCM.poll(SCM.java:373)
                at hudson.model.AbstractProject._poll(AbstractProject.java:1415)
                at hudson.model.AbstractProject.poll(AbstractProject.java:1335)
                at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420)
                at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449)
                at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
                - locked <3eb652c6> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
Environment: SunOS 5.10 Generic_142901-03 i86pc i386 i86pc, Tomcat 7
Project: Jenkins
Priority: Critical Critical
Reporter: Ioannis Mavroukakis
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
JIRA noreply@jenkins-ci.org
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

JIRA noreply@jenkins-ci.org
Change By: Ioannis Mavroukakis (14/Aug/12 9:45 AM)
Component/s: core
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
JIRA noreply@jenkins-ci.org
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

JIRA noreply@jenkins-ci.org
In reply to this post by JIRA noreply@jenkins-ci.org
Change By: Rob Petti (14/Aug/12 2:35 PM)
Assignee: Rob Petti
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
JIRA noreply@jenkins-ci.org
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

JIRA noreply@jenkins-ci.org
In reply to this post by JIRA noreply@jenkins-ci.org
Rob Petti commented on Bug JENKINS-14782

It's just waiting for perforce to finish it's operation. I'm not sure how this would cripple your server. How many polling threads do you have?

You may want to check out the p4 processes running on the machine using truss or something similar to see what it's up to.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
JIRA noreply@jenkins-ci.org
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

JIRA noreply@jenkins-ci.org
In reply to this post by JIRA noreply@jenkins-ci.org

We've had anything from 2 to unbounded and the behaviour is largely consistent. With unbounded it sometimes recovers quicker.
Normal checkout as part of a build is not an issue, it's just polling that brings everything to standstill. I should clarify that it's Tomcat that gets crippled, to the point of other applications deployed on it becoming unresponsive too.
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
JIRA noreply@jenkins-ci.org
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

JIRA noreply@jenkins-ci.org
In reply to this post by JIRA noreply@jenkins-ci.org
Rob Petti commented on Bug JENKINS-14782

I having a tough time understanding what the actual issue is. Does Tomcat start using 100% cpu, or does it just sit there doing nothing? If it's the latter, then I can't see how polling would be affect the entire application server like that.

Have you checked the p4 processes to make sure they aren't hanging?
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
JIRA noreply@jenkins-ci.org
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

JIRA noreply@jenkins-ci.org
In reply to this post by JIRA noreply@jenkins-ci.org
 
Rob Petti edited a comment on Bug JENKINS-14782

I having a tough time understanding what the actual issue is. Does Tomcat start using 100% cpu, or does it just sit there doing nothing? If it's the latter, then I can't see how polling would be affect the entire application server like that.

Have you checked the p4 processes to make sure they aren't hanging? What about IO?
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
JIRA noreply@jenkins-ci.org
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

JIRA noreply@jenkins-ci.org
In reply to this post by JIRA noreply@jenkins-ci.org

It's the latter. We are also suspecting the current p4 command line client also, and we will be upgrading it. There have been no I/O issues that we could see. What confuses is me is how a forked process could be making Tomcat grind to a halt and leave the rest of the box unaffected, unless it's a deeper systemic issue, but where that the case, we would have had far worse problems than this. I will let you know what happens as soon as we upgrade the p4 binary and re-test
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
JIRA noreply@jenkins-ci.org
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

JIRA noreply@jenkins-ci.org
In reply to this post by JIRA noreply@jenkins-ci.org
 
Ioannis Mavroukakis edited a comment on Bug JENKINS-14782

It's the latter. We are also suspecting the current p4 command line client, and we will be upgrading it. There have been no I/O issues that we could see. What confuses is me is how a forked process could be making Tomcat grind to a halt and leave the rest of the box unaffected, unless it's a deeper systemic issue, but where that the case, we would have had far worse problems than this. I will let you know what happens as soon as we upgrade the p4 binary and re-test
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
JIRA noreply@jenkins-ci.org
Reply | Threaded
Open this post in threaded view
♦
♦
|  
Report Content as Inappropriate
♦
♦
star
♦
♦

[JIRA] (JENKINS-14782) SCM polling cripples the entire application server

JIRA noreply@jenkins-ci.org
In reply to this post by JIRA noreply@jenkins-ci.org
Rob Petti resolved Bug JENKINS-14782 as Cannot Reproduce

Haven't heard from reporter in a while, so I'm going to assume that this has been resolved on their end.
Change By: Rob Petti (29/Nov/12 4:20 PM)
Status: Open Resolved
Resolution: Cannot Reproduce
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Loading...
Powered by Nabble Edit this page