[JIRA] (JENKINS-13253) Slave connection reset issues since 1.456

Cees Bos created JENKINS-13253:
----------------------------------

             Summary: Slave connection reset issues since 1.456
                 Key: JENKINS-13253
                 URL: https://issues.jenkins-ci.org/browse/JENKINS-13253
             Project: Jenkins
          Issue Type: Bug
          Components: core
         Environment: 1.456
            Reporter: Cees Bos
            Priority: Blocker


We upgrade from 1.451 to 1.456 and now we face several issues with connection reset.
Due to that issue, the jobs are not successful.
On a particular machine we see this very often.

I have captured some stacktraces from the jobs:

{code}
FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
        at hudson.remoting.Request.call(Request.java:149)
        at hudson.remoting.Channel.call(Channel.java:681)
        at hudson.Launcher$RemoteLauncher.kill(Launcher.java:821)
        at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:496)
        at hudson.model.Run.run(Run.java:1410)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:238)
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
        at hudson.remoting.Request.abort(Request.java:273)
        at hudson.remoting.Channel.terminate(Channel.java:732)
        at hudson.remoting.Channel$ReaderThread.run(Channel.java:1157)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:185)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
        at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
        at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
        at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
        at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
{code}

{code}
Archiving artifacts
ERROR: Failed to archive artifacts: allLogs/**/*.*
hudson.util.IOException2: java.io.IOException
        at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:175)
        at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61)
        at java.io.FilterInputStream.read(FilterInputStream.java:107)
        at hudson.util.HeadBufferingStream.fillSide(HeadBufferingStream.java:83)
        at hudson.FilePath$TarCompression$2.extract(FilePath.java:612)
        at hudson.FilePath.copyRecursiveTo(FilePath.java:1729)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
        at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
        at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:678)
        at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:656)
        at hudson.model.Build$RunnerImpl.post2(Build.java:162)
        at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:625)
        at hudson.model.Run.run(Run.java:1435)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:238)

        at hudson.FilePath.copyRecursiveTo(FilePath.java:1736)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
        at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
        at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:678)
        at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:656)
        at hudson.model.Build$RunnerImpl.post2(Build.java:162)
        at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:625)
        at hudson.model.Run.run(Run.java:1435)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:238)
Caused by: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
        at hudson.remoting.Request$1.get(Request.java:252)
        at hudson.remoting.Request$1.get(Request.java:184)
        at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
        at hudson.FilePath.copyRecursiveTo(FilePath.java:1732)
        ... 11 more
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
        at hudson.remoting.Request.abort(Request.java:273)
        at hudson.remoting.Channel.terminate(Channel.java:732)
        at hudson.remoting.Channel$ReaderThread.run(Channel.java:1157)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:185)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
        at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
        at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
        at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
        at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
{code}

{code}
FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
        at hudson.remoting.Request.call(Request.java:149)
        at hudson.remoting.Channel.call(Channel.java:681)
        at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
        at $Proxy45.join(Unknown Source)
        at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:859)
        at hudson.Launcher$ProcStarter.join(Launcher.java:345)
        at hudson.plugins.groovy.Groovy.perform(Groovy.java:116)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
        at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
        at hudson.model.Build$RunnerImpl.build(Build.java:178)
        at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
        at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:473)
        at hudson.model.Run.run(Run.java:1410)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:238)
Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
        at hudson.remoting.Request.abort(Request.java:273)
        at hudson.remoting.Channel.terminate(Channel.java:732)
        at hudson.remoting.Channel$ReaderThread.run(Channel.java:1157)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:185)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
        at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
        at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
        at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
        at hudson.remoting.Channel$ReaderThread.run(Channel.java:1127)
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Andrey Myatlyuk commented on JENKINS-13253

Slave connection reset issues since 1.456

Hi Cees, Please provide more details Can you confirm that you see it more often on some slaves? Do you have slaves without this problem? Are all slaves configured the same way? Can it be network congestion? Can you monitor packets between master and slave? When it is happening, do you have connectivity between master and slave? What do you see in the log on the slave? Andrey

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Andrey Myatlyuk assigned JENKINS-13253 to Andrey Myatlyuk

Slave connection reset issues since 1.456

Change By:	Andrey Myatlyuk (19/Jun/12 6:03 PM)
Assignee:	Andrey Myatlyuk

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Cees Bos commented on JENKINS-13253

Slave connection reset issues since 1.456

The number of jobs failing with this is less now. We use 1.470 at this moment. A specific job fails now and then with this error. The slave is a windows slave connected via JNLP.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Andrey Myatlyuk commented on JENKINS-13253

Slave connection reset issues since 1.456

Cees, The information that you have provided is not enough to analyze or attempt to replicate the issue. Please let me know if you would like to work on it further, by giving us more information or we can close the issue for now and in case of you gathering more data or if similar problem arises, we could always reopen the ticket. Andrey

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Cees Bos commented on JENKINS-13253

Slave connection reset issues since 1.456

Last Friday we faced the same issue with a linux machine (connected via ssh). As far as I know there are no issues in connectivity. All servers we have are Virtual Machines. Where can I find more information / more logging?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Andrey Myatlyuk commented on JENKINS-13253

Slave connection reset issues since 1.456

Cees, Here's the wiki page on how to enable a finer level of logging in Jenkins: https://wiki.jenkins-ci.org/display/JENKINS/Logging

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Cees Bos commented on JENKINS-13253

Slave connection reset issues since 1.456

We have 90+ slaves. What is the impact of enabling a finer logging level on performance of Jenkins? Now and then we face this issues, so the change of logging level is required for several days I guess to get some information.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

naveen karnati commented on JENKINS-13253

Slave connection reset issues since 1.456

We have similar issues over here. The socket gets disconnected intermittently. hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset Jenkins ver. 1.410

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Ben Ernst commented on JENKINS-13253

Slave connection reset issues since 1.456

Similar issues here. Server is linux VM, agents are windows VMs.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

wei he commented on JENKINS-13253

Slave connection reset issues since 1.456

We have similar issues here.Jenkins ver1.466

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

hagzag commented on JENKINS-13253

Slave connection reset issues since 1.456

I have the exact same issue on 1.506 windows 2003 slaves via windows service builds start running and I get: 10:31:02 FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset 10:31:02 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Nicholas Lun commented on JENKINS-13253

Slave connection reset issues since 1.456

Is there any update to this issue? All I can tell from the stacktrace is that at the end of the build, Jenkins requests that the slave kill the remaining processes. But, before it can do so it receives a response that the Connection was terminated due to some failure during a read operation(can't see the reason behind the failure in the stacktrace). Doesn't this seem like some kind of race condition where the slave is trying to finish up and kill the remaining processes while the master terminates the connection before the slave can finish?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.