Setting up slave and running a Maven build with Git takes an unbearably long time

[Collin Peters]

Just setup a new Jenkins slave running on Debian 6/Squeeze.  Did the simple SSH method and everything went quite smoothly.  Now I have configured one of my builds to use this new slave with a very odd effect.  That is that there are long delays periodically in the build.

Here is some sample output:

Building remotely on dcc-jenkins-slave in workspace /home/intouch/jenkins_work/workspace/intouch-qa
Checkout:intouch-qa / /home/intouch/jenkins_work/workspace/intouch-qa - hudson.remoting.Channel@41cc3f28:dcc-jenkins-slave
Using strategy: Default
Last Built Revision: Revision 29258d9508c3cf5a7b8a6e867d1e18ef88e9408d (origin/nexus)
<<< 10 MINUTE DELAY HERE >>>
Fetching changes from 1 remote Git repository
Fetching upstream changes from ssh://[hidden email]/foo/bar
Commencing build of Revision 29258d9508c3cf5a7b8a6e867d1e18ef88e9408d (origin/nexus)
Checking out Revision 29258d9508c3cf5a7b8a6e867d1e18ef88e9408d (origin/nexus)
Starting xvnc
[intouch-qa] $ vncserver :11 -geometry 800x600
New 'X' desktop is jenkins-slave:11

Starting applications specified in /home/intouch/.vnc/xstartup
Log file is /home/intouch/.vnc/jenkins-slave:11.log

Parsing POMs
<<< 10 MINUTE DELAY HERE >>>
etc...


Does anybody have any ideas what could be going on here? I don't see any errors anywhere. The slave Debian install is running in a virtual machine (OpenVZ) if that helps.


Here is the log of the slave launch:


[05/03/12 08:32:56] [SSH] Opening SSH connection to 192.168.80.150:22.
[05/03/12 08:32:58] [SSH] Authenticating as intouch with /home/intouch/.ssh/id_rsa.
[05/03/12 08:32:58] [SSH] Authentication successful.
[05/03/12 08:32:58] [SSH] The remote users environment is:
BASH=/bin/bash
BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="1" [2]="5" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
BASH_VERSION='4.1.5(1)-release'
DIRSTACK=()
EUID=1000
GROUPS=()
HOME=/home/intouch
HOSTNAME=jenkins-slave
HOSTTYPE=x86_64
IFS=$' \t\n'
LANG=en_CA.UTF-8
LOGNAME=intouch
MACHTYPE=x86_64-pc-linux-gnu
MAIL=/var/mail/intouch
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
PIPESTATUS=([0]="0")
PPID=910
PS4='+ '
PWD=/home/intouch
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
SSH_CLIENT='192.168.10.150 44221 22'
SSH_CONNECTION='192.168.10.150 44221 192.168.80.150 22'
TERM=dumb
UID=1000
USER=intouch
_=']'
[05/03/12 08:32:59] [SSH] Checking java version of java
Couldn't figure out the Java version of java
bash: java: command not found

[05/03/12 08:32:59] [SSH] Checking java version of /usr/bin/java
Couldn't figure out the Java version of /usr/bin/java
bash: /usr/bin/java: No such file or directory

[05/03/12 08:33:00] [SSH] Checking java version of /usr/java/default/bin/java
Couldn't figure out the Java version of /usr/java/default/bin/java
bash: /usr/java/default/bin/java: No such file or directory

[05/03/12 08:33:00] [SSH] Checking java version of /usr/java/latest/bin/java
Couldn't figure out the Java version of /usr/java/latest/bin/java
bash: /usr/java/latest/bin/java: No such file or directory

[05/03/12 08:33:01] [SSH] Checking java version of /usr/local/bin/java
Couldn't figure out the Java version of /usr/local/bin/java
bash: /usr/local/bin/java: No such file or directory

[05/03/12 08:33:01] [SSH] Checking java version of /usr/local/java/bin/java
Couldn't figure out the Java version of /usr/local/java/bin/java
bash: /usr/local/java/bin/java: No such file or directory

[05/03/12 08:33:02] [SSH] Checking java version of /home/intouch/jenkins_work/jdk/bin/java
Couldn't figure out the Java version of /home/intouch/jenkins_work/jdk/bin/java
bash: /home/intouch/jenkins_work/jdk/bin/java: No such file or directory

[05/03/12 08:33:02] [SSH] Checking java version of /opt/jdk/bin/java
[05/03/12 08:33:03] [SSH] /opt/jdk/bin/java -version returned 1.7.0_03.
[05/03/12 08:33:03] [SSH] Starting sftp client.
[05/03/12 08:33:04] [SSH] Copying latest slave.jar...
[05/03/12 08:33:13] [SSH] Copied 266,750 bytes.
[05/03/12 08:33:14] [SSH] Starting slave process: cd '/home/intouch/jenkins_work' && /opt/jdk/bin/java  -jar slave.jar
<===[JENKINS REMOTING CAPACITY]===>channel started
Slave.jar version: 2.13
This is a Unix slave
Copied maven-agent.jar
Copied maven3-agent.jar
Copied maven3-interceptor.jar
Copied maven-interceptor.jar
Copied maven2.1-interceptor.jar
Copied plexus-classworld.jar
Copied classworlds.jar
Evacuated stdout
Slave successfully connected and online

[Collin Peters]

The following groovy runs nice and quick on the master, but on my slave it takes 6-7 minutes!!

def proc = "ssh -T [hidden email]".execute()

def b = new StringBuffer()

proc.consumeProcessErrorStream(b)

println proc.text

println b.toString()

Any help would be much appreciated!!

[Collin Peters]

My build that takes around 8 minutes locally took 3 hours and 56 minutes on the slave!!  Does anyone have any useful advice that would help me diagnose this?  

My initial thought that it was an SSH issue involving reverse lookups.  However, my operations guys say that reverse lookup is working fine.  That being said, it seems that it isn't just SSH actions that are taking a long time, as there are massive delays throughout the entire build.

/Collin

[Mark Waite]

You might consider configuring various small test jobs, each using some subset of the capabilities you need in your build.  For example, use the Git plugin, but clone a small repository from github.com with the git protocol rather than cloning your own repository.  If that is fast, then clone a small repository from github.com with ssh.  If your job is a maven job, then try a sample maven project from outside your organization.

Invoke a trivial Hello World job with your preferred build technique to see if the build environment is the problem.

Etc.

Mark Waite

---

From: Collin Peters <[hidden email]>
To: [hidden email]
Sent: Wednesday, May 9, 2012 3:35 PM
Subject: Re: Setting up slave and running a Maven build with Git takes an unbearably long time

My build that takes around 8 minutes locally took 3 hours and 56 minutes on the slave!!  Does anyone have any useful advice that would help me diagnose this?  

My initial thought that it was an SSH issue involving reverse lookups.  However, my operations guys say that reverse lookup is working fine.  That being said, it seems that it isn't just SSH actions that are taking a long time, as there are massive delays throughout the entire build.

/Collin

[Collin Peters]

It is a Maven job.  I built the Twilio (https://github.com/twilio/twilio-java) library using all different ways.  Results is 25 seconds local (both git & ssh) and 5 min 30 seconds on slave (both git & ssh)

The only odd thing that I have noticed so far is that the slave is using twice as much memory as the master when both are idle. Master is 720 GB, slave is 1.5 GB.  Something seems off there.  During the build the master goes to 1.91 GB and the slave to 2.74.  Something seems wrong with that.  Is 1.5GB on idle usage seem normal?  If I disconnect the slave the memory goes to 90MB.  Both the master and slave are Debian 6.0 (squeeze) installs running in virtual machines.  The other thing of note is that they are in physically different locations running across a VPN. Ping time is 400ms.

/Collin

On Wed, May 9, 2012 at 9:02 PM, Mark Waite <[hidden email]> wrote:

You might consider configuring various small test jobs, each using some subset of the capabilities you need in your build.  For example, use the Git plugin, but clone a small repository from github.com with the git protocol rather than cloning your own repository.  If that is fast, then clone a small repository from github.com with ssh.  If your job is a maven job, then try a sample maven project from outside your organization.

Invoke a trivial Hello World job with your preferred build technique to see if the build environment is the problem.

Etc.

Mark Waite

---

From: Collin Peters <[hidden email]>
To: [hidden email]
Sent: Wednesday, May 9, 2012 3:35 PM
Subject: Re: Setting up slave and running a Maven build with Git takes an unbearably long time

My build that takes around 8 minutes locally took 3 hours and 56 minutes on the slave!!  Does anyone have any useful advice that would help me diagnose this?  

My initial thought that it was an SSH issue involving reverse lookups.  However, my operations guys say that reverse lookup is working fine.  That being said, it seems that it isn't just SSH actions that are taking a long time, as there are massive delays throughout the entire build.

/Collin

[Mark Waite]

That seems to hint that the difference between the machines is outside the specific job definition.  I'm not sure what hints to offer to seek for other differences between the machines, though you might check for things like:

 

- Are they the same architecture (both x64, both x86, etc.)?

- Are they running the same Java virtual machine and the same version of JVM?

 

Sorry I don't have other insights to offer...

Mark Waite

From: Collin Peters <[hidden email]>
To: [hidden email]
Sent: Wednesday, May 9, 2012 11:26 PM
Subject: Re: Setting up slave and running a Maven build with Git takes an unbearably long time

It is a Maven job.  I built the Twilio (https://github.com/twilio/twilio-java) library using all different ways.  Results is 25 seconds local (both git & ssh) and 5 min 30 seconds on slave (both git & ssh)

The only odd thing that I have noticed so far is that the slave is using twice as much memory as the master when both are idle. Master is 720 GB, slave is 1.5 GB.  Something seems off there.  During the build the master goes to 1.91 GB and the slave to 2.74.  Something seems wrong with that.  Is 1.5GB on idle usage seem normal?  If I disconnect the slave the memory goes to 90MB.  Both the master and slave are Debian 6.0 (squeeze) installs running in virtual machines.  The other thing of note is that they are in physically different locations running across a VPN. Ping time is 400ms.

/Collin

On Wed, May 9, 2012 at 9:02 PM, Mark Waite <[hidden email]> wrote:

You might consider configuring various small test jobs, each using some subset of the capabilities you need in your build.  For example, use the Git plugin, but clone a small repository from github.com with the git protocol rather than cloning your own repository.  If that is fast, then clone a small repository from github.com with ssh.  If your job is a maven job, then try a sample maven project from outside your organization.

Invoke a trivial Hello World job with your preferred build technique to see if the build environment is the problem.

Etc.

Mark Waite

From: Collin Peters <[hidden email]>
To: [hidden email]
Sent: Wednesday, May 9, 2012 3:35 PM
Subject: Re: Setting up slave and running a Maven build with Git takes an unbearably long time

My build that takes around 8 minutes locally took 3 hours and 56 minutes on the slave!!  Does anyone have any useful advice that would help me diagnose this?  

My initial thought that it was an SSH issue involving reverse lookups.  However, my operations guys say that reverse lookup is working fine.  That being said, it seems that it isn't just SSH actions that are taking a long time, as there are massive delays throughout the entire build.

/Collin

[Sami Tikka]

What OS is your master running? 32 or 64 bit? How much memory do the master and slave have? Are they doing a lot of other things than running Jenkins? What JVM are you running? 32 or 64 bit? What are those memory figures you quoted? Are they from top? From jconsole? Is that RSS usage or virtual?

What are the JVM command line options you use, both for master and slave?

Is the slave swapping during the build?

I suggest you run top/htop, vmstat, and jconsole attached to both master and slave and try to gather information about what is going on. 

-- Sami

Collin Peters <[hidden email]> kirjoitti 10.5.2012 kello 8.26:

It is a Maven job.  I built the Twilio (https://github.com/twilio/twilio-java) library using all different ways.  Results is 25 seconds local (both git & ssh) and 5 min 30 seconds on slave (both git & ssh)

The only odd thing that I have noticed so far is that the slave is using twice as much memory as the master when both are idle. Master is 720 GB, slave is 1.5 GB.  Something seems off there.  During the build the master goes to 1.91 GB and the slave to 2.74.  Something seems wrong with that.  Is 1.5GB on idle usage seem normal?  If I disconnect the slave the memory goes to 90MB.  Both the master and slave are Debian 6.0 (squeeze) installs running in virtual machines.  The other thing of note is that they are in physically different locations running across a VPN. Ping time is 400ms.

/Collin

On Wed, May 9, 2012 at 9:02 PM, Mark Waite <[hidden email]> wrote:

You might consider configuring various small test jobs, each using some subset of the capabilities you need in your build.  For example, use the Git plugin, but clone a small repository from github.com with the git protocol rather than cloning your own repository.  If that is fast, then clone a small repository from github.com with ssh.  If your job is a maven job, then try a sample maven project from outside your organization.

Invoke a trivial Hello World job with your preferred build technique to see if the build environment is the problem.

Etc.

Mark Waite

---

From: Collin Peters <[hidden email]>
To: [hidden email]
Sent: Wednesday, May 9, 2012 3:35 PM
Subject: Re: Setting up slave and running a Maven build with Git takes an unbearably long time

My build that takes around 8 minutes locally took 3 hours and 56 minutes on the slave!!  Does anyone have any useful advice that would help me diagnose this?  

My initial thought that it was an SSH issue involving reverse lookups.  However, my operations guys say that reverse lookup is working fine.  That being said, it seems that it isn't just SSH actions that are taking a long time, as there are massive delays throughout the entire build.

/Collin

[Collin Peters]

Thanks for the help guys.  Here is some info

• Debian 6.0 64 bit
• They are VMs dedicated to Jenkins and the host machines are not busy during the builds
• Also using 64 Bit Java 7 JVM
• Memory figures are as reported by the host, for the individual VMs

I will try to run some more advanced stats during a build.  Is there a way to tell what exactly Jenkins is doing any any particular second?  i.e. a break?

/Collin

On Fri, May 11, 2012 at 11:50 PM, Sami Tikka <[hidden email]> wrote:

What OS is your master running? 32 or 64 bit? How much memory do the master and slave have? Are they doing a lot of other things than running Jenkins? What JVM are you running? 32 or 64 bit? What are those memory figures you quoted? Are they from top? From jconsole? Is that RSS usage or virtual?

What are the JVM command line options you use, both for master and slave?

Is the slave swapping during the build?

I suggest you run top/htop, vmstat, and jconsole attached to both master and slave and try to gather information about what is going on. 

-- Sami

Collin Peters <[hidden email]> kirjoitti 10.5.2012 kello 8.26:

It is a Maven job.  I built the Twilio (https://github.com/twilio/twilio-java) library using all different ways.  Results is 25 seconds local (both git & ssh) and 5 min 30 seconds on slave (both git & ssh)

The only odd thing that I have noticed so far is that the slave is using twice as much memory as the master when both are idle. Master is 720 GB, slave is 1.5 GB.  Something seems off there.  During the build the master goes to 1.91 GB and the slave to 2.74.  Something seems wrong with that.  Is 1.5GB on idle usage seem normal?  If I disconnect the slave the memory goes to 90MB.  Both the master and slave are Debian 6.0 (squeeze) installs running in virtual machines.  The other thing of note is that they are in physically different locations running across a VPN. Ping time is 400ms.

/Collin

On Wed, May 9, 2012 at 9:02 PM, Mark Waite <[hidden email]> wrote:

You might consider configuring various small test jobs, each using some subset of the capabilities you need in your build.  For example, use the Git plugin, but clone a small repository from github.com with the git protocol rather than cloning your own repository.  If that is fast, then clone a small repository from github.com with ssh.  If your job is a maven job, then try a sample maven project from outside your organization.

Invoke a trivial Hello World job with your preferred build technique to see if the build environment is the problem.

Etc.

Mark Waite

---

From: Collin Peters <[hidden email]>
To: [hidden email]
Sent: Wednesday, May 9, 2012 3:35 PM
Subject: Re: Setting up slave and running a Maven build with Git takes an unbearably long time

My build that takes around 8 minutes locally took 3 hours and 56 minutes on the slave!!  Does anyone have any useful advice that would help me diagnose this?  

My initial thought that it was an SSH issue involving reverse lookups.  However, my operations guys say that reverse lookup is working fine.  That being said, it seems that it isn't just SSH actions that are taking a long time, as there are massive delays throughout the entire build.

/Collin