Feedback of your CI architecture

[mpapo - Michaël Pailloncy]

Hi all,

Currently, we manage near 250 jobs related to several java projects and several teams in our company.

Our CI architecture characteristics : 

- a Jenkins master on AIX 5.3 with 8 processors and 10 GB of RAM

- 3 RHEL slaves with 2 processors and 4 GB of RAM

- 2 Windows slaves with 1 processors and 2 GB of RAM

We arrive sometimes has limitations with Jenkins. Especially during releases.

Due to an increase of job number in our Jenkins in my company,  we want to change our architecture to meet our needs. We are going to manage near 500 jobs.

We want to switch the master on a RHEL OS (our new main target platform) and add an AIX slave for projects that need this platform.

Here are our thoughts :

- Do you think it would be interesting to have a master only used to monitor the jobs (0 executor, all builds are delegate to slaves) ? 

This could be interesting to have a more stable and more reliable master.

- On the other hand, do you think it could be interesting to use multiple masters ? One for each team for example ?

This could be nice to have a more specialized master per team but I guess it must increase the maintenance time.

I need feedbacks from the community to choose the best architecture :

How many jobs do you manage ?

What are the characteristics of your architecture ? (number of slaves, of master ??)

Would you prefer to use a slave with large capacity, or more with capacities smaller?

Thanks in advance from any help and feeback !

Michaël Pailloncy

 

[mpapo - Michaël Pailloncy]

Other question : What do you use to launch Jenkins ?

Currently,  we are using Winstone. Is this the best way?

2012/9/26 mpapo - Michaël Pailloncy <[hidden email]>

Hi all,

Currently, we manage near 250 jobs related to several java projects and several teams in our company.

Our CI architecture characteristics : 

- a Jenkins master on AIX 5.3 with 8 processors and 10 GB of RAM

- 3 RHEL slaves with 2 processors and 4 GB of RAM

- 2 Windows slaves with 1 processors and 2 GB of RAM

We arrive sometimes has limitations with Jenkins. Especially during releases.

Due to an increase of job number in our Jenkins in my company,  we want to change our architecture to meet our needs. We are going to manage near 500 jobs.

We want to switch the master on a RHEL OS (our new main target platform) and add an AIX slave for projects that need this platform.

Here are our thoughts :

- Do you think it would be interesting to have a master only used to monitor the jobs (0 executor, all builds are delegate to slaves) ? 

This could be interesting to have a more stable and more reliable master.

- On the other hand, do you think it could be interesting to use multiple masters ? One for each team for example ?

This could be nice to have a more specialized master per team but I guess it must increase the maintenance time.

I need feedbacks from the community to choose the best architecture :

How many jobs do you manage ?

What are the characteristics of your architecture ? (number of slaves, of master ??)

Would you prefer to use a slave with large capacity, or more with capacities smaller?

Thanks in advance from any help and feeback !

Michaël Pailloncy

 

[Les Mikesell]

On Wed, Sep 26, 2012 at 7:58 AM, mpapo - Michaël Pailloncy
<[hidden email]> wrote:
>
> - Do you think it would be interesting to have a master only used to monitor
> the jobs (0 executor, all builds are delegate to slaves) ?
>
> This could be interesting to have a more stable and more reliable master.

I don't think it really matters as long as you have the resources in
the right places for the jobs you are running.  However, that is
somewhat easier to manage when all jobs go to a pool of slaves.   You
at least would not want the builds to be happening on the same
physical disk where  the jenkins  master job directory resides.

> - On the other hand, do you think it could be interesting to use multiple
> masters ? One for each team for example ?

I would only do that if you need to closely control access permissions
or have problems with sharing resources.

> This could be nice to have a more specialized master per team but I guess it
> must increase the maintenance time.

There is hardly any maintenance time for a yum-installed jenkins
instance, but it might be harder to keep the pools of slaves matched
to the jobs with several masters.   I'd make this decision more on
management/budget concerns than anything technical.


> I need feedbacks from the community to choose the best architecture :
> How many jobs do you manage ?

200+ currently, and don't expect problems with more.  Frequency of
builds might be a better metric and I don't know where to find a
meaningful metric for that (like average queue wait).

> What are the characteristics of your architecture ? (number of slaves, of
> master ??)

1 master, 17 slaves, some are seldom-used architectures.

> Would you prefer to use a slave with large capacity, or more with capacities
> smaller?

More, but capacity needs to be 'large enough'.  Most of our slaves are
VMs running about 4 to a physical host (a mix of frequently/seldom
used instances...).   I like to have jobs tied to labels, enough
slaves in a labeled group that having one or two down for maintenance
won't be noticed, and nothing of importance ever stored on a slave so
you can just spin up a new VM image as a replacement if you have any
problems.

--
   Les Mikesell
      [hidden email]

[Les Mikesell]

On Wed, Sep 26, 2012 at 8:19 AM, Michaël Pailloncy <[hidden email]> wrote:
> Other question : What do you use to launch Jenkins ?
> Currently,  we are using Winstone. Is this the best way?

Since you mentioned moving to RHEL, you'll probably want to use the
RPM that will include winstone and a working configuration.  I don't
think any other approach would be enough different to break what the
rpm will do automatically.   You can change the port to not conflict
with a stock tomcat and/or proxy through apache if you want it to
appear on port 80, though.

--
  Les Mikesell
    [hidden email]

[Mandeville, Rob]

Having the Jenkins server on a host that doesn’t run builds isn’t just interesting, it’s a good idea.  The only downside that I see is that the slave-master communication (probably dominated by your build logs and artifacts) is slowed down to network speeds rather than the speed of your loopback adapter.  I’m running in this configuration with 170+ slave nodes scattered across over a dozen build hosts (each build takes up a user account, so we need multiple slaves per host).  We basically build 60 different SCM branches of the same project.

 

Multiple masters can be useful.  I have one master for my build jobs, another one for housekeeping (I find it better than cron because it’s more visible and doesn’t spam you with email, and it’s also useful for some of your more routine command lines), and some people on a side project set up their own.  Part of the question is organizational: Jenkins is small enough that somebody working on a small project can set their own up, if they’re willing to spend a little of maintenance time.  That can keep you less involved in smaller projects, if this sort of a thing is acceptable to your organization.

 

--Rob

 

 

 

From: [hidden email] [hidden email] On Behalf Of mpapo - Michaël Pailloncy
Sent: Wednesday, September 26, 2012 8:58 AM
To: [hidden email]
Subject: Feedback of your CI architecture

 

Hi all,

 

Currently, we manage near 250 jobs related to several java projects and several teams in our company.

 

Our CI architecture characteristics : 

- a Jenkins master on AIX 5.3 with 8 processors and 10 GB of RAM

- 3 RHEL slaves with 2 processors and 4 GB of RAM

- 2 Windows slaves with 1 processors and 2 GB of RAM

 

We arrive sometimes has limitations with Jenkins. Especially during releases.

 

Due to an increase of job number in our Jenkins in my company,  we want to change our architecture to meet our needs. We are going to manage near 500 jobs.

We want to switch the master on a RHEL OS (our new main target platform) and add an AIX slave for projects that need this platform.

 

Here are our thoughts :

- Do you think it would be interesting to have a master only used to monitor the jobs (0 executor, all builds are delegate to slaves) ? 

This could be interesting to have a more stable and more reliable master.

 

- On the other hand, do you think it could be interesting to use multiple masters ? One for each team for example ?

This could be nice to have a more specialized master per team but I guess it must increase the maintenance time.

 

 

I need feedbacks from the community to choose the best architecture :

How many jobs do you manage ?

What are the characteristics of your architecture ? (number of slaves, of master ??)

Would you prefer to use a slave with large capacity, or more with capacities smaller?

 

 

Thanks in advance from any help and feeback !

 

Michaël Pailloncy

 

 

The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.

[mpapo - Michaël Pailloncy]

Thank you very much for your feedback ! 

I think we're going to explore the idea of ​​creating one or more physical hosts creating VMs.

Do you use a particular tool or plugin to create those VMs 'on the fly' or do you create them manually ?

thank you again!

Le mercredi 26 septembre 2012 18:25:48 UTC+2, Mandeville, Rob a écrit :

Having the Jenkins server on a host that doesn’t run builds isn’t just interesting, it’s a good idea.  The only downside that I see is that the slave-master communication (probably dominated by your build logs and artifacts) is slowed down to network speeds rather than the speed of your loopback adapter.  I’m running in this configuration with 170+ slave nodes scattered across over a dozen build hosts (each build takes up a user account, so we need multiple slaves per host).  We basically build 60 different SCM branches of the same project.

 

Multiple masters can be useful.  I have one master for my build jobs, another one for housekeeping (I find it better than cron because it’s more visible and doesn’t spam you with email, and it’s also useful for some of your more routine command lines), and some people on a side project set up their own.  Part of the question is organizational: Jenkins is small enough that somebody working on a small project can set their own up, if they’re willing to spend a little of maintenance time.  That can keep you less involved in smaller projects, if this sort of a thing is acceptable to your organization.

 

--Rob

 

 

 

From: <a href="javascript:" target="_blank" gdf-obfuscated-mailto="mMkIES5PVP0J">[e-mail] <a href="javascript:" target="_blank" gdf-obfuscated-mailto="mMkIES5PVP0J">[mailto:[e-mail]] On Behalf Of mpapo - Michaël Pailloncy
Sent: Wednesday, September 26, 2012 8:58 AM
To: <a href="javascript:" target="_blank" gdf-obfuscated-mailto="mMkIES5PVP0J">[e-mail]
Subject: Feedback of your CI architecture

 

Hi all,

 

Currently, we manage near 250 jobs related to several java projects and several teams in our company.

 

Our CI architecture characteristics : 

- a Jenkins master on AIX 5.3 with 8 processors and 10 GB of RAM

- 3 RHEL slaves with 2 processors and 4 GB of RAM

- 2 Windows slaves with 1 processors and 2 GB of RAM

 

We arrive sometimes has limitations with Jenkins. Especially during releases.

 

Due to an increase of job number in our Jenkins in my company,  we want to change our architecture to meet our needs. We are going to manage near 500 jobs.

We want to switch the master on a RHEL OS (our new main target platform) and add an AIX slave for projects that need this platform.

 

Here are our thoughts :

- Do you think it would be interesting to have a master only used to monitor the jobs (0 executor, all builds are delegate to slaves) ? 

This could be interesting to have a more stable and more reliable master.

 

- On the other hand, do you think it could be interesting to use multiple masters ? One for each team for example ?

This could be nice to have a more specialized master per team but I guess it must increase the maintenance time.

 

 

I need feedbacks from the community to choose the best architecture :

How many jobs do you manage ?

What are the characteristics of your architecture ? (number of slaves, of master ??)

Would you prefer to use a slave with large capacity, or more with capacities smaller?

 

 

Thanks in advance from any help and feeback !

 

Michaël Pailloncy

 

 

The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.

[Mandeville, Rob]

The only VMs we use in development are for “side boxes” that need to run in Windows (we’re a Linux shop, but we run Selenium in a Windows VM to more closely approximate our users, who are mostly on Linux clients).  We find Windows VMs easier to manage than physical Windows machines.  We create these manually by creating one “master” and cloning it as many times as we need.  If one of our VMs are suspect, we just drop it and rebuild it from the master.  We considered the possibility of giving each of our slave nodes its own Linux VM, but it was too hardware-expensive for us.  We have 12 hour test cycles and need all the RAM and CPU we have, so we’re loath to let a VM layer consume either.  But that’s our own situation.

 

--Rob

 

 

From: mpapo - Michaël Pailloncy [mailto:[hidden email]]
Sent: Thursday, September 27, 2012 5:16 AM
To: [hidden email]
Cc: Mandeville, Rob
Subject: Re: Feedback of your CI architecture

 

Thank you very much for your feedback ! 

I think we're going to explore the idea of ​​creating one or more physical hosts creating VMs.

Do you use a particular tool or plugin to create those VMs 'on the fly' or do you create them manually ?

 

 

thank you again!

 

Le mercredi 26 septembre 2012 18:25:48 UTC+2, Mandeville, Rob a écrit :

Having the Jenkins server on a host that doesn’t run builds isn’t just interesting, it’s a good idea.  The only downside that I see is that the slave-master communication (probably dominated by your build logs and artifacts) is slowed down to network speeds rather than the speed of your loopback adapter.  I’m running in this configuration with 170+ slave nodes scattered across over a dozen build hosts (each build takes up a user account, so we need multiple slaves per host).  We basically build 60 different SCM branches of the same project.

 

Multiple masters can be useful.  I have one master for my build jobs, another one for housekeeping (I find it better than cron because it’s more visible and doesn’t spam you with email, and it’s also useful for some of your more routine command lines), and some people on a side project set up their own.  Part of the question is organizational: Jenkins is small enough that somebody working on a small project can set their own up, if they’re willing to spend a little of maintenance time.  That can keep you less involved in smaller projects, if this sort of a thing is acceptable to your organization.

 

--Rob

 

 

 

From: <a href="javascript:" target="_blank">jenkins...@... <a href="javascript:" target="_blank"> [mailto:jen...@...] On Behalf Of mpapo - Michaël Pailloncy
Sent: Wednesday, September 26, 2012 8:58 AM
To: <a href="javascript:" target="_blank">jenkins...@...
Subject: Feedback of your CI architecture

 

Hi all,

 

Currently, we manage near 250 jobs related to several java projects and several teams in our company.

 

Our CI architecture characteristics : 

- a Jenkins master on AIX 5.3 with 8 processors and 10 GB of RAM

- 3 RHEL slaves with 2 processors and 4 GB of RAM

- 2 Windows slaves with 1 processors and 2 GB of RAM

 

We arrive sometimes has limitations with Jenkins. Especially during releases.

 

Due to an increase of job number in our Jenkins in my company,  we want to change our architecture to meet our needs. We are going to manage near 500 jobs.

We want to switch the master on a RHEL OS (our new main target platform) and add an AIX slave for projects that need this platform.

 

Here are our thoughts :

- Do you think it would be interesting to have a master only used to monitor the jobs (0 executor, all builds are delegate to slaves) ? 

This could be interesting to have a more stable and more reliable master.

 

- On the other hand, do you think it could be interesting to use multiple masters ? One for each team for example ?

This could be nice to have a more specialized master per team but I guess it must increase the maintenance time.

 

 

I need feedbacks from the community to choose the best architecture :

How many jobs do you manage ?

What are the characteristics of your architecture ? (number of slaves, of master ??)

Would you prefer to use a slave with large capacity, or more with capacities smaller?

 

 

Thanks in advance from any help and feeback !

 

Michaël Pailloncy

 

 

The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.