queueing project dependencies

[Cindy Brown]

Hello,

 

We are evaluating Hudson to use as our continuous integration build server for a large number of .NET projects with complicated dependency structures. Because of the dependencies we are using the checkbox “block build when upstream project is building” in connection with project triggers. This gives us a queue based on the project triggers.

 

What we would like to have in addition though is a checkbox that does the reverse – “block build when downstream project is building”. One of the problems we are trying to solve is that since we have the builds poll the source control system for changes and kick off a build at check in the queue will continuously get restarted if there is a lot of checkin activity on a top level project, lower level projects are waiting sometimes 8 hrs for a build. We would like to configure some way where the projects are aware of queue iterations and let one iteration finish before beginning another or inserting themselves into the top of the queue.

 

Has anyone else experienced this issue or have suggestions for configuring projects to solve this?

 

Thanks for any comments or suggestions.

 

[Michael Donohue]

Blocking the queue limits your CI throughput, as you've noticed.   If you uncheck the "block build when upstream project is building" what bad outcomes are there?    There are a number of plugins that could help you get better performance while maintaining certain invariants, but without specifics, it's hard to recommend one solution over another.

The join plugin and 'locks-and-latches' come to mind as likely useful for you.

-Michael
[phone]

On Tue, Mar 16, 2010 at 10:46 AM, Cindy Brown <[hidden email]> wrote:

Hello,

 

We are evaluating Hudson to use as our continuous integration build server for a large number of .NET projects with complicated dependency structures. Because of the dependencies we are using the checkbox “block build when upstream project is building” in connection with project triggers. This gives us a queue based on the project triggers.

 

What we would like to have in addition though is a checkbox that does the reverse – “block build when downstream project is building”. One of the problems we are trying to solve is that since we have the builds poll the source control system for changes and kick off a build at check in the queue will continuously get restarted if there is a lot of checkin activity on a top level project, lower level projects are waiting sometimes 8 hrs for a build. We would like to configure some way where the projects are aware of queue iterations and let one iteration finish before beginning another or inserting themselves into the top of the queue.

 

Has anyone else experienced this issue or have suggestions for configuring projects to solve this?

 

Thanks for any comments or suggestions.

 

[Cindy Brown]

Thanks for your reply Michael.

The bad outcome from unchecking the “block build when upstream project is building” is our projects may not build with the latest code in their dependencies and we get more broken builds. We have considered running most builds without the above checkbox and then turn it on for the last release candidate builds.

 

I’ve been looking at the join plugin and the locks-and-latches plugin. It looks like the one we could use the most would be locks-and-latches but from what I can tell only the locks part has been implemented. Do you or does anyone know of an eta on the latches part? Or perhaps some industrious soul would venture to do a “block build when downstream project is building” plugin.

 

I am more than willing to provide any of our details in an effort to explore your suggestions.

 

Thanks again for your reply.

 

 

From: Michael Donohue [mailto:[hidden email]]
Sent: Tuesday, March 16, 2010 12:08 PM
To: [hidden email]
Subject: Re: queueing project dependencies

 

Blocking the queue limits your CI throughput, as you've noticed.   If you uncheck the "block build when upstream project is building" what bad outcomes are there?    There are a number of plugins that could help you get better performance while maintaining certain invariants, but without specifics, it's hard to recommend one solution over another.

 

The join plugin and 'locks-and-latches' come to mind as likely useful for you.

 

-Michael
[phone]

On Tue, Mar 16, 2010 at 10:46 AM, Cindy Brown <[hidden email]> wrote:

Hello,

 

We are evaluating Hudson to use as our continuous integration build server for a large number of .NET projects with complicated dependency structures. Because of the dependencies we are using the checkbox “block build when upstream project is building” in connection with project triggers. This gives us a queue based on the project triggers.

 

What we would like to have in addition though is a checkbox that does the reverse – “block build when downstream project is building”. One of the problems we are trying to solve is that since we have the builds poll the source control system for changes and kick off a build at check in the queue will continuously get restarted if there is a lot of checkin activity on a top level project, lower level projects are waiting sometimes 8 hrs for a build. We would like to configure some way where the projects are aware of queue iterations and let one iteration finish before beginning another or inserting themselves into the top of the queue.

 

Has anyone else experienced this issue or have suggestions for configuring projects to solve this?

 

Thanks for any comments or suggestions.

 

 

[Michael Donohue]

It's never been clear to me what the 'latches' part of that plugin would provide, but that may just be a deficiency in my terminology.    But my best guess for an ETA is never, since that plugin hardly ever changes these days.  But if you want that feature, and since you seem to know what it means, plenty of folks would give you pointers on implementing it, if you choose to do so.   That would be a topic for the dev list, however.

As for 'block when downstream is building' isn't that the same as just having one giant build job?   Can you combine the build steps together into one giant Hudson project?

-Michael
[phone]

On Tue, Mar 16, 2010 at 2:00 PM, Cindy Brown <[hidden email]> wrote:

Thanks for your reply Michael.

The bad outcome from unchecking the “block build when upstream project is building” is our projects may not build with the latest code in their dependencies and we get more broken builds. We have considered running most builds without the above checkbox and then turn it on for the last release candidate builds.

 

I’ve been looking at the join plugin and the locks-and-latches plugin. It looks like the one we could use the most would be locks-and-latches but from what I can tell only the locks part has been implemented. Do you or does anyone know of an eta on the latches part? Or perhaps some industrious soul would venture to do a “block build when downstream project is building” plugin.

 

I am more than willing to provide any of our details in an effort to explore your suggestions.

 

Thanks again for your reply.

 

 

From: Michael Donohue [mailto:[hidden email]]
Sent: Tuesday, March 16, 2010 12:08 PM
To: [hidden email]
Subject: Re: queueing project dependencies

 

Blocking the queue limits your CI throughput, as you've noticed.   If you uncheck the "block build when upstream project is building" what bad outcomes are there?    There are a number of plugins that could help you get better performance while maintaining certain invariants, but without specifics, it's hard to recommend one solution over another.

 

The join plugin and 'locks-and-latches' come to mind as likely useful for you.

 

-Michael
[phone]

On Tue, Mar 16, 2010 at 10:46 AM, Cindy Brown <[hidden email]> wrote:

Hello,

 

We are evaluating Hudson to use as our continuous integration build server for a large number of .NET projects with complicated dependency structures. Because of the dependencies we are using the checkbox “block build when upstream project is building” in connection with project triggers. This gives us a queue based on the project triggers.

 

What we would like to have in addition though is a checkbox that does the reverse – “block build when downstream project is building”. One of the problems we are trying to solve is that since we have the builds poll the source control system for changes and kick off a build at check in the queue will continuously get restarted if there is a lot of checkin activity on a top level project, lower level projects are waiting sometimes 8 hrs for a build. We would like to configure some way where the projects are aware of queue iterations and let one iteration finish before beginning another or inserting themselves into the top of the queue.

 

Has anyone else experienced this issue or have suggestions for configuring projects to solve this?

 

Thanks for any comments or suggestions.

 

 

[Sami Tikka]

2010/3/16 Cindy Brown <[hidden email]>:
> Thanks for your reply Michael.
>
> The bad outcome from unchecking the “block build when upstream project is
> building” is our projects may not build with the latest code in their
> dependencies and we get more broken builds. We have considered running most
> builds without the above checkbox and then turn it on for the last release
> candidate builds.

Broken builds are not bad in itself. (Unless your bosses have set up
an incentive program based on the ratio of successful and broken
builds, but that is another problem and something Hudson cannot
solve.)

If a build is broken because of incompatible changes in its
dependencies, isn't it good that you and the team knows it?

Then the problem will be fixed and a new build will be successful.

If your problem is a slow feedback cycle caused by e.g. long build or
test times, then that's where your real problem is. Solve the problem
instead of mitigating the symptoms. Try to parallelize. Look into
distributing your builds and tests using Hudson slaves.

For taking a snapshot for release candidate builds, you could consider
the Promoted builds plugin. We like it.

-- Sami

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Cindy Brown]

Thanks again for your response Michael. I wasn’t exactly sure about the latches either, however I was game for giving it a try to see what it would do (if it existed). I don’t think combining the build steps for all of the projects in the q is a viable option for us although it was an interesting suggestion and I will keep it in mind. We will do some more testing to see what we can come up with.

 

Thank you very much for your time. I appreciate your responses.

 

 

From: Michael Donohue [mailto:[hidden email]]
Sent: Tuesday, March 16, 2010 3:18 PM
To: [hidden email]
Subject: Re: Re: queueing project dependencies

 

It's never been clear to me what the 'latches' part of that plugin would provide, but that may just be a deficiency in my terminology.    But my best guess for an ETA is never, since that plugin hardly ever changes these days.  But if you want that feature, and since you seem to know what it means, plenty of folks would give you pointers on implementing it, if you choose to do so.   That would be a topic for the dev list, however.

 

As for 'block when downstream is building' isn't that the same as just having one giant build job?   Can you combine the build steps together into one giant Hudson project?

-Michael
[phone]

On Tue, Mar 16, 2010 at 2:00 PM, Cindy Brown <[hidden email]> wrote:

Thanks for your reply Michael.

The bad outcome from unchecking the “block build when upstream project is building” is our projects may not build with the latest code in their dependencies and we get more broken builds. We have considered running most builds without the above checkbox and then turn it on for the last release candidate builds.

 

I’ve been looking at the join plugin and the locks-and-latches plugin. It looks like the one we could use the most would be locks-and-latches but from what I can tell only the locks part has been implemented. Do you or does anyone know of an eta on the latches part? Or perhaps some industrious soul would venture to do a “block build when downstream project is building” plugin.

 

I am more than willing to provide any of our details in an effort to explore your suggestions.

 

Thanks again for your reply.

 

 

From: Michael Donohue [mailto:[hidden email]]
Sent: Tuesday, March 16, 2010 12:08 PM
To: [hidden email]
Subject: Re: queueing project dependencies

 

Blocking the queue limits your CI throughput, as you've noticed.   If you uncheck the "block build when upstream project is building" what bad outcomes are there?    There are a number of plugins that could help you get better performance while maintaining certain invariants, but without specifics, it's hard to recommend one solution over another.

 

The join plugin and 'locks-and-latches' come to mind as likely useful for you.

 

-Michael
[phone]

On Tue, Mar 16, 2010 at 10:46 AM, Cindy Brown <[hidden email]> wrote:

Hello,

 

We are evaluating Hudson to use as our continuous integration build server for a large number of .NET projects with complicated dependency structures. Because of the dependencies we are using the checkbox “block build when upstream project is building” in connection with project triggers. This gives us a queue based on the project triggers.

 

What we would like to have in addition though is a checkbox that does the reverse – “block build when downstream project is building”. One of the problems we are trying to solve is that since we have the builds poll the source control system for changes and kick off a build at check in the queue will continuously get restarted if there is a lot of checkin activity on a top level project, lower level projects are waiting sometimes 8 hrs for a build. We would like to configure some way where the projects are aware of queue iterations and let one iteration finish before beginning another or inserting themselves into the top of the queue.

 

Has anyone else experienced this issue or have suggestions for configuring projects to solve this?

 

Thanks for any comments or suggestions.

 

 

 

[Cindy Brown]

Sami,

Thanks for your comments. We will be looking into the things you mention
below.


-----Original Message-----
From: Sami Tikka [mailto:[hidden email]]
Sent: Tuesday, March 16, 2010 6:16 PM
To: [hidden email]
Subject: Re: Re: queueing project dependencies

2010/3/16 Cindy Brown <[hidden email]>:
> Thanks for your reply Michael.
>
> The bad outcome from unchecking the "block build when upstream project
is
> building" is our projects may not build with the latest code in their
> dependencies and we get more broken builds. We have considered running
most
> builds without the above checkbox and then turn it on for the last
release
> candidate builds.

Broken builds are not bad in itself. (Unless your bosses have set up
an incentive program based on the ratio of successful and broken
builds, but that is another problem and something Hudson cannot
solve.)

If a build is broken because of incompatible changes in its
dependencies, isn't it good that you and the team knows it?

Then the problem will be fixed and a new build will be successful.

If your problem is a slow feedback cycle caused by e.g. long build or
test times, then that's where your real problem is. Solve the problem
instead of mitigating the symptoms. Try to parallelize. Look into
distributing your builds and tests using Hudson slaves.

For taking a snapshot for release candidate builds, you could consider
the Promoted builds plugin. We like it.

-- Sami

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]