Build with for loop like finesse

[rusty]

PROBLEM:   I need to build a project with certain parameters which then kicks of various other projects.  I then need to build the same first project with a different set of parameters as  soon as the full build pipeline finishes from the original build.

eg:

project1 with first set of parameters:
    project2:
        project3:
        project4:
            project5:
                project6:
        project7:
project1 with second set of parameters:
    project2:
        project3:
        project4:
            project5:
                project6:
        project7:

Note that all children projects depend on parameters passed down from project1 and that due to lack of hardware, I cannot build the two different project1's in parallel.

QUESTION:  How do I accomplish top level builds that spawn of a plethora of children projects and then be able to spawn another top level build when the current top level build, and all children builds including project6 from my example, has finished?

[Sami Tikka]

Use a multi-configuration project with one, two or more axes.

-- Sami

rusty <[hidden email]> kirjoitti 14.8.2012 kello 5.39:

> PROBLEM:   I need to build a project with certain parameters which then kicks of various other projects.  I then need to build the same first project with a different set of parameters as  soon as the full build pipeline finishes from the original build.
>
> eg:
>
> project1 with first set of parameters:
>     project2:
>         project3:
>         project4:
>             project5:
>                 project6:
>         project7:
> project1 with second set of parameters:
>     project2:
>         project3:
>         project4:
>             project5:
>                 project6:
>         project7:
>
> Note that all children projects depend on parameters passed down from project1 and that due to lack of hardware, I cannot build the two different project1's in parallel.
>
> QUESTION:  How do I accomplish top level builds that spawn of a plethora of children projects and then be able to spawn another top level build when the current top level build, and all children builds including project6 from my example, has finished?
>

[rusty]

I am already using a multi-configuration project, I think that the disconnect is stemming from how do I use multiple axis to accompolish this.

Won't adding another axis start the two projects in parallel?  I can't have two projects in paralelll...

On Tuesday, August 14, 2012 3:06:38 AM UTC-6, sti wrote:

Use a multi-configuration project with one, two or more axes.

-- Sami

rusty <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="1IHvA7HI7boJ">[e-mail]> kirjoitti 14.8.2012 kello 5.39:

[Sami Tikka]

All the builds will enter the queue at the same time, but how many will actually run in parallel depends on how many executors you have or if you have used other means of concurrency control like Throttle Concurrent Builds plugin. 

-- Sami

rusty <[hidden email]> kirjoitti 14.8.2012 kello 19.26:

I am already using a multi-configuration project, I think that the disconnect is stemming from how do I use multiple axis to accompolish this.

Won't adding another axis start the two projects in parallel?  I can't have two projects in paralelll...

On Tuesday, August 14, 2012 3:06:38 AM UTC-6, sti wrote:

Use a multi-configuration project with one, two or more axes.

-- Sami

rusty <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="1IHvA7HI7boJ">[e-mail]> kirjoitti 14.8.2012 kello 5.39:

[rusty]

The problem is that I can have no concurrency in between the two top projects nor any child projects that they have.  I have looked at throttle builds  plugin, but it doesn't provide what I need which is full atomicity on the top level builds and all recursive underlying builds. 

On Tuesday, August 14, 2012 10:37:05 AM UTC-6, sti wrote:

All the builds will enter the queue at the same time, but how many will actually run in parallel depends on how many executors you have or if you have used other means of concurrency control like Throttle Concurrent Builds plugin. 

-- Sami

rusty <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="sq2YP8uTbSUJ">[e-mail]> kirjoitti 14.8.2012 kello 19.26:

I am already using a multi-configuration project, I think that the disconnect is stemming from how do I use multiple axis to accompolish this.

Won't adding another axis start the two projects in parallel?  I can't have two projects in paralelll...

On Tuesday, August 14, 2012 3:06:38 AM UTC-6, sti wrote:

Use a multi-configuration project with one, two or more axes.

-- Sami

rusty <[e-mail]> kirjoitti 14.8.2012 kello 5.39:

[cjo9900]

If all of the builds are triggered by project 1, and you only want to run one set of parameters at a time through the chain at a time.

You should be able to use the "Block build when downstream project is building" and "Block build when upstream project is building" advanced options on all of the projects which will enforce this behaviour.

Doing this will allow you to add Project1 to the queue multiple times with different parameters but only have one run for each set of parameters.

Warning: We found an issue with this if a downstream project is blocked (waiting for a resource/slave to come online) which will allow Project 1 to run again.

(Due to fix for https://issues.jenkins-ci.org/browse/JENKINS-8929)

We ran this for several months without problem and in some cases had 5-10 instances of Project 1 queued with different parameters.

This was needed as we had a dependency between multiple projects needing to share workspaces, due to the SCM and size of the project we were using along with the 30 different build configurations.

Chris

On Tuesday, August 14, 2012 5:54:20 PM UTC+1, rusty wrote:

The problem is that I can have no concurrency in between the two top projects nor any child projects that they have.  I have looked at throttle builds  plugin, but it doesn't provide what I need which is full atomicity on the top level builds and all recursive underlying builds. 

On Tuesday, August 14, 2012 10:37:05 AM UTC-6, sti wrote:

All the builds will enter the queue at the same time, but how many will actually run in parallel depends on how many executors you have or if you have used other means of concurrency control like Throttle Concurrent Builds plugin. 

-- Sami

rusty <[e-mail]> kirjoitti 14.8.2012 kello 19.26:

I am already using a multi-configuration project, I think that the disconnect is stemming from how do I use multiple axis to accompolish this.

Won't adding another axis start the two projects in parallel?  I can't have two projects in paralelll...

On Tuesday, August 14, 2012 3:06:38 AM UTC-6, sti wrote:

Use a multi-configuration project with one, two or more axes.

-- Sami

rusty <[e-mail]> kirjoitti 14.8.2012 kello 5.39:

[rusty]

so.....  f I understand this completly
If I "Block build when downstream project is building" Jenkins will resolve all of the downstream builds recursively allowing all children as well as grandchildren projects to be built before building project 1 again with different parameters?

On Tuesday, August 14, 2012 11:31:36 AM UTC-6, cjo wrote:

If all of the builds are triggered by project 1, and you only want to run one set of parameters at a time through the chain at a time.

You should be able to use the "Block build when downstream project is building" and "Block build when upstream project is building" advanced options on all of the projects which will enforce this behaviour.

Doing this will allow you to add Project1 to the queue multiple times with different parameters but only have one run for each set of parameters.

Warning: We found an issue with this if a downstream project is blocked (waiting for a resource/slave to come online) which will allow Project 1 to run again.

(Due to fix for https://issues.jenkins-ci.org/browse/JENKINS-8929)

We ran this for several months without problem and in some cases had 5-10 instances of Project 1 queued with different parameters.

This was needed as we had a dependency between multiple projects needing to share workspaces, due to the SCM and size of the project we were using along with the 30 different build configurations.

Chris

On Tuesday, August 14, 2012 5:54:20 PM UTC+1, rusty wrote:

The problem is that I can have no concurrency in between the two top projects nor any child projects that they have.  I have looked at throttle builds  plugin, but it doesn't provide what I need which is full atomicity on the top level builds and all recursive underlying builds. 

On Tuesday, August 14, 2012 10:37:05 AM UTC-6, sti wrote:

All the builds will enter the queue at the same time, but how many will actually run in parallel depends on how many executors you have or if you have used other means of concurrency control like Throttle Concurrent Builds plugin. 

-- Sami

rusty <[e-mail]> kirjoitti 14.8.2012 kello 19.26:

I am already using a multi-configuration project, I think that the disconnect is stemming from how do I use multiple axis to accompolish this.

Won't adding another axis start the two projects in parallel?  I can't have two projects in paralelll...

On Tuesday, August 14, 2012 3:06:38 AM UTC-6, sti wrote:

Use a multi-configuration project with one, two or more axes.

-- Sami

rusty <[e-mail]> kirjoitti 14.8.2012 kello 5.39:

[cjo9900]

Yes, apart from the issue I mentioned, haven't had a chance to look through existing issues yet to see if it is reported.

On Tuesday, August 14, 2012 9:21:21 PM UTC+1, rusty wrote:

so.....  f I understand this completly
If I "Block build when downstream project is building" Jenkins will resolve all of the downstream builds recursively allowing all children as well as grandchildren projects to be built before building project 1 again with different parameters?

On Tuesday, August 14, 2012 11:31:36 AM UTC-6, cjo wrote:

If all of the builds are triggered by project 1, and you only want to run one set of parameters at a time through the chain at a time.

You should be able to use the "Block build when downstream project is building" and "Block build when upstream project is building" advanced options on all of the projects which will enforce this behaviour.

Doing this will allow you to add Project1 to the queue multiple times with different parameters but only have one run for each set of parameters.

Warning: We found an issue with this if a downstream project is blocked (waiting for a resource/slave to come online) which will allow Project 1 to run again.

(Due to fix for https://issues.jenkins-ci.org/browse/JENKINS-8929)

We ran this for several months without problem and in some cases had 5-10 instances of Project 1 queued with different parameters.

This was needed as we had a dependency between multiple projects needing to share workspaces, due to the SCM and size of the project we were using along with the 30 different build configurations.

Chris

On Tuesday, August 14, 2012 5:54:20 PM UTC+1, rusty wrote:

The problem is that I can have no concurrency in between the two top projects nor any child projects that they have.  I have looked at throttle builds  plugin, but it doesn't provide what I need which is full atomicity on the top level builds and all recursive underlying builds. 

On Tuesday, August 14, 2012 10:37:05 AM UTC-6, sti wrote:

All the builds will enter the queue at the same time, but how many will actually run in parallel depends on how many executors you have or if you have used other means of concurrency control like Throttle Concurrent Builds plugin. 

-- Sami

rusty <[e-mail]> kirjoitti 14.8.2012 kello 19.26:

I am already using a multi-configuration project, I think that the disconnect is stemming from how do I use multiple axis to accompolish this.

Won't adding another axis start the two projects in parallel?  I can't have two projects in paralelll...

On Tuesday, August 14, 2012 3:06:38 AM UTC-6, sti wrote:

Use a multi-configuration project with one, two or more axes.

-- Sami

rusty <[e-mail]> kirjoitti 14.8.2012 kello 5.39:

[rusty]

Unfortunatly, after attempting the said method..  jenkins still ended up attempting both in parallel. It looks like that method has a race condition where it only checks at the beginning of the build if any downstream projects are building;  which is a problem if any builds next to the build end up executing anything in the downstream.   Adding a wait period to one  of the builds will fix the race condition, but that is super hacky in my opinion.

On Wednesday, August 15, 2012 5:15:53 AM UTC-6, cjo wrote:

Yes, apart from the issue I mentioned, haven't had a chance to look through existing issues yet to see if it is reported.

On Tuesday, August 14, 2012 9:21:21 PM UTC+1, rusty wrote:

so.....  f I understand this completly
If I "Block build when downstream project is building" Jenkins will resolve all of the downstream builds recursively allowing all children as well as grandchildren projects to be built before building project 1 again with different parameters?

On Tuesday, August 14, 2012 11:31:36 AM UTC-6, cjo wrote:

If all of the builds are triggered by project 1, and you only want to run one set of parameters at a time through the chain at a time.

You should be able to use the "Block build when downstream project is building" and "Block build when upstream project is building" advanced options on all of the projects which will enforce this behaviour.

Doing this will allow you to add Project1 to the queue multiple times with different parameters but only have one run for each set of parameters.

Warning: We found an issue with this if a downstream project is blocked (waiting for a resource/slave to come online) which will allow Project 1 to run again.

(Due to fix for https://issues.jenkins-ci.org/browse/JENKINS-8929)

We ran this for several months without problem and in some cases had 5-10 instances of Project 1 queued with different parameters.

This was needed as we had a dependency between multiple projects needing to share workspaces, due to the SCM and size of the project we were using along with the 30 different build configurations.

Chris

On Tuesday, August 14, 2012 5:54:20 PM UTC+1, rusty wrote:

The problem is that I can have no concurrency in between the two top projects nor any child projects that they have.  I have looked at throttle builds  plugin, but it doesn't provide what I need which is full atomicity on the top level builds and all recursive underlying builds. 

On Tuesday, August 14, 2012 10:37:05 AM UTC-6, sti wrote:

All the builds will enter the queue at the same time, but how many will actually run in parallel depends on how many executors you have or if you have used other means of concurrency control like Throttle Concurrent Builds plugin. 

-- Sami

rusty <[e-mail]> kirjoitti 14.8.2012 kello 19.26:

I am already using a multi-configuration project, I think that the disconnect is stemming from how do I use multiple axis to accompolish this.

Won't adding another axis start the two projects in parallel?  I can't have two projects in paralelll...

On Tuesday, August 14, 2012 3:06:38 AM UTC-6, sti wrote:

Use a multi-configuration project with one, two or more axes.

-- Sami

rusty <[e-mail]> kirjoitti 14.8.2012 kello 5.39:

[Sami Tikka]

So assign the jobs to run on a slave with only 1 executor. I guarantee only one at a time will run.

-- Sami

rusty <[hidden email]> kirjoitti 14.8.2012 kello 19.54:

The problem is that I can have no concurrency in between the two top projects nor any child projects that they have.  I have looked at throttle builds  plugin, but it doesn't provide what I need which is full atomicity on the top level builds and all recursive underlying builds. 

On Tuesday, August 14, 2012 10:37:05 AM UTC-6, sti wrote:

All the builds will enter the queue at the same time, but how many will actually run in parallel depends on how many executors you have or if you have used other means of concurrency control like Throttle Concurrent Builds plugin. 

-- Sami

rusty <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="sq2YP8uTbSUJ">[e-mail]> kirjoitti 14.8.2012 kello 19.26:

I am already using a multi-configuration project, I think that the disconnect is stemming from how do I use multiple axis to accompolish this.

Won't adding another axis start the two projects in parallel?  I can't have two projects in paralelll...

On Tuesday, August 14, 2012 3:06:38 AM UTC-6, sti wrote:

Use a multi-configuration project with one, two or more axes.

-- Sami

rusty <[e-mail]> kirjoitti 14.8.2012 kello 5.39: