Making new plugin - question

[ThomasBrouwer]

Hi there!

I am currently planning on making a new plugin that extends the functionality of the Exclusion plugin (https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin). What the plugin will allow you to do is to specify multiple resources of the same type, so that a pool of resources is created. When a job then needs that resource, it will be given one of them and there will be one less resource in the pool. This means we can specify how many jobs requiring the same type of resource can run (by making that many resources). When a resource is allocated, its name should also be passed to the job so that we know which of the resources was given. Also, when a resource is not available the job should go back into the queue, rather than waiting and taking up an execution slot. 

This is useful when, for example, you have multiple devices set up in a lab that you want a job to connect to, but there are multiple boards a job can use, and some it can't (because it requires a specific type of board). Currently we need to specify which of the boards it always runs on, but that is not very flexible. Sometimes a job may require multiple resources as well.

Now the question is: Does Jenkins allow you to change the way it schedules its jobs? That seems to be the most challenging part of this plugin, and I got the impression some things cannot be changed. Can this?

Any help is greatly appreciated!

Thomas Brouwer

[ffromm]

Hi Thomas,

 

have a look at https://wiki.jenkins-ci.org/display/JENKINS/Build+Blocker+Plugin. I used the QueueTaskDispatcher as Extension Point for a similar problem. There you just implement canRun or canTake method with your resource managed logic.

 

Hope this helps,

 

Cheers, Frederik

2012/8/13 ThomasBrouwer <[hidden email]>

Hi there!

I am currently planning on making a new plugin that extends the functionality of the Exclusion plugin (https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin). What the plugin will allow you to do is to specify multiple resources of the same type, so that a pool of resources is created. When a job then needs that resource, it will be given one of them and there will be one less resource in the pool. This means we can specify how many jobs requiring the same type of resource can run (by making that many resources). When a resource is allocated, its name should also be passed to the job so that we know which of the resources was given. Also, when a resource is not available the job should go back into the queue, rather than waiting and taking up an execution slot. 

This is useful when, for example, you have multiple devices set up in a lab that you want a job to connect to, but there are multiple boards a job can use, and some it can't (because it requires a specific type of board). Currently we need to specify which of the boards it always runs on, but that is not very flexible. Sometimes a job may require multiple resources as well.

Now the question is: Does Jenkins allow you to change the way it schedules its jobs? That seems to be the most challenging part of this plugin, and I got the impression some things cannot be changed. Can this?

Any help is greatly appreciated!

Thomas Brouwer

[cjo9900]

You also might want to look at the Throttle Concurrent Builds Plugin [1] which also provides similar functionality, also implementing the QueueTaskDispatcher and JobProperty.

You might also want to ask Andrew Bayer, the maintainer of that plugin to see if it feasible to extend that plugin with the features you require.

Chris

https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin 

On Monday, August 13, 2012 12:14:36 PM UTC+1, ffromm wrote:

Hi Thomas,

 

have a look at https://wiki.jenkins-ci.org/display/JENKINS/Build+Blocker+Plugin. I used the QueueTaskDispatcher as Extension Point for a similar problem. There you just implement canRun or canTake method with your resource managed logic.

 

Hope this helps,

 

Cheers, Frederik

2012/8/13 ThomasBrouwer <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="j1rYKcP3yQUJ">thomas.a...@...>

Hi there!

I am currently planning on making a new plugin that extends the functionality of the Exclusion plugin (https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin). What the plugin will allow you to do is to specify multiple resources of the same type, so that a pool of resources is created. When a job then needs that resource, it will be given one of them and there will be one less resource in the pool. This means we can specify how many jobs requiring the same type of resource can run (by making that many resources). When a resource is allocated, its name should also be passed to the job so that we know which of the resources was given. Also, when a resource is not available the job should go back into the queue, rather than waiting and taking up an execution slot. 

This is useful when, for example, you have multiple devices set up in a lab that you want a job to connect to, but there are multiple boards a job can use, and some it can't (because it requires a specific type of board). Currently we need to specify which of the boards it always runs on, but that is not very flexible. Sometimes a job may require multiple resources as well.

Now the question is: Does Jenkins allow you to change the way it schedules its jobs? That seems to be the most challenging part of this plugin, and I got the impression some things cannot be changed. Can this?

Any help is greatly appreciated!

Thomas Brouwer

[ThomasBrouwer]

Thanks both for the replies!

The Throttle Concurrent Builds Plugin comes close, but it is vital that the plugin can distribute resources rather than prevent more than N jobs running at the same time since we plan on using it to connect to boards in the lab, and we need to retrieve the IP address of the board a job gets allocated. I'll send Andrew an email :)

Thomas

On Monday, August 13, 2012 1:17:09 PM UTC+1, cjo wrote:

You also might want to look at the Throttle Concurrent Builds Plugin [1] which also provides similar functionality, also implementing the QueueTaskDispatcher and JobProperty.

You might also want to ask Andrew Bayer, the maintainer of that plugin to see if it feasible to extend that plugin with the features you require.

Chris

https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin 

On Monday, August 13, 2012 12:14:36 PM UTC+1, ffromm wrote:

Hi Thomas,

 

have a look at https://wiki.jenkins-ci.org/display/JENKINS/Build+Blocker+Plugin. I used the QueueTaskDispatcher as Extension Point for a similar problem. There you just implement canRun or canTake method with your resource managed logic.

 

Hope this helps,

 

Cheers, Frederik

2012/8/13 ThomasBrouwer <[hidden email]>

Hi there!

I am currently planning on making a new plugin that extends the functionality of the Exclusion plugin (https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin). What the plugin will allow you to do is to specify multiple resources of the same type, so that a pool of resources is created. When a job then needs that resource, it will be given one of them and there will be one less resource in the pool. This means we can specify how many jobs requiring the same type of resource can run (by making that many resources). When a resource is allocated, its name should also be passed to the job so that we know which of the resources was given. Also, when a resource is not available the job should go back into the queue, rather than waiting and taking up an execution slot. 

This is useful when, for example, you have multiple devices set up in a lab that you want a job to connect to, but there are multiple boards a job can use, and some it can't (because it requires a specific type of board). Currently we need to specify which of the boards it always runs on, but that is not very flexible. Sometimes a job may require multiple resources as well.

Now the question is: Does Jenkins allow you to change the way it schedules its jobs? That seems to be the most challenging part of this plugin, and I got the impression some things cannot be changed. Can this?

Any help is greatly appreciated!

Thomas Brouwer

[domi]

why not integrating it into the plugin you already mentioned?

The "Exclusion Plugin" even uses the same terminology already.

…sorry, but I really think we should take more care on improving existing plugins then just adding new ones with 80% the same functionality of an already existing one.

Domi

On 13.08.2012, at 11:51, ThomasBrouwer <[hidden email]> wrote:

Hi there!

I am currently planning on making a new plugin that extends the functionality of the Exclusion plugin (https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin). What the plugin will allow you to do is to specify multiple resources of the same type, so that a pool of resources is created. When a job then needs that resource, it will be given one of them and there will be one less resource in the pool. This means we can specify how many jobs requiring the same type of resource can run (by making that many resources). When a resource is allocated, its name should also be passed to the job so that we know which of the resources was given. Also, when a resource is not available the job should go back into the queue, rather than waiting and taking up an execution slot. 

This is useful when, for example, you have multiple devices set up in a lab that you want a job to connect to, but there are multiple boards a job can use, and some it can't (because it requires a specific type of board). Currently we need to specify which of the boards it always runs on, but that is not very flexible. Sometimes a job may require multiple resources as well.

Now the question is: Does Jenkins allow you to change the way it schedules its jobs? That seems to be the most challenging part of this plugin, and I got the impression some things cannot be changed. Can this?

Any help is greatly appreciated!

Thomas Brouwer

[slide]

+1

On Wed, Aug 15, 2012 at 9:36 AM, domi <[hidden email]> wrote:

> why not integrating it into the plugin you already mentioned?
> The "Exclusion Plugin" even uses the same terminology already.
> …sorry, but I really think we should take more care on improving existing
> plugins then just adding new ones with 80% the same functionality of an
> already existing one.
> Domi
>
>
> On 13.08.2012, at 11:51, ThomasBrouwer <[hidden email]> wrote:
>
> Hi there!
>
> I am currently planning on making a new plugin that extends the
> functionality of the Exclusion plugin
> (https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin). What the
> plugin will allow you to do is to specify multiple resources of the same
> type, so that a pool of resources is created. When a job then needs that
> resource, it will be given one of them and there will be one less resource
> in the pool. This means we can specify how many jobs requiring the same type
> of resource can run (by making that many resources). When a resource is
> allocated, its name should also be passed to the job so that we know which
> of the resources was given. Also, when a resource is not available the job
> should go back into the queue, rather than waiting and taking up an
> execution slot.
> This is useful when, for example, you have multiple devices set up in a lab
> that you want a job to connect to, but there are multiple boards a job can
> use, and some it can't (because it requires a specific type of board).
> Currently we need to specify which of the boards it always runs on, but that
> is not very flexible. Sometimes a job may require multiple resources as
> well.
>
> Now the question is: Does Jenkins allow you to change the way it schedules
> its jobs? That seems to be the most challenging part of this plugin, and I
> got the impression some things cannot be changed. Can this?
>
> Any help is greatly appreciated!
> Thomas Brouwer
>
>

--
Website: http://earl-of-code.com

[ThomasBrouwer]

The Exclusion plugin is implemented in such a way that the resource is only obtained during certain (specified) build steps of a job. As a result, it is impossible to put this job back into the waiting queue whilst waiting for a resource. For my proposed plugin I intend to make it so that a job gets a resource during its entire build, thus allowing this functionality. If I were to extend the Exclusion plugin, I would not be able to provide this functionality as there are cases in which both of them could be useful. I would not mind extending the current plugin, but that would not result in the plugin intended.

Op woensdag 15 augustus 2012 17:36:10 UTC+1 schreef domi het volgende:

why not integrating it into the plugin you already mentioned?

The "Exclusion Plugin" even uses the same terminology already.

…sorry, but I really think we should take more care on improving existing plugins then just adding new ones with 80% the same functionality of an already existing one.

Domi

On 13.08.2012, at 11:51, ThomasBrouwer <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="C0P11B_LX1AJ">thomas.a...@...> wrote:

Hi there!

I am currently planning on making a new plugin that extends the functionality of the Exclusion plugin (https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin). What the plugin will allow you to do is to specify multiple resources of the same type, so that a pool of resources is created. When a job then needs that resource, it will be given one of them and there will be one less resource in the pool. This means we can specify how many jobs requiring the same type of resource can run (by making that many resources). When a resource is allocated, its name should also be passed to the job so that we know which of the resources was given. Also, when a resource is not available the job should go back into the queue, rather than waiting and taking up an execution slot. 

This is useful when, for example, you have multiple devices set up in a lab that you want a job to connect to, but there are multiple boards a job can use, and some it can't (because it requires a specific type of board). Currently we need to specify which of the boards it always runs on, but that is not very flexible. Sometimes a job may require multiple resources as well.

Now the question is: Does Jenkins allow you to change the way it schedules its jobs? That seems to be the most challenging part of this plugin, and I got the impression some things cannot be changed. Can this?

Any help is greatly appreciated!

Thomas Brouwer

[domi]

Still I think this is just another flavor of the same thing and worth talking with the maintainer to find a solution.

as a Jenkins developer I also tend and like to implement new stuff, but as a Jenkins User looking at the plugins page - I feel overloaded and think there is to much of the same thing to choose from...

just my 5cent… up to you

On 15.08.2012, at 20:45, ThomasBrouwer <[hidden email]> wrote:

The Exclusion plugin is implemented in such a way that the resource is only obtained during certain (specified) build steps of a job. As a result, it is impossible to put this job back into the waiting queue whilst waiting for a resource. For my proposed plugin I intend to make it so that a job gets a resource during its entire build, thus allowing this functionality. If I were to extend the Exclusion plugin, I would not be able to provide this functionality as there are cases in which both of them could be useful. I would not mind extending the current plugin, but that would not result in the plugin intended.

Op woensdag 15 augustus 2012 17:36:10 UTC+1 schreef domi het volgende:

why not integrating it into the plugin you already mentioned?

The "Exclusion Plugin" even uses the same terminology already.

…sorry, but I really think we should take more care on improving existing plugins then just adding new ones with 80% the same functionality of an already existing one.

Domi

On 13.08.2012, at 11:51, ThomasBrouwer <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="C0P11B_LX1AJ">thomas.a...@...> wrote:

Hi there!

I am currently planning on making a new plugin that extends the functionality of the Exclusion plugin (https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin). What the plugin will allow you to do is to specify multiple resources of the same type, so that a pool of resources is created. When a job then needs that resource, it will be given one of them and there will be one less resource in the pool. This means we can specify how many jobs requiring the same type of resource can run (by making that many resources). When a resource is allocated, its name should also be passed to the job so that we know which of the resources was given. Also, when a resource is not available the job should go back into the queue, rather than waiting and taking up an execution slot. 

This is useful when, for example, you have multiple devices set up in a lab that you want a job to connect to, but there are multiple boards a job can use, and some it can't (because it requires a specific type of board). Currently we need to specify which of the boards it always runs on, but that is not very flexible. Sometimes a job may require multiple resources as well.

Now the question is: Does Jenkins allow you to change the way it schedules its jobs? That seems to be the most challenging part of this plugin, and I got the impression some things cannot be changed. Can this?

Any help is greatly appreciated!

Thomas Brouwer

[ThomasBrouwer]

I definitely agree with you :) the plugin will be made for my company's use anyways, so it will be released. Afterwards, a collaboration with the Exclusion plugin can always be made  to merge them into one.

Op woensdag 15 augustus 2012 20:05:49 UTC+1 schreef domi het volgende:

Still I think this is just another flavor of the same thing and worth talking with the maintainer to find a solution.

as a Jenkins developer I also tend and like to implement new stuff, but as a Jenkins User looking at the plugins page - I feel overloaded and think there is to much of the same thing to choose from...

just my 5cent… up to you

On 15.08.2012, at 20:45, ThomasBrouwer <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="rLBZ7Q1KpmYJ">thomas.a...@...> wrote:

The Exclusion plugin is implemented in such a way that the resource is only obtained during certain (specified) build steps of a job. As a result, it is impossible to put this job back into the waiting queue whilst waiting for a resource. For my proposed plugin I intend to make it so that a job gets a resource during its entire build, thus allowing this functionality. If I were to extend the Exclusion plugin, I would not be able to provide this functionality as there are cases in which both of them could be useful. I would not mind extending the current plugin, but that would not result in the plugin intended.

Op woensdag 15 augustus 2012 17:36:10 UTC+1 schreef domi het volgende:

why not integrating it into the plugin you already mentioned?

The "Exclusion Plugin" even uses the same terminology already.

…sorry, but I really think we should take more care on improving existing plugins then just adding new ones with 80% the same functionality of an already existing one.

Domi

On 13.08.2012, at 11:51, ThomasBrouwer <[hidden email]> wrote:

Hi there!

I am currently planning on making a new plugin that extends the functionality of the Exclusion plugin (https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin). What the plugin will allow you to do is to specify multiple resources of the same type, so that a pool of resources is created. When a job then needs that resource, it will be given one of them and there will be one less resource in the pool. This means we can specify how many jobs requiring the same type of resource can run (by making that many resources). When a resource is allocated, its name should also be passed to the job so that we know which of the resources was given. Also, when a resource is not available the job should go back into the queue, rather than waiting and taking up an execution slot. 

This is useful when, for example, you have multiple devices set up in a lab that you want a job to connect to, but there are multiple boards a job can use, and some it can't (because it requires a specific type of board). Currently we need to specify which of the boards it always runs on, but that is not very flexible. Sometimes a job may require multiple resources as well.

Now the question is: Does Jenkins allow you to change the way it schedules its jobs? That seems to be the most challenging part of this plugin, and I got the impression some things cannot be changed. Can this?

Any help is greatly appreciated!

Thomas Brouwer

[cjo9900]

As I mentioned earlier in the thread I think Thomas's use case is a lot closer to the Throttle Concurrent builds plugin, 

where the number of concurrent builds is limited by a set of resources rather than a numeric value.

which is then provided to the build as an environment value.

And from a quick look at the code really looks to be an extension of the ThrottleJobProperty where the prebuild should be overridden to allow categories to add an EnvContributingAction.

The ThrottleCategory would would have a separate extension for resource based Category which can keep track of used resources a maximum number of resources in a set.

https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin  

Chris.

On Wednesday, August 15, 2012 10:09:17 PM UTC+1, ThomasBrouwer wrote:

I definitely agree with you :) the plugin will be made for my company's use anyways, so it will be released. Afterwards, a collaboration with the Exclusion plugin can always be made  to merge them into one.

Op woensdag 15 augustus 2012 20:05:49 UTC+1 schreef domi het volgende:

Still I think this is just another flavor of the same thing and worth talking with the maintainer to find a solution.

as a Jenkins developer I also tend and like to implement new stuff, but as a Jenkins User looking at the plugins page - I feel overloaded and think there is to much of the same thing to choose from...

just my 5cent… up to you

On 15.08.2012, at 20:45, ThomasBrouwer <[hidden email]> wrote:

The Exclusion plugin is implemented in such a way that the resource is only obtained during certain (specified) build steps of a job. As a result, it is impossible to put this job back into the waiting queue whilst waiting for a resource. For my proposed plugin I intend to make it so that a job gets a resource during its entire build, thus allowing this functionality. If I were to extend the Exclusion plugin, I would not be able to provide this functionality as there are cases in which both of them could be useful. I would not mind extending the current plugin, but that would not result in the plugin intended.

Op woensdag 15 augustus 2012 17:36:10 UTC+1 schreef domi het volgende:

why not integrating it into the plugin you already mentioned?

The "Exclusion Plugin" even uses the same terminology already.

…sorry, but I really think we should take more care on improving existing plugins then just adding new ones with 80% the same functionality of an already existing one.

Domi

On 13.08.2012, at 11:51, ThomasBrouwer <[hidden email]> wrote:

Hi there!

I am currently planning on making a new plugin that extends the functionality of the Exclusion plugin (https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin). What the plugin will allow you to do is to specify multiple resources of the same type, so that a pool of resources is created. When a job then needs that resource, it will be given one of them and there will be one less resource in the pool. This means we can specify how many jobs requiring the same type of resource can run (by making that many resources). When a resource is allocated, its name should also be passed to the job so that we know which of the resources was given. Also, when a resource is not available the job should go back into the queue, rather than waiting and taking up an execution slot. 

This is useful when, for example, you have multiple devices set up in a lab that you want a job to connect to, but there are multiple boards a job can use, and some it can't (because it requires a specific type of board). Currently we need to specify which of the boards it always runs on, but that is not very flexible. Sometimes a job may require multiple resources as well.

Now the question is: Does Jenkins allow you to change the way it schedules its jobs? That seems to be the most challenging part of this plugin, and I got the impression some things cannot be changed. Can this?

Any help is greatly appreciated!

Thomas Brouwer