Pipelines in Jenkins...

[Jeff Vincent]

I'm trying to implement Jenkins with Maven as the build CI/CD tool since it does so well with the technologies we are using out of the box instead of GO that the larger .NET-based organization is  using.

 

I've done Jenkins jobs for stand-alone projects but not jobs chained from dependent projects.

 

Conceptually, I can see how dependent jobs are configured using the build pipeline plugin (though we only have one project so far).  What is unclear is how downstream jobs are configured to consume the artifacts and/or outputs produced upstream.

 

Off the top of my head, how would jobs be created/configured to perform the steps in the pipeline sequence below:

1. (JOB) Build shared library
1. Run Unit tests
2. Upload SNAPSHOT to Maven repository (e.g., Shared-1.6.3-SNAPSHOT-12.jar)
2. (JOB) Trigger build of WebApp
1. Build WebApp and include latest SNAPSHOT shared library (e.g., Shared-1.6.3-SNAPSHOT-12.jar)
2. Run Unit tests for WebApp
3. Upload WAR to repository (e.g., WebApp-1.4.2-SNAPSHOT-6.war)
3. (JOB) Deploy WebApp-1.4.2-SNAPSHOT-6.war
1. Grab WebApp-1.4.2-SNAPSHOT-6.war
2. Deploy to DEV server (tomcat container)
1. use DEV config.properties file (not sure how to switch out properties based on deploy env)
4. (JOB) Run Integration tests
5. (JOB) Deploy to STAGE environment
1. Grab (somehow) WebApp-1.4.2-SNAPSHOT-6.war
2. Deploy to ALL STAGE server (tomcat container) nodes (load balanced - two to 4 nodes)
1. use STAGE config.properties file (not sure how to switch out properties based on deploy env)
6. (JOB) Run Integration Tests (Validation only?)
7. (JOB) Run Load tests
8. (JOB) Deploy to LIVE

 

I've read through the Jenkins Build pipeline plugin page, but could really use some additional perspectives/resources, particularly with steps 3-8. 

 

Any tips/resources that I should read or insight into how you do this would be appreciated.

--

Jeff Vincent
[hidden email]
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

I ♥ DropBox !! 

[R. Tyler Croy]

On Wed, 15 Feb 2012, Jeff wrote:

> I'm trying to implement Jenkins with Maven as the build CI/CD tool since it
> does so well with the technologies we are using out of the box instead of
> GO that the larger .NET-based organization is  using.
>
> I've done Jenkins jobs for stand-alone projects but not jobs chained from
> dependent projects.
>
> Conceptually, I can see how dependent jobs are configured using the build
> pipeline plugin (though we only have one project so far).  What is unclear
> is how downstream jobs are configured to consume the
> artifacts and/or outputs produced upstream.

Plugins to check out:

 * Join plugin
 * Copy Artifacts plugin


Those might meet your needs, I use them both heavily to create longish
pipelines.

- R. Tyler Croy
--------------------------------------
    Code: http://github.com/rtyler
 Chatter: http://twitter.com/agentdero
          [hidden email]

[Geoff Bullen]

The join plugin doesn't work with the build pipeline plugin.  However you can achieve pretty much the same thing by wrapping up multiple jobs a build steps using the parametrised triggers plugin

In terms of passing artefacts down a pipeline of jobs, what I tend to do is pass a unique identifier along the pipeline that is used to publish the built artefact early on and to retrieve built artefacts later on.   You can use the copy artefact plugin to pass built artefacts around as well

On Thu, Feb 16, 2012 at 12:44 PM, R. Tyler Croy <[hidden email]> wrote:

On Wed, 15 Feb 2012, Jeff wrote:

> I'm trying to implement Jenkins with Maven as the build CI/CD tool since it
> does so well with the technologies we are using out of the box instead of
> GO that the larger .NET-based organization is  using.
>
> I've done Jenkins jobs for stand-alone projects but not jobs chained from
> dependent projects.
>
> Conceptually, I can see how dependent jobs are configured using the build
> pipeline plugin (though we only have one project so far).  What is unclear
> is how downstream jobs are configured to consume the
> artifacts and/or outputs produced upstream.


Plugins to check out:

 * Join plugin
 * Copy Artifacts plugin


Those might meet your needs, I use them both heavily to create longish
pipelines.

- R. Tyler Croy
--------------------------------------
   Code: http://github.com/rtyler
 Chatter: http://twitter.com/agentdero
         [hidden email]

[Geoff Bullen]

Jeff,

We have a pretty similar set up to what you describe.

We use a unique ID based on the build# of the first job in the pipeline.  This is used to publish to the maven repo in the 1st step and retrieve artifacts in the subsequent steps.

Geoff Bullen 

Centrum Systems 
Level 8, 5 Elizabeth Street 
Sydney 
  
0408 659 395 (m) 
02 9235 0516 (t) 
http://www.centrumsystems.com.au

On Thu, Feb 16, 2012 at 7:15 AM, Jeff <[hidden email]> wrote:

I'm trying to implement Jenkins with Maven as the build CI/CD tool since it does so well with the technologies we are using out of the box instead of GO that the larger .NET-based organization is  using.

 

I've done Jenkins jobs for stand-alone projects but not jobs chained from dependent projects.

 

Conceptually, I can see how dependent jobs are configured using the build pipeline plugin (though we only have one project so far).  What is unclear is how downstream jobs are configured to consume the artifacts and/or outputs produced upstream.

 

Off the top of my head, how would jobs be created/configured to perform the steps in the pipeline sequence below:

1. (JOB) Build shared library
1. Run Unit tests
2. Upload SNAPSHOT to Maven repository (e.g., Shared-1.6.3-SNAPSHOT-12.jar)
2. (JOB) Trigger build of WebApp
1. Build WebApp and include latest SNAPSHOT shared library (e.g., Shared-1.6.3-SNAPSHOT-12.jar)
2. Run Unit tests for WebApp
3. Upload WAR to repository (e.g., WebApp-1.4.2-SNAPSHOT-6.war)
3. (JOB) Deploy WebApp-1.4.2-SNAPSHOT-6.war
1. Grab WebApp-1.4.2-SNAPSHOT-6.war
2. Deploy to DEV server (tomcat container)
1. use DEV config.properties file (not sure how to switch out properties based on deploy env)
4. (JOB) Run Integration tests
5. (JOB) Deploy to STAGE environment
1. Grab (somehow) WebApp-1.4.2-SNAPSHOT-6.war
2. Deploy to ALL STAGE server (tomcat container) nodes (load balanced - two to 4 nodes)
1. use STAGE config.properties file (not sure how to switch out properties based on deploy env)
6. (JOB) Run Integration Tests (Validation only?)
7. (JOB) Run Load tests
8. (JOB) Deploy to LIVE

 

I've read through the Jenkins Build pipeline plugin page, but could really use some additional perspectives/resources, particularly with steps 3-8. 

 

Any tips/resources that I should read or insight into how you do this would be appreciated.

--

Jeff Vincent
[hidden email]
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

I ♥ DropBox !! 

[Nigel Magnay]

This is partially what the jenkins maven repository plugin [1] was built for -- each build is exposed as a separate build repository, so you don't need to go copying things around.

What we do is :

1) Install the plugin

2) Add the following to the jenkins settings.xml  [2]:

3) In the downstream project, add "-Pupstream" into the maven settings

4) Check the 'Define upstream maven repository' checkbox (build environment), and set the upstream build that you want to add as a build repository.

There is a bug to do with snapshot IDs in the current version - there is a patch that is in master, but I haven't had a chance to roll a release with it yet. I'd use 0.6.1 until it's fixed if you rely on SNAPSHOT artifacts..

[1] https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Maven+Repository+Server

[2]

<profile>

<id>upstream</id>

 <repositories>

             <repository>

                <id>upstream</id>

                <url>${env.Jenkins.Repository}</url>

                <releases>

                    <enabled>true</enabled>

                </releases>

                <snapshots>

                    <enabled>true</enabled>

                </snapshots>

             </repository>

    </repositories>

<pluginRepositories>

  <pluginRepository>

<id>upstream</id>

                <url>${env.Jenkins.Repository}</url>

                <releases>

                    <enabled>true</enabled>

                </releases>

                <snapshots>

                    <enabled>true</enabled>

                </snapshots>

  </pluginRepository>

</pluginRepositories>

</profile>

On Thu, Feb 16, 2012 at 1:44 AM, R. Tyler Croy <[hidden email]> wrote:

On Wed, 15 Feb 2012, Jeff wrote:

> I'm trying to implement Jenkins with Maven as the build CI/CD tool since it
> does so well with the technologies we are using out of the box instead of
> GO that the larger .NET-based organization is  using.
>
> I've done Jenkins jobs for stand-alone projects but not jobs chained from
> dependent projects.
>
> Conceptually, I can see how dependent jobs are configured using the build
> pipeline plugin (though we only have one project so far).  What is unclear
> is how downstream jobs are configured to consume the
> artifacts and/or outputs produced upstream.

Plugins to check out:

 * Join plugin
 * Copy Artifacts plugin


Those might meet your needs, I use them both heavily to create longish
pipelines.

- R. Tyler Croy
--------------------------------------
   Code: http://github.com/rtyler
 Chatter: http://twitter.com/agentdero
         [hidden email]

[Jeff Vincent]

Getting back to this...(BTW thanks for the responses)

 

How do you generate/control the unique ID?  Is the SVN repository revision a good ID to use?  Seems that it would be good to correlate the builds based on a known reference point.  That may not work though with multiple projects that are based on different SVN numbers .... hmm.

If you use Maven (and assuming you are also building web applications) it seems that the POM would be configured to do all the deploy's and possibly run the tests at any given point.  However, the project with the POM is part of the workspace of the upstream job and I don't think it would be good to copy the entire workspace for every job. 

 

Can multiple jobs share the same workspace?

 

On Mon, Feb 20, 2012 at 4:55 PM, Geoff Bullen <[hidden email]> wrote:

The join plugin doesn't work with the build pipeline plugin.  However you can achieve pretty much the same thing by wrapping up multiple jobs a build steps using the parametrised triggers plugin

In terms of passing artefacts down a pipeline of jobs, what I tend to do is pass a unique identifier along the pipeline that is used to publish the built artefact early on and to retrieve built artefacts later on.   You can use the copy artefact plugin to pass built artefacts around as well

On Thu, Feb 16, 2012 at 12:44 PM, R. Tyler Croy <[hidden email]> wrote:

On Wed, 15 Feb 2012, Jeff wrote:

> I'm trying to implement Jenkins with Maven as the build CI/CD tool since it
> does so well with the technologies we are using out of the box instead of
> GO that the larger .NET-based organization is  using.
>
> I've done Jenkins jobs for stand-alone projects but not jobs chained from
> dependent projects.
>
> Conceptually, I can see how dependent jobs are configured using the build
> pipeline plugin (though we only have one project so far).  What is unclear
> is how downstream jobs are configured to consume the
> artifacts and/or outputs produced upstream.


Plugins to check out:

 * Join plugin
 * Copy Artifacts plugin


Those might meet your needs, I use them both heavily to create longish
pipelines.

- R. Tyler Croy
--------------------------------------
   Code: http://github.com/rtyler
 Chatter: http://twitter.com/agentdero
         [hidden email]

--

Jeff Vincent
[hidden email]
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

I ♥ DropBox !! 

[Jeff Vincent]

I'm sure I'm missing something conceptually, but with a modular architecture, build artifacts may still be needed in the development environments, particularly if one team is producing a module consumed by another team and they want access to the SNAPSHOT build artifacts during development.

 

Doesn't the "Jenkins Maven Repository" plugin effectively isolate those artifacts to the build servers, thus preventing it's use outside the Jenkins world?

 

I'm curious also why the standard SNAPSHOT repository can't be used?  Again, I'm sure I'm missing something.

On Tue, Feb 21, 2012 at 3:28 AM, Nigel Magnay <[hidden email]> wrote:

This is partially what the jenkins maven repository plugin [1] was built for -- each build is exposed as a separate build repository, so you don't need to go copying things around.

What we do is :

1) Install the plugin

2) Add the following to the jenkins settings.xml  [2]:

3) In the downstream project, add "-Pupstream" into the maven settings

4) Check the 'Define upstream maven repository' checkbox (build environment), and set the upstream build that you want to add as a build repository.

There is a bug to do with snapshot IDs in the current version - there is a patch that is in master, but I haven't had a chance to roll a release with it yet. I'd use 0.6.1 until it's fixed if you rely on SNAPSHOT artifacts.

[1] https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Maven+Repository+Server

[2]

<profile>

<id>upstream</id>

 <repositories>

             <repository>

                <id>upstream</id>

                <url>${env.Jenkins.Repository}</url>

                <releases>

                    <enabled>true</enabled>

                </releases>

                <snapshots>

                    <enabled>true</enabled>

                </snapshots>

             </repository>

    </repositories>

<pluginRepositories>

  <pluginRepository>

<id>upstream</id>

                <url>${env.Jenkins.Repository}</url>

                <releases>

                    <enabled>true</enabled>

                </releases>

                <snapshots>

                    <enabled>true</enabled>

                </snapshots>

  </pluginRepository>

</pluginRepositories>

</profile>

On Thu, Feb 16, 2012 at 1:44 AM, R. Tyler Croy <[hidden email]> wrote:

On Wed, 15 Feb 2012, Jeff wrote:

> I'm trying to implement Jenkins with Maven as the build CI/CD tool since it
> does so well with the technologies we are using out of the box instead of
> GO that the larger .NET-based organization is  using.
>
> I've done Jenkins jobs for stand-alone projects but not jobs chained from
> dependent projects.
>
> Conceptually, I can see how dependent jobs are configured using the build
> pipeline plugin (though we only have one project so far).  What is unclear
> is how downstream jobs are configured to consume the
> artifacts and/or outputs produced upstream.

Plugins to check out:

 * Join plugin
 * Copy Artifacts plugin


Those might meet your needs, I use them both heavily to create longish
pipelines.

- R. Tyler Croy
--------------------------------------
   Code: http://github.com/rtyler
 Chatter: http://twitter.com/agentdero
         [hidden email]

--

Jeff Vincent
[hidden email]
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

I ♥ DropBox !!