Getting One Job Invocation Per Git Commit

[Randall R Schulz]

Hi,

I'm using Jenkins to drive automated tests triggered by Git commits. Because of the duration of the tests and the frequency of commits (at certain times of the day), often multiple commits occur while the tests from an earlier commit are running.

I need to perform a full test run for each commit without this "batching" or "clumping" of multiple commits arriving during a test run.

How might I achieve that?


Randall Schulz

[Mark Waite]

From: Randall R Schulz <[hidden email]>
To: [hidden email]
Sent: Friday, July 8, 2011 1:24 PM
Subject: Getting One Job Invocation Per Git Commit

Hi,

I'm using Jenkins to drive automated tests triggered by Git commits. Because of the duration of the tests and the frequency of commits (at certain times of the day), often multiple commits occur while the tests from an earlier commit are running.

I need to perform a full test run for each commit without this "batching" or "clumping" of multiple commits arriving during a test run.

How might I achieve that?

If you have enough slave nodes to run the jobs in parallel, you could consider enabling the "Execute concurrent builds if necessary (beta)" check box on your jobs.  Then the builds could start as frequently as your polling interval (as soon as a change is detected).

That won't avoid batching of commits which arrive within the same minute (the minimum length of the polling interval), but it might keep many of your builds running as individual jobs.

Mark Waite

Randall Schulz

[Randall R Schulz]

That's not feasible for us since there's a single host that runs the tests and the app under test. It cannot concurrently check out multiple revisions of the app+test.


Randall Schulz

[Sami Tikka]

AFAIK the only option for you is to make the Jenkins job
parameterized. The parameter should be the git commit to build and
test. This is needed to prevent Jenkins from combining commits into
builds. Unfortunately Jenkins git plugin doesn't allow you to pass the
commit to build as a parameter, so you need to configure the job with
no SCM and check out the commit in your build step.

And because you cannot configure git in the job SCM settings, you need
to set up a hook script in the git repository that calls wget/curl to
trigger a build and pass the commit as a parameter.

Soon you will have a looooong build queue, so better tell your boss to
add some slaves to the next quarter budget :)

-- Sami

2011/7/9 Randall R Schulz <[hidden email]>:
> That's not feasible for us since there's a single host that runs the tests
> and the app under test. It cannot concurrently check out multiple revisions
> of the app+test.
>
>
> Randall Schulz
>

[R. Tyler Croy]

On Fri, 08 Jul 2011, Randall R Schulz wrote:

> Hi,
>
> I'm using Jenkins to drive automated tests triggered by Git commits. Because
> of the duration of the tests and the frequency of commits (at certain times
> of the day), often multiple commits occur while the tests from an earlier
> commit are running.
>
> I need to perform a full test run for each commit without this "batching" or
> "clumping" of multiple commits arriving during a test run.
>
> How might I achieve that?

I might be sounding like a broken record but I think this is acheivable with
Gerrit.

If you wanted to bypass the code review portion and just have each commit
tested, you could set up a hook on the Gerrit side to automatically submit
every commit after Jenkins has been testing it.


If you have time this wednesday, I can probably go over this workflow in my
Office Hours session: <https://wiki.jenkins-ci.org/display/JENKINS/Office+Hours>



- R. Tyler Croy
--------------------------------------
    Code: http://github.com/rtyler
 Chatter: http://identi.ca/agentdero
          http://twitter.com/agentdero

[Manuel Doninger]

On Mon, Jul 11, 2011 at 03:25, R. Tyler Croy <[hidden email]> wrote:
>
> If you have time this wednesday, I can probably go over this workflow in my
> Office Hours session: <https://wiki.jenkins-ci.org/display/JENKINS/Office+Hours>
>

Unfortunately i don't have the time on Wednesday to join the Office
Hours, but i recently setup such a workflow for a team in my company
(push to review, Jenkins builds the change and submit it, if
successful), so i can share my experiences with it later.

Manuel

[Andrew Bayer]

It also should be possible to write a new BuildChooser for the git plugin itself that implements this approach.

A.

On Sun, Jul 10, 2011 at 6:25 PM, R. Tyler Croy <[hidden email]> wrote:

On Fri, 08 Jul 2011, Randall R Schulz wrote:

> Hi,
>
> I'm using Jenkins to drive automated tests triggered by Git commits. Because
> of the duration of the tests and the frequency of commits (at certain times
> of the day), often multiple commits occur while the tests from an earlier
> commit are running.
>
> I need to perform a full test run for each commit without this "batching" or
> "clumping" of multiple commits arriving during a test run.
>
> How might I achieve that?

I might be sounding like a broken record but I think this is acheivable with
Gerrit.

If you wanted to bypass the code review portion and just have each commit
tested, you could set up a hook on the Gerrit side to automatically submit
every commit after Jenkins has been testing it.


If you have time this wednesday, I can probably go over this workflow in my
Office Hours session: <https://wiki.jenkins-ci.org/display/JENKINS/Office+Hours>



- R. Tyler Croy
--------------------------------------
   Code: http://github.com/rtyler
 Chatter: http://identi.ca/agentdero
         http://twitter.com/agentdero

[Randall R Schulz]

Here's the idea I was working with (though it seems not to work):

Take the "Last Built Revision" and "Checking out Revision" Git hashes from the Jenkins job run's console log and, search backward through the commit ancestry links until the commit immediately following the Last Built Revision is found and doing a "git reset -hard" to that revision.

Unfortunately, once the repository is reset to the older revision, the Git plug-in does not see that the remote repository head is ahead of the one on the Jenkins host. It does not trigger another job run until a subsequent commit occurs.


Randall Schulz

[wizard]

Gerrit will keep a local git repo, right?

On Jul 11, 9:25 am, "R. Tyler Croy" <[hidden email]> wrote:

> On Fri, 08 Jul 2011, Randall R Schulz wrote:
> > Hi,
>
> > I'm using Jenkins to drive automated tests triggered by Git commits. Because
> > of the duration of the tests and the frequency of commits (at certain times
> > of the day), often multiple commits occur while the tests from an earlier
> > commit are running.
>
> > I need to perform a full test run for each commit without this "batching" or
> > "clumping" of multiple commits arriving during a test run.
>
> > How might I achieve that?
>
> I might be sounding like a broken record but I think this is acheivable with
> Gerrit.
>
> If you wanted to bypass the code review portion and just have each commit
> tested, you could set up a hook on the Gerrit side to automatically submit
> every commit after Jenkins has been testing it.
>
> If you have time this wednesday, I can probably go over this workflow in my
> Office Hours session: <https://wiki.jenkins-ci.org/display/JENKINS/Office+Hours>
>
> - R. Tyler Croy
> --------------------------------------
>     Code:http://github.com/rtyler
>  Chatter:http://identi.ca/agentdero
>          http://twitter.com/agentdero
>
>  application_pgp-signature_part
> < 1KViewDownload

[Sebastian Bergmann-2]

Am 11.07.2011 19:57, schrieb Andrew Bayer:
> It also should be possible to write a new BuildChooser for the git plugin
> itself that implements this approach.

 That would be great as I have worked with development teams that also
 wanted to integrate every commit rather than "just" every push.

--
Sebastian Bergmann                    Co-Founder and Principal Consultant
http://sebastian-bergmann.de/                           http://thePHP.cc/

[R. Tyler Croy]

On Mon, 05 Sep 2011, wizard wrote:

> Gerrit will keep a local git repo, right?

Yes, Gerrit operates as a git repository which allows you to add it as a
git-remote(1) in order to push and pull to/from it.


In fact, in our use of the Gerrit Trigger plugin for Jenkins we clone directly
from Gerrit.

> On Jul 11, 9:25 am, "R. Tyler Croy" <[hidden email]> wrote:
> > On Fri, 08 Jul 2011, Randall R Schulz wrote:
> > > Hi,
> >
> > > I'm using Jenkins to drive automated tests triggered by Git commits. Because
> > > of the duration of the tests and the frequency of commits (at certain times
> > > of the day), often multiple commits occur while the tests from an earlier
> > > commit are running.
> >
> > > I need to perform a full test run for each commit without this "batching" or
> > > "clumping" of multiple commits arriving during a test run.
> >
> > > How might I achieve that?
> >
> > I might be sounding like a broken record but I think this is acheivable with
> > Gerrit.
> >
> > If you wanted to bypass the code review portion and just have each commit
> > tested, you could set up a hook on the Gerrit side to automatically submit
> > every commit after Jenkins has been testing it.
> >
> > If you have time this wednesday, I can probably go over this workflow in my
> > Office Hours session: <https://wiki.jenkins-ci.org/display/JENKINS/Office+Hours>

- R. Tyler Croy
--------------------------------------
    Code: http://github.com/rtyler
 Chatter: http://identi.ca/agentdero
          http://twitter.com/agentdero

[Mike Rooney-5]

A late reply, but as this thread appears on Google for trying to solve this problem, I figured I'd post my solution. It isn't great as you can't use the Git plugin, so a new BuildChooser would be superior, but it does result in one build per git commit across all branches.

First, modify your job to echo the last built commit to somewhere in the workspace (such as last_built_commit), or preferably throw it in S3 or something more persistent.

Now, use the ScriptTrigger plugin and create a script to see if the last_built_commit is still the latest, exiting 0 if not. Here's my example:

LAST_BUILT=`cat $WORKSPACE/last_built_commit` # or curl from S3 or wherever
NEXT_COMMIT=`cd $WORKSPACE/repo && git rev-list --all --remotes --reverse -n 100 | grep -A 1 $LAST_BUILT | tail -n 1`
echo $NEXT_COMMIT > $WORKSPACE/next_build_commit
# If the last built commit is not the newest, exit 0, meaning we need a new build.
[[ $LAST_BUILT != $NEXT_COMMIT ]]

Finally, handle git clone / pulling yourself in a build step: "if [ ! -e repo ]; then git clone repo.git; fi; cd repo && git fetch --all && git checkout `cat $WORKSPACE/next_build_commit`"

Ta-da! As far as I can tell, this works well, since a Jenkins restart or job failure will result in another attempt at that commit. Depending on where you write your last_built_commit you can control whether it will retry commits or skip failures.

I'd definitely love to see a BuildChooser that handled this, including for a branch specifier of **; that would be quite wonderful. However for now I figured this solution might help others, or maybe someone will point out a better way for me :)

- Mike

On Friday, July 8, 2011 3:24:59 PM UTC-4, Randall R Schulz wrote:

Hi,

I'm using Jenkins to drive automated tests triggered by Git commits. Because of the duration of the tests and the frequency of commits (at certain times of the day), often multiple commits occur while the tests from an earlier commit are running.

I need to perform a full test run for each commit without this "batching" or "clumping" of multiple commits arriving during a test run.

How might I achieve that?


Randall Schulz