Optional File Parameterization?

[Andrew Melo]

Hello, everyone-

We currently use jenkins to test commits that are in our Git
repository, but I'd like to add support for developers to test their
current workspace with jenkins before the commit makes it up. I added
some logic to the beginning of the build process to unpack a tarball
passed in with the file parameterization, which works great. However,
any existing processes that try to submit a job end up failing if they
don't pass in a file with the http request.

Is there a way to have jenkins ignore if a file isn't passed in
(perhaps making a zero-length file)? Or is there a better way to go
about this? (Our job configuration is super complex, so maintaining
two jobs with identical options modulo the build parameterization
would probably end up with them desynchronizing).

Thanks,
Andrew

[Sami Tikka]

You just have to change your build scripts to use the uploaded file if it exists in the workspace and do something else if it doesn't.

-- Sami

Andrew Melo <[hidden email]> kirjoitti 20.2.2012 kello 0.09:

> Hello, everyone-
>
> We currently use jenkins to test commits that are in our Git
> repository, but I'd like to add support for developers to test their
> current workspace with jenkins before the commit makes it up. I added
> some logic to the beginning of the build process to unpack a tarball
> passed in with the file parameterization, which works great. However,
> any existing processes that try to submit a job end up failing if they
> don't pass in a file with the http request.
>
> Is there a way to have jenkins ignore if a file isn't passed in
> (perhaps making a zero-length file)? Or is there a better way to go
> about this? (Our job configuration is super complex, so maintaining
> two jobs with identical options modulo the build parameterization
> would probably end up with them desynchronizing).
>
> Thanks,
> Andrew

[Andrew Melo]

Hey Sami,

On Mon, Feb 20, 2012 at 11:32 PM, Sami Tikka <[hidden email]> wrote:

You just have to change your build scripts to use the uploaded file if it exists in the workspace and do something else if it doesn't.

I would like to do that, but it doesn't seem to work. If I make the call with urllib (I'm using python), I get a 500 error back (so the job never fires). If I add a file to it, the job works fine. If I remove the file parameterization, it works. It's just when I try to submit a job and don't include the file that things blow up.

Unfortunately (well, fortunately in a way), I'm doing my qualifying exam tomorrow AM, so I can't dig out the backtrace, but one of the stack frames was in buildWithParameters. I can send that tomorrow once I'm done if that helps.

Thanks!

Andrew

 

-- Sami

Andrew Melo <[hidden email]> kirjoitti 20.2.2012 kello 0.09:

> Hello, everyone-
>
> We currently use jenkins to test commits that are in our Git
> repository, but I'd like to add support for developers to test their
> current workspace with jenkins before the commit makes it up. I added
> some logic to the beginning of the build process to unpack a tarball
> passed in with the file parameterization, which works great. However,
> any existing processes that try to submit a job end up failing if they
> don't pass in a file with the http request.
>
> Is there a way to have jenkins ignore if a file isn't passed in
> (perhaps making a zero-length file)? Or is there a better way to go
> about this? (Our job configuration is super complex, so maintaining
> two jobs with identical options modulo the build parameterization
> would probably end up with them desynchronizing).
>
> Thanks,
> Andrew

--
--
Andrew Melo

[Sami Tikka]

Sorry, but you lost me. How does urllib come into the picture?

You configure the job to take file parameter and name it foo.tmp. Then your build script checks if file foo.tmp exists. If it does, rename it to foo and start testing it. If it doesn't, build foo and start testing it.

-- Sami

Andrew Melo <[hidden email]> kirjoitti 21.2.2012 kello 7.36:

Hey Sami,

On Mon, Feb 20, 2012 at 11:32 PM, Sami Tikka <[hidden email]> wrote:

You just have to change your build scripts to use the uploaded file if it exists in the workspace and do something else if it doesn't.

I would like to do that, but it doesn't seem to work. If I make the call with urllib (I'm using python), I get a 500 error back (so the job never fires). If I add a file to it, the job works fine. If I remove the file parameterization, it works. It's just when I try to submit a job and don't include the file that things blow up.

Unfortunately (well, fortunately in a way), I'm doing my qualifying exam tomorrow AM, so I can't dig out the backtrace, but one of the stack frames was in buildWithParameters. I can send that tomorrow once I'm done if that helps.

Thanks!

Andrew

 

-- Sami

Andrew Melo <[hidden email]> kirjoitti 20.2.2012 kello 0.09:

> Hello, everyone-
>
> We currently use jenkins to test commits that are in our Git
> repository, but I'd like to add support for developers to test their
> current workspace with jenkins before the commit makes it up. I added
> some logic to the beginning of the build process to unpack a tarball
> passed in with the file parameterization, which works great. However,
> any existing processes that try to submit a job end up failing if they
> don't pass in a file with the http request.
>
> Is there a way to have jenkins ignore if a file isn't passed in
> (perhaps making a zero-length file)? Or is there a better way to go
> about this? (Our job configuration is super complex, so maintaining
> two jobs with identical options modulo the build parameterization
> would probably end up with them desynchronizing).
>
> Thanks,
> Andrew

--
--
Andrew Melo

[Andrew Melo]

On Mon, Feb 20, 2012 at 11:42 PM, Sami Tikka <[hidden email]> wrote:

Sorry, but you lost me. How does urllib come into the picture?

Sorry, I should've clarified.

We have a number of different tools that trigger builds with jenkins through the REST interface. For instance, when someone posts a pull request to github, a script makes a new branch in git and tells jenkins to build against it. This works great, we have a build parameterization that tells jenkins what branch to build, so I can just make a request to http://jenkins.server/job/something/something/buildWithParameters?branch=test_branch9999 and jenkins queues the build. We have a few different build parameters that accept strings to let us configure different things about the job (i.e. someone can request only an oracle or mysql backend if they're testing specific functionality and want to save time). If all the parameters aren't there, jenkins doesn't complain, it just takes the defaults.

If I put in a file parameter into the configuration and try to trigger a build, and no file is uploaded, jenkins throws a 500 error instead of simply not putting a file into the job workdir. I would think the expected behavior would be to just not put a file in the workdir.

Thanks again,

Andrew

 

You configure the job to take file parameter and name it foo.tmp. Then your build script checks if file foo.tmp exists. If it does, rename it to foo and start testing it. If it doesn't, build foo and start testing it.

-- Sami

Andrew Melo <[hidden email]> kirjoitti 21.2.2012 kello 7.36:

Hey Sami,

On Mon, Feb 20, 2012 at 11:32 PM, Sami Tikka <[hidden email]> wrote:

You just have to change your build scripts to use the uploaded file if it exists in the workspace and do something else if it doesn't.

I would like to do that, but it doesn't seem to work. If I make the call with urllib (I'm using python), I get a 500 error back (so the job never fires). If I add a file to it, the job works fine. If I remove the file parameterization, it works. It's just when I try to submit a job and don't include the file that things blow up.

Unfortunately (well, fortunately in a way), I'm doing my qualifying exam tomorrow AM, so I can't dig out the backtrace, but one of the stack frames was in buildWithParameters. I can send that tomorrow once I'm done if that helps.

Thanks!

Andrew

 

-- Sami

Andrew Melo <[hidden email]> kirjoitti 20.2.2012 kello 0.09:

> Hello, everyone-
>
> We currently use jenkins to test commits that are in our Git
> repository, but I'd like to add support for developers to test their
> current workspace with jenkins before the commit makes it up. I added
> some logic to the beginning of the build process to unpack a tarball
> passed in with the file parameterization, which works great. However,
> any existing processes that try to submit a job end up failing if they
> don't pass in a file with the http request.
>
> Is there a way to have jenkins ignore if a file isn't passed in
> (perhaps making a zero-length file)? Or is there a better way to go
> about this? (Our job configuration is super complex, so maintaining
> two jobs with identical options modulo the build parameterization
> would probably end up with them desynchronizing).
>
> Thanks,
> Andrew

--
--
Andrew Melo

--
--
Andrew Melo

[Sami Tikka]

Funny... I just created a test job with one file parameter and
triggered without specifying a file. The job ran fine and there was no
file in the workspace.

I'm running Jenkins 1.451 on Mac OS X 10.7. My Jenkins is not wrapped
in any servlet container.

-- Sami

2012/2/21 Andrew Melo <[hidden email]>:

>
>
> On Mon, Feb 20, 2012 at 11:42 PM, Sami Tikka <[hidden email]> wrote:
>>
>> Sorry, but you lost me. How does urllib come into the picture?
>
>
> Sorry, I should've clarified.
>
> We have a number of different tools that trigger builds with jenkins through
> the REST interface. For instance, when someone posts a pull request to
> github, a script makes a new branch in git and tells jenkins to build
> against it. This works great, we have a build parameterization that tells
> jenkins what branch to build, so I can just make a request to
> http://jenkins.server/job/something/something/buildWithParameters?branch=test_branch9999
> and jenkins queues the build. We have a few different build parameters that
> accept strings to let us configure different things about the job (i.e.
> someone can request only an oracle or mysql backend if they're testing
> specific functionality and want to save time). If all the parameters aren't
> there, jenkins doesn't complain, it just takes the defaults.
>
> If I put in a file parameter into the configuration and try to trigger a
> build, and no file is uploaded, jenkins throws a 500 error instead of simply
> not putting a file into the job workdir. I would think the expected behavior
> would be to just not put a file in the workdir.
>
> Thanks again,
> Andrew
>
>
>
>>
>>
>> You configure the job to take file parameter and name it foo.tmp. Then
>> your build script checks if file foo.tmp exists. If it does, rename it to
>> foo and start testing it. If it doesn't, build foo and start testing it.
>>
>> -- Sami
>>
>> Andrew Melo <[hidden email]> kirjoitti 21.2.2012 kello 7.36:
>>
>> Hey Sami,
>>
>> On Mon, Feb 20, 2012 at 11:32 PM, Sami Tikka <[hidden email]> wrote:
>>>
>>> You just have to change your build scripts to use the uploaded file if it
>>> exists in the workspace and do something else if it doesn't.
>>
>>
>> I would like to do that, but it doesn't seem to work. If I make the call
>> with urllib (I'm using python), I get a 500 error back (so the job never
>> fires). If I add a file to it, the job works fine. If I remove the file
>> parameterization, it works. It's just when I try to submit a job and don't
>> include the file that things blow up.
>>
>> Unfortunately (well, fortunately in a way), I'm doing my qualifying exam
>> tomorrow AM, so I can't dig out the backtrace, but one of the stack frames
>> was in buildWithParameters. I can send that tomorrow once I'm done if that
>> helps.
>>
>> Thanks!
>> Andrew
>>
>>>
>>>
>>> -- Sami
>>>
>>> Andrew Melo <[hidden email]> kirjoitti 20.2.2012 kello 0.09:
>>>
>>> > Hello, everyone-
>>> >
>>> > We currently use jenkins to test commits that are in our Git
>>> > repository, but I'd like to add support for developers to test their
>>> > current workspace with jenkins before the commit makes it up. I added
>>> > some logic to the beginning of the build process to unpack a tarball
>>> > passed in with the file parameterization, which works great. However,
>>> > any existing processes that try to submit a job end up failing if they
>>> > don't pass in a file with the http request.
>>> >
>>> > Is there a way to have jenkins ignore if a file isn't passed in
>>> > (perhaps making a zero-length file)? Or is there a better way to go
>>> > about this? (Our job configuration is super complex, so maintaining
>>> > two jobs with identical options modulo the build parameterization
>>> > would probably end up with them desynchronizing).
>>> >
>>> > Thanks,
>>> > Andrew
>>
>>
>>
>>
>> --
>> --
>> Andrew Melo
>>
>
>
>
> --
> --
> Andrew Melo
>

[Andrew Melo]

On Wed, Feb 22, 2012 at 2:35 PM, Sami Tikka <[hidden email]> wrote:

Funny... I just created a test job with one file parameter and
triggered without specifying a file. The job ran fine and there was no
file in the workspace.

I'm running Jenkins 1.451 on Mac OS X 10.7. My Jenkins is not wrapped
in any servlet container.

Hey Sami-

Were you submitting using a browser? This is what I get on the command line when I try to submit a job with a file parameterization (note, it works fine and builds properly when there's no file parameterization). When submitting a form on a browser and there is no file selected, it will silently do the equivalent of submitting an empty file. When triggering the REST interface directly, that doesn't happen and Jenkins barfs on the input. The stacktrace is below.

Thanks,

Andrew

[dmwm] /afs/cern.ch/user/m/meloam > curl http://dmwm.cern.ch:8080/job/sandbox-melo-matrixtest/buildWithParameters

<html><head><title>Error 500</title></head><body bgcolor="#ffffff"><h1>Status Code: 500</h1>Exception: <br>Stacktrace: <pre>java.lang.UnsupportedOperationException

at hudson.model.FileParameterDefinition.createValue(FileParameterDefinition.java:72)

at hudson.model.ParametersDefinitionProperty.buildWithParameters(ParametersDefinitionProperty.java:141)

at hudson.model.AbstractProject.doBuildWithParameters(AbstractProject.java:1623)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:616)

at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:282)

at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:149)

at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:88)

at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)

at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)

at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:563)

at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648)

at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)

at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)

at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:563)

at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648)

at org.kohsuke.stapler.Stapler.invoke(Stapler.java:477)

at org.kohsuke.stapler.Stapler.service(Stapler.java:159)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)

at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)

at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)

at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)

at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)

at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)

at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)

at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:74)

at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)

at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)

at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:91)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)

at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)

at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:61)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)

at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)

at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)

at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)

at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:244)

at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)

at java.lang.Thread.run(Thread.java:636)

</pre><br><hr size="1" width="90%"><i>Generated by Winstone Servlet Engine v0.9.10 at Wed Feb 22 22:19:35 CET 2012</i></body></html>[dmwm] /afs/cern.ch/user/m/meloam > 

 

-- Sami

2012/2/21 Andrew Melo <[hidden email]>:

>
>
> On Mon, Feb 20, 2012 at 11:42 PM, Sami Tikka <[hidden email]> wrote:
>>
>> Sorry, but you lost me. How does urllib come into the picture?
>
>
> Sorry, I should've clarified.
>
> We have a number of different tools that trigger builds with jenkins through
> the REST interface. For instance, when someone posts a pull request to
> github, a script makes a new branch in git and tells jenkins to build
> against it. This works great, we have a build parameterization that tells
> jenkins what branch to build, so I can just make a request to
> http://jenkins.server/job/something/something/buildWithParameters?branch=test_branch9999
> and jenkins queues the build. We have a few different build parameters that
> accept strings to let us configure different things about the job (i.e.
> someone can request only an oracle or mysql backend if they're testing
> specific functionality and want to save time). If all the parameters aren't
> there, jenkins doesn't complain, it just takes the defaults.
>
> If I put in a file parameter into the configuration and try to trigger a
> build, and no file is uploaded, jenkins throws a 500 error instead of simply
> not putting a file into the job workdir. I would think the expected behavior
> would be to just not put a file in the workdir.
>
> Thanks again,
> Andrew
>
>
>
>>
>>
>> You configure the job to take file parameter and name it foo.tmp. Then
>> your build script checks if file foo.tmp exists. If it does, rename it to
>> foo and start testing it. If it doesn't, build foo and start testing it.
>>
>> -- Sami
>>
>> Andrew Melo <[hidden email]> kirjoitti 21.2.2012 kello 7.36:
>>
>> Hey Sami,
>>
>> On Mon, Feb 20, 2012 at 11:32 PM, Sami Tikka <[hidden email]> wrote:
>>>
>>> You just have to change your build scripts to use the uploaded file if it
>>> exists in the workspace and do something else if it doesn't.
>>
>>
>> I would like to do that, but it doesn't seem to work. If I make the call
>> with urllib (I'm using python), I get a 500 error back (so the job never
>> fires). If I add a file to it, the job works fine. If I remove the file
>> parameterization, it works. It's just when I try to submit a job and don't
>> include the file that things blow up.
>>
>> Unfortunately (well, fortunately in a way), I'm doing my qualifying exam
>> tomorrow AM, so I can't dig out the backtrace, but one of the stack frames
>> was in buildWithParameters. I can send that tomorrow once I'm done if that
>> helps.
>>
>> Thanks!
>> Andrew
>>
>>>
>>>
>>> -- Sami
>>>
>>> Andrew Melo <[hidden email]> kirjoitti 20.2.2012 kello 0.09:
>>>
>>> > Hello, everyone-
>>> >
>>> > We currently use jenkins to test commits that are in our Git
>>> > repository, but I'd like to add support for developers to test their
>>> > current workspace with jenkins before the commit makes it up. I added
>>> > some logic to the beginning of the build process to unpack a tarball
>>> > passed in with the file parameterization, which works great. However,
>>> > any existing processes that try to submit a job end up failing if they
>>> > don't pass in a file with the http request.
>>> >
>>> > Is there a way to have jenkins ignore if a file isn't passed in
>>> > (perhaps making a zero-length file)? Or is there a better way to go
>>> > about this? (Our job configuration is super complex, so maintaining
>>> > two jobs with identical options modulo the build parameterization
>>> > would probably end up with them desynchronizing).
>>> >
>>> > Thanks,
>>> > Andrew
>>
>>
>>
>>
>> --
>> --
>> Andrew Melo
>>
>
>
>
> --
> --
> Andrew Melo
>

--
--
Andrew Melo

[Les Mikesell]

On Wed, Feb 22, 2012 at 3:29 PM, Andrew Melo <[hidden email]> wrote:

>
>
> On Wed, Feb 22, 2012 at 2:35 PM, Sami Tikka <[hidden email]> wrote:
>>
>> Funny... I just created a test job with one file parameter and
>> triggered without specifying a file. The job ran fine and there was no
>> file in the workspace.
>>
>> I'm running Jenkins 1.451 on Mac OS X 10.7. My Jenkins is not wrapped
>> in any servlet container.
>
>
> Hey Sami-
>
> Were you submitting using a browser? This is what I get on the command line
> when I try to submit a job with a file parameterization (note, it works fine
> and builds properly when there's no file parameterization). When submitting
> a form on a browser and there is no file selected, it will silently do the
> equivalent of submitting an empty file. When triggering the REST interface
> directly, that doesn't happen and Jenkins barfs on the input. The stacktrace
> is below.

So can you make whatever you are using to hit the REST interface
submit an empty file when you don't have a real one?

--
  Les Mikesell
     [hidden email]

[Andrew Melo]

On Wed, Feb 22, 2012 at 3:36 PM, Les Mikesell <[hidden email]> wrote:

On Wed, Feb 22, 2012 at 3:29 PM, Andrew Melo <[hidden email]> wrote:
>
>
> On Wed, Feb 22, 2012 at 2:35 PM, Sami Tikka <[hidden email]> wrote:
>>
>> Funny... I just created a test job with one file parameter and
>> triggered without specifying a file. The job ran fine and there was no
>> file in the workspace.
>>
>> I'm running Jenkins 1.451 on Mac OS X 10.7. My Jenkins is not wrapped
>> in any servlet container.
>
>
> Hey Sami-
>
> Were you submitting using a browser? This is what I get on the command line
> when I try to submit a job with a file parameterization (note, it works fine
> and builds properly when there's no file parameterization). When submitting
> a form on a browser and there is no file selected, it will silently do the
> equivalent of submitting an empty file. When triggering the REST interface
> directly, that doesn't happen and Jenkins barfs on the input. The stacktrace
> is below.

So can you make whatever you are using to hit the REST interface
submit an empty file when you don't have a real one?

That would be the easiest option. But we already have a bunch of different scripts that are hitting it. It's weird that jenkins will let you ignore posting all other types of parameterizations (the sandbox-melo-matrixtest job has a bunch of parameters, for instance), but will blow up on missing a file. I'm terrible at java, but i'm going to walk through the stacktrace and see if I can't find what's barfing.

Thanks,

Andrew

 

--
 Les Mikesell
    [hidden email]

--
--
Andrew Melo