Guidelines/best practices building embedded linux

[Bue Petersen]

Hi,

anyone having experience building an embedded linux setup with Jenkins?

Currently I build on a linux host using a shell script getting
repositories and setting up environment, and the uses bitbake and
OpenEmbedded to build a toolchain and an image.

I have searched for some time, but can not really find any best
practices and recommendations, except using Jenkins to mimic my shell
script or just run it.
This being also a little tricky, as environment isn't always
configured as expected.

In advance, thanks for tips and replies,
best regards
Bue Petersen


PS: Also sent an mail to dev-list, asking about a plugin for the
bitbake and OpenEmbedded combo.

[Sami Tikka]

I do not have real experience on how to build embedded linux with Jenkins, but I did once work for a project embedding FreeBSD, so it is probably close enough.

I don't think there's anything tricky about it. If you cannot set up an environment for building and make the build with one or two commands, you have a problem, but the problem is not Jenkins.

If you have your source tree set up such that you can check it out and run something like: "sudo make build-dep && make && make tests" and your build will work, things are good and it is easy to make Jenkins execute it. If you have to spend a lot of time to do complex setup for the build machine, taking Jenkins into use will be difficult.

Working on the environment setup will pay back later. I have seen it many times when a new guy joins the team or a new build machine needs to be set up or ... many times.

If the environment setup cannot be easily scripted, maybe you can capsulate it into a virtual machine? So instead of running a script to set up libraries and compilers you run a script that sets up a vm which already has the needed environment. Take a look at Vagrant. http://vagrantup.com (There is a Jenkins plugin for vagrant vms.)

-- Sami

Bue Petersen <[hidden email]> kirjoitti 27.7.2012 kello 17.46:

> Hi,
>
> anyone having experience building an embedded linux setup with Jenkins?
>
> Currently I build on a linux host using a shell script getting
> repositories and setting up environment, and the uses bitbake and
> OpenEmbedded to build a toolchain and an image.
>
> I have searched for some time, but can not really find any best
> practices and recommendations, except using Jenkins to mimic my shell
> script or just run it.
> This being also a little tricky, as environment isn't always
> configured as expected.
>
> In advance, thanks for tips and replies,
> best regards
> Bue Petersen
>
>
> PS: Also sent an mail to dev-list, asking about a plugin for the
> bitbake and OpenEmbedded combo.

[Rich Pixley]

I've been building OE using my own builders for years.  I've only just
started with jenkins but it doesn't seem like there's any real trick.

You just have to understand that jenkins doesn't really do much in the
way of work space management so hosed builds mean you'll have to clear
the workspace and start all over from scratch.

It would appear that jenkins is primarily based around a bunch of much,
much, much smaller builds than something like oe so there are likely to
be some scaling issues.  Eg, I'm finding that stuff like clearing the
workspace take long enough that jenkins thinks they've failed.

Or maybe I'm not understanding the question.

--rich

On 20120727 07:46, Bue Petersen wrote:

> Hi,
>
> anyone having experience building an embedded linux setup with Jenkins?
>
> Currently I build on a linux host using a shell script getting
> repositories and setting up environment, and the uses bitbake and
> OpenEmbedded to build a toolchain and an image.
>
> I have searched for some time, but can not really find any best
> practices and recommendations, except using Jenkins to mimic my shell
> script or just run it.
> This being also a little tricky, as environment isn't always
> configured as expected.
>
> In advance, thanks for tips and replies,
> best regards
> Bue Petersen
>
>
> PS: Also sent an mail to dev-list, asking about a plugin for the
> bitbake and OpenEmbedded combo.

[Sami Tikka]

Rich Pixley <[hidden email]> kirjoitti 28.7.2012 kello 0.28:

> You just have to understand that jenkins doesn't really do much in the way of work space management so hosed builds mean you'll have to clear the workspace and start all over from scratch.

Jenkins does have the option of clearing the workspace at the beginning of the build. It is usually there in the SCM (= version control) settings and is usually off.

>
> It would appear that jenkins is primarily based around a bunch of much, much, much smaller builds than something like oe so there are likely to be some scaling issues.  Eg, I'm finding that stuff like clearing the workspace take long enough that jenkins thinks they've failed.

Interesting. Could you elaborate? I wasn't aware of any time limits in Jenkins builds. I have a Jenkins which runs 24 hour performance tests and Jenkins has never failed it because it runs too long.

-- Sami

[Rich Pixley]

On 7/29/12 09:03 , Sami Tikka wrote:
> Rich Pixley <[hidden email]> kirjoitti 28.7.2012 kello 0.28:
>> You just have to understand that jenkins doesn't really do much in the way of work space management so hosed builds mean you'll have to clear the workspace and start all over from scratch.
> Jenkins does have the option of clearing the workspace at the beginning of the build. It is usually there in the SCM (= version control) settings and is usually off.
I haven't found that yet.  Core jenkins apparently doesn't have that
ability, although it would seem to be core functionality.

And from my perspective, manual clearing is extremely minimal support.
>> It would appear that jenkins is primarily based around a bunch of much, much, much smaller builds than something like oe so there are likely to be some scaling issues.  Eg, I'm finding that stuff like clearing the workspace take long enough that jenkins thinks they've failed.
> Interesting. Could you elaborate? I wasn't aware of any time limits in Jenkins builds. I have a Jenkins which runs 24 hour performance tests and Jenkins has never failed it because it runs too long.
I'm not sure what else to say.  I click "clear workspace', it runs for a
while, I can see the disk space being cleared, and then it fails with a
time out.

A typical oe build directory can run 30gig per device and you may well
build for 5 devices.  That's a lot of disk space which takes time to
deallocate.

--rich

[Sami Tikka]

Rich Pixley <[hidden email]> kirjoitti 30.7.2012 kello 20.13:

> On 7/29/12 09:03 , Sami Tikka wrote:
>> Rich Pixley <[hidden email]> kirjoitti 28.7.2012 kello 0.28:
>>> You just have to understand that jenkins doesn't really do much in the way of work space management so hosed builds mean you'll have to clear the workspace and start all over from scratch.
>> Jenkins does have the option of clearing the workspace at the beginning of the build. It is usually there in the SCM (= version control) settings and is usually off.
> I haven't found that yet.  Core jenkins apparently doesn't have that ability, although it would seem to be core functionality.

Well, it depends. If you can clean up the workspace by only deleting files that are not controlled by version control system, you can save a lot of time in checkout of large repositories. But then you need help from the version control system => Jenkins implements workspace cleaning in the SCM plugins.

If you prefer to simply nuke the whole workspace before the build starts, you can do that too, but unfortunately you need https://wiki.jenkins-ci.org/display/JENKINS/Workspace+Cleanup+Plugin for it.

If you have a large repository, you probably do not want to start every build with a checkout from scratch.

If you use Subversion, click on the "Subversion" radiobutton under "Source code management" (in a job configuration.) Then click on the Advanced button. (There are many Advanced buttons on the page. Click on the Advanced button in the Source Code Management section.) for "Checkout Strategy", choose "Emulate clean checkout by first deleting unversioned/ignored files, then 'svn update'".

If you use Git, click on "Git" radiobutton under "Source code management". Then click on the 2nd Advanced button of the Source Code Management section (yes, Jenkins could use a helping hand from a professional UX designer...) Check the box "Clean after checkout".

If you use something else, maybe someone else can tell you how to do it.


>
> And from my perspective, manual clearing is extremely minimal support.
>>> It would appear that jenkins is primarily based around a bunch of much, much, much smaller builds than something like oe so there are likely to be some scaling issues.  Eg, I'm finding that stuff like clearing the workspace take long enough that jenkins thinks they've failed.
>> Interesting. Could you elaborate? I wasn't aware of any time limits in Jenkins builds. I have a Jenkins which runs 24 hour performance tests and Jenkins has never failed it because it runs too long.
> I'm not sure what else to say.  I click "clear workspace', it runs for a while, I can see the disk space being cleared, and then it fails with a time out.

Aaah! That's the way you do it. Yes, I have seen that time out on large workspaces. I use it so seldom I haven't bothered even making a bug report about it.

> A typical oe build directory can run 30gig per device and you may well build for 5 devices.  That's a lot of disk space which takes time to deallocate.

I hope the SCM plugin for your version control system has support for intelligent workspace cleaning.

-- Sami

[Rich Pixley]

I'm not just thinking of using the source code control system to do the cleaning here.

I'm coming to jenkins after developing my own automated builder based on btrfs snapshots.  With btrfs snapshots, it's possible to both keep an old build, (all artifacts intact), lying around while also creating a fresh incremental build on top of it.  The two builds then share files in the file system so much as they are common and the file system takes care of which ones stay around.

With this approach, you can also automate recovery from failures.  When an incremental fails, you simply start the next incremental from the last successful snapshot.  This also allows for "pre-commit" checking in the sense that changes can be built, (as an incremental on the previous build), and if successful, the changes can be committed and the work space becomes the next successful snapshot.  If it fails, the next build starts from the previously successful snapshot.

This doesn't have the static disk allocation of jenkins, (one job, maximum one workspace), but you can use the entire disk as a big ring buffer - remove the oldest builds until there's enough space for the next build, then build.

This is orders of magnitude faster than using the source code control system to do the cleaning.  It's also significantly faster in all work flows and doesn't require any source code control system hooks at all.  It's completely source code control system agnostic.

If you put btrfs on root, it also protects your build host by effectively running the build in a disk jail.  Builds can muck about with the state of the build host and each build will still be completely independent with a zero time setup cost.

--rich

[Bue Petersen]

On Mon, Jul 30, 2012 at 11:23 PM, Rich Pixley <[hidden email]> wrote:

> I'm not just thinking of using the source code control system to do the
> cleaning here.
>
> I'm coming to jenkins after developing my own automated builder based on
> btrfs snapshots.  With btrfs snapshots, it's possible to both keep an old
> build, (all artifacts intact), lying around while also creating a fresh
> incremental build on top of it.  The two builds then share files in the file
> system so much as they are common and the file system takes care of which
> ones stay around.
>
> With this approach, you can also automate recovery from failures.  When an
> incremental fails, you simply start the next incremental from the last
> successful snapshot.  This also allows for "pre-commit" checking in the
> sense that changes can be built, (as an incremental on the previous build),
> and if successful, the changes can be committed and the work space becomes
> the next successful snapshot.  If it fails, the next build starts from the
> previously successful snapshot.
>
> This doesn't have the static disk allocation of jenkins, (one job, maximum
> one workspace), but you can use the entire disk as a big ring buffer -
> remove the oldest builds until there's enough space for the next build, then
> build.
>
> This is orders of magnitude faster than using the source code control system
> to do the cleaning.  It's also significantly faster in all work flows and
> doesn't require any source code control system hooks at all.  It's
> completely source code control system agnostic.
>
> If you put btrfs on root, it also protects your build host by effectively
> running the build in a disk jail.  Builds can muck about with the state of
> the build host and each build will still be completely independent with a
> zero time setup cost.
>
> --rich

Hi, I just want to write a follow up on all your interesting
consideration on my question and this discussion.

I think we pretty much follow what some of you suggest, as our build
now follows this flow:
- we build inside directory stuff/build and have a local.conf inside
stuff/build/local.conf
- we optain bitbake from internet/cache stuff-directory
- we optain OE from internet/cache to stuff-directory
- we set up the environment eg. path to bitbake etc.

This follows the OE guide http://www.openembedded.org/wiki/Getting_started

The above steps is incorported in a shell-script, but I consider
making the SCM (git) set it up, using submodules, so a checkout/clone
makes a folder-structure that we can build in.
A bitbake target is triggerede by a build.sh-script that is also used
in the jenkins job as the only build-step.
Jenkins can then follow-up on warnings and exit status codes.

I think I will look into your consideration on cleaning, as this still
is an issue for us. Both regarding the bitbake cache and the
build-target folder.

But thanks for sharing your ideas and setups!
Best regards,
Bue Petersen

[Sami Tikka]

That sure sounds nice. I frequently have problems with my slaves getting messed up and I have to clean them up. Do you have anything in github elsewhere?

-- Sami

Rich Pixley <[hidden email]> kirjoitti 31.7.2012 kello 0.23:

> I'm not just thinking of using the source code control system to do the cleaning here.
>
> I'm coming to jenkins after developing my own automated builder based on btrfs snapshots.  With btrfs snapshots, it's possible to both keep an old build, (all artifacts intact), lying around while also creating a fresh incremental build on top of it.  The two builds then share files in the file system so much as they are common and the file system takes care of which ones stay around.
>
> With this approach, you can also automate recovery from failures.  When an incremental fails, you simply start the next incremental from the last successful snapshot.  This also allows for "pre-commit" checking in the sense that changes can be built, (as an incremental on the previous build), and if successful, the changes can be committed and the work space becomes the next successful snapshot.  If it fails, the next build starts from the previously successful snapshot.
>
> This doesn't have the static disk allocation of jenkins, (one job, maximum one workspace), but you can use the entire disk as a big ring buffer - remove the oldest builds until there's enough space for the next build, then build.
>
> This is orders of magnitude faster than using the source code control system to do the cleaning.  It's also significantly faster in all work flows and doesn't require any source code control system hooks at all.  It's completely source code control system agnostic.
>
> If you put btrfs on root, it also protects your build host by effectively running the build in a disk jail.  Builds can muck about with the state of the build host and each build will still be completely independent with a zero time setup cost.
>
> --rich

[Rich Pixley]

I'm working on getting it opened and released.

--rich

On 7/31/12 11:19 , Sami Tikka wrote:

> That sure sounds nice. I frequently have problems with my slaves getting messed up and I have to clean them up. Do you have anything in github elsewhere?
>
> -- Sami
>
> Rich Pixley <[hidden email]> kirjoitti 31.7.2012 kello 0.23:
>
>> I'm not just thinking of using the source code control system to do the cleaning here.
>>
>> I'm coming to jenkins after developing my own automated builder based on btrfs snapshots.  With btrfs snapshots, it's possible to both keep an old build, (all artifacts intact), lying around while also creating a fresh incremental build on top of it.  The two builds then share files in the file system so much as they are common and the file system takes care of which ones stay around.
>>
>> With this approach, you can also automate recovery from failures.  When an incremental fails, you simply start the next incremental from the last successful snapshot.  This also allows for "pre-commit" checking in the sense that changes can be built, (as an incremental on the previous build), and if successful, the changes can be committed and the work space becomes the next successful snapshot.  If it fails, the next build starts from the previously successful snapshot.
>>
>> This doesn't have the static disk allocation of jenkins, (one job, maximum one workspace), but you can use the entire disk as a big ring buffer - remove the oldest builds until there's enough space for the next build, then build.
>>
>> This is orders of magnitude faster than using the source code control system to do the cleaning.  It's also significantly faster in all work flows and doesn't require any source code control system hooks at all.  It's completely source code control system agnostic.
>>
>> If you put btrfs on root, it also protects your build host by effectively running the build in a disk jail.  Builds can muck about with the state of the build host and each build will still be completely independent with a zero time setup cost.
>>
>> --rich

[J Arrizza]

We've been using Hudson for Linux (from scratch) and are now working on getting Android to build (almost done!).

A few differences between the two projects might be applicable to you:

- in Linux we used a few tarballs for common components. They were untarred during the build and then the compiled from source after that. In Android we went straight from source for the whole thing (100K+ files). The mercurial clone or status takes a long time in Android. So based on that one data point, I recommend you use tarballs for those parts of the distribution that don't change much. 

- We have a need to build from a clean workspace. Since the clone took so long we used the mercurial "hg clean" which seems to take less time then the workspace delete followed by the hg clone.

- We have an application that we install into the distribution during the build. We wanted our application project to be source controlled separately from the Linux build. So we had to split the Linux (and Android) builds into two parts. One for the compilation, one for the   "packaging", i.e. the step where we merged our application and data files into the Linux/Android distribution. It wasn't trivial to do this for either project, but much more complicated for Android. I recommend you start working out how that will work for you. There were a lot of details to get right for it to all work correctly.

- We are (currently) stuck with an old PC with not much horsepower. As soon as I can, I am going to swap it out with something with a very fast disk (SSD!) and with multiple cores so I can start using "make -j". So, I recommend you get big iron up front.

- We use Ubuntu so I created a set of installation scripts to install all prerequisite packages for Linux. Very useful for configuring slaves to be identical. I started with bash but eventually converted to perl. Much easier to work with.

John

On Fri, Jul 27, 2012 at 7:46 AM, Bue Petersen <[hidden email]> wrote:

Hi,

anyone having experience building an embedded linux setup with Jenkins?

Currently I build on a linux host using a shell script getting
repositories and setting up environment, and the uses bitbake and
OpenEmbedded to build a toolchain and an image.

I have searched for some time, but can not really find any best
practices and recommendations, except using Jenkins to mimic my shell
script or just run it.
This being also a little tricky, as environment isn't always
configured as expected.

In advance, thanks for tips and replies,
best regards
Bue Petersen


PS: Also sent an mail to dev-list, asking about a plugin for the
bitbake and OpenEmbedded combo.