For the past couple of days I’ve been attending the first ever sprint held by the Linaro Tools and Infrastructure team. The sprint itself lasts all week but I arranged to be there for just Tuesday and Wednesday to find out what they had planned and to join up with a few guests.

This event is slightly different from what we typically do at Ubuntu sprints. The team hasn’t been together for that long so there was equal focus on both information sharing and collaboration. To that end, mornings were for presentations and discussion, afternoons for hands-on hacking.

A link to the whole schedule is here; for my account of what happened whilst I was there, read on.

Tuesday started with a discussion on image building by Michael Hudson. Often lively and full of good information, the key steps to producing the Linaro images was the focus of conversation. Two launchpad specifications cover this effort, the first, arm-m-image-building-tool concerns improving live-helper to accommodate Linaro’s needs, the second, arm-m-image-building-console aims to produce a web based front-end for the image building tools.

Zygmunt Krynicki’s project entitled Validation Dashboard was next (slides can be found here). An ambitious project to produce a front end visual representation for various testing tools; this effort is of great importance as validation and correctness is paramount to the software releases Linaro makes.

Another requirement for Linaro is the ability to branch and rebuild parts of the Ubuntu archive. Linaro benefits from the great effort that is put into the Ubuntu archive but as Linaro’s needs are at times, different, being able to diverge without affecting Ubuntu itself is important. This effort is being lead by James Westby.

Tuesday morning finished with Paul Larson’s testsuite and profiler presentation. Closely related to Zygmunt’s dashboard effort, Paul is working on combing the various testsuites and profiles into one big validation effort.

At this point the guys from Code Sourcery joined us to discuss toolchain’s and related tools. Will Deacon, Matthias Klose and David Rusling joined the conversation and many points were ironed out. Linaro will benefit from the great work Code Sourcery do around GCC and related tools, producing what we think will be the industry standard tool chain for ARM based devices. A lofty goal but everyone behind the effort is confident.

Talk on tools and toolchains continued to be the focus for the rest of Tuesday and Wednesday. Code Sourcery went away with a pile of work items and the Linaro members todo lists were equally full. There was just enough time Wednesday to sit down with Dave Rusling to formally sign off the technical requirements.

As one Linux Weekly News author put it,

the list of planned achievements for the five months before the release is quite ambitious … even completing a big chunk of it would be quite an accomplishment

We believe we can do it and with the great individuals such as those at the sprint this week, the 10.11 release will be something special.

“Any intelligent fool can make things bigger and more complex… It takes a touch of genius – and a lot of courage to move in the opposite direction”.
- Albert Einstein

A little history

Perhaps a little known fact is that well over 15 billion chips have been shipped using ARM’s technologies, 1.3 billion in the last quarter of 2009 alone and as the internet goes truely mobile this is set to sky rocket. ARM have sold over 600 processor licenses to more than 200 companies and 12 of the top 20 semiconductor companies use their technologies. ARM based devices really are everywhere. In contrast, Intel have shipped just 3.3 billion to date. Renowned for their low power consumption and high performance, ARM based devices really do seem to be the holy grail of computing but why is it that Intel continues to dominates the desktop, laptop, netbook and server market?

Fragmentation

Developing for ARM devices isn’t without its problems. There are a lot of companies all working independently on producing their own products, often duplicating effort. Kernels, boot loaders, and to a lesser extent middleware are being worked on in isolation with little in the way of standards and a common direction. This is scary for those who are used to working in the Intel world where one kernel and one boot loader will pretty much work on all compatible devices. To really push ARM devices into the standard spaces Intel currently enjoy’s, something needs to be done.

Ubuntu’s Linux on ARM initative

Canonical, creators of the renowned Ubuntu distribution, and ARM saw the need to rally around an effort to produce a modern, full-featured Linux distribution tailored for ARM devices. Together, starting in 2008, Canonical and ARM took on the task of bring Ubuntu to ARM platforms. Release 9.04, codenamed “Jaunty Jackalope” was the first Ubuntu release supporting Freescales iMX51 and Marvell’s Dove platforms. This distribution was further refined in the 9.10 Karmic Koala release and 10.04, Lucid Lynx even added support for a third ARM platform, TI’s Beagle Board. With a completely redesigned user interface, a web based office solution and many more improvements, Lucid is a magnificent release; however there is still a need for a more consolidated effort. This is where Linaro fits into the story.

What is Linaro?

Linaro is an initiative undertaken by ARM, Canonical and partners with the task of improving the state of the whole Linux on ARM ecosphere. It brings together the vast talents of the open source community and ARM’s wealth of experience in the electronics industry to work on key and game-changing projects. It will work in the various upstreams where possible and provide engineering, technical and guidance support for a wide and diverse set of problem area’s. Linaro will not just help other projects, there will also be a clear set of deliverables which will culminate Linaro’s efforts into regular 6 monthly engineering releases, starting this November.

Release Objectives

So this is the area which gets me the most excited. As the release manager for Linaro I am responsible for making sure each and every release captures exactly what the essence of Linaro is, consolidation, collaboration, improvement and robustness. Each of these qualities are goals for the Linaro releases and to that end our first release, 10.11, will contain all of these in abundance.

Building upon the already successful Linux on ARM effort, Linaro will utilize Canonical’s Launchpad framework including bug management, code hosting and blueprints.

Just some of the high-level highlights are below. I’ll leave the other Linaro team members to blog the details about their teams efforts later.

1. Help standardize the industry on common kernel versions and features.
2. Improve debugging and performance analysis at the kernel level.
3. Bring power management and performance improvements including boot speed reduction.
4. Promote and implement device-tree’s on ARM hardware.
5. Provide test ‘heads’, whole vertical software stacks and distributions, to show what can be done on top of Linaro.
6. Explore integrating telephony right into the distribution.
7. Improve the state of graphics acceleration.
8. Provide QA and validation harnesses to ensure anything built with Linaro is of the highest standard.
9. Offer performance analysis and suggestions on how to improve.
10. Supply a whole host of development, archive and image management, and distribution creation and customization tools.

And much, much more. A full list of blueprints which capture what we are trying to achieve can be found on the Linaro wiki.

Get Involved

For more information on the Linaro initiative please see the official website and the wiki. We have mailing lists and a Launchpad project page and I invite you all to come and participate in this exciting and ground-breaking venture.

What next?

So whats next? Well, the famous words of Albert Einstein that opened this post are most relevant here. We are the Genius’s trying to make ARM development both easier and simpler. So with courage and a bit of luck, Linaro will succeed in bring Linux and ARM to a whole new level.

A new user interface

One of the most obvious changes is the user interface. The ARM version of Ubuntu’s previous release, Karmic Koala, booted to the default Ubuntu desktop. For some this was fine but typically today’s ARM devices tend to be different. At present, they tend to have smaller screens, less resources and little in the way of graphics acceleration. To overcome some of these limitations in the x86 netbook world, the netbook-launcher user interface was created. Based on Clutter, netbook-launcher could not run on the ARM devices Ubuntu was targeting due to a lack of 3D acceleration. Enter netbook-launcher-efl, a 2D version of the x86 netbook interface written using EFL packages.

Read more about the 2D EFL based launcher.

Faster Live CD boots

Booting a Live CD is something that most new Ubuntu users do (and many existing users too). Its often their first experience of an Ubuntu release and should give a good impression. Well, on some ARM hardware, booting this Live CD image took over 3 minutes, not exactly the impression we would hope. So investigations happened into what was causing this slowness. In the end the final boot time was reduced by around 35% on all Ubuntu Live images, not just ARM ones.

Read more about the Live CD boot time improvements.

Web Office and Web Mail integration

Open Office on a resource limited platform isn’t the greatest experience and to make matters worse, on the ARM architecture there are issues building it correctly. A new way of viewing, editing and saving office documents was needed and for the Lucid cycle a web-based solution was integrated into the desktop called webservice-office-zoho.

Read more about the web office integration.

Similarly, Evolution could be considered too heavy-weight for ARM device needs. A solution was implemented to enable integration with several online mail providers.

Read more about the web mail integration.

Optimized Tool Chain Defaults

This release includes a complete archive rebuild using more modern tool chain defaults that the latest ARM hardware can take advantage of. As of Lucid Lynx, packages are built using Thumb-2 to reduce code size and improve performance, NEON for accelerate multimedia and signal processing, and are optimized for ARMv7A based chips. Although this means that some older hardware will not work with the latest Ubuntu release it does mean that the images perform much better on modern hardware.

Other Improvements

Much bug fixing went on this cycle. The fail to build list (FTBFS), a list of packages that fail to build on a given architecture, was a focal point of activity. For the first time ever, the number of packages that failed to build on ARM from the main archive was zero (apart from libx86 which refuses to leave the build queue for ARM due to a bug), a great achievement.

The Chromium browser now works on ARM, rootstock, the tool to build ARM rootfs tarballs gained a gui frontend, we added support for the very popular OMAP platform (beagle board) and many small improvements were implemented, making this the best Ubuntu ARM release ever.

We here at Canonical are very proud of the Lucid Lynx on ARM and are extremely excited at what future releases will bring.

Vendors are trying to work around it, especially as there is the expectation of a ramp-up in the availability of ARM based hardware. Super long-life netbooks, low powered touch based computers, and even a flurry of smaller embedded devices are forecast to hit the market this year, many of which will be based on the Linux operating system. Ubuntu would be a great match for this.

Ubuntu and ARM

Ubuntu runs very well on some ARM based platforms and there is a sustained effort to make it work more ubiquitously across many more. To that end our goal is to have Ubuntu running on any ARM based device (as long as there is hardware available). A lofty goal but one which we would like to see happen.

So what can we do about the 3D graphics licensing issue? Legally not very much. The companies that own the IP (Intellectual Property) rights to these drivers often want large licensing fees for their technology. This is a model for single product lines (take the Nokia N900 for instance) but for Ubuntu where we are targeting a more broad approach, this isn’t ideal.

So when you buy your new, ARM based netbook that has an obscene amount of battery life and you just want to install the 3D clutter based, wonderfully rich UI that Ubuntu Netbook Edition offers, what do you do?

Well Ubuntu recognizes this problem and as part of the Lucid Lynx release there is an effort to bring a similarly wonderfully rich UI to non-3D-accelerated hardware.

The new 2D EFL based Launcher

Above you can see the default UI for Ubuntu’s ARM based releases starting from Lucid (10.04). It’s a direct clone of the UI found in the 9.10 Karmic release on i386 although this one is based on EFL (Enlightenment Foundation Libraries) meaning that its fast on non-accelerated platforms. If there is 3D hardware available it can use that but it works perfectly fine without.

Another great thing about the 2D launcher is that isn’t not restricted to ARM hardware only, in fact if you have Lucid installed now, getting the launcher couldn’t be simpler. At the command prompt just type the following (make sure you have the universe repository enabled):

sudo apt-get install netbook-launcher-efl

and voila, your UI switches to the new launcher. Of course a simple:

sudo apt-get remove netbook-launcher-efl

will remove it if you decide its not what you want.

Beyond Netbook Launcher

Another of the great things about this launcher, as apposed to the 3D launcher shipped with Karmic, is that its extremely theme-able. The theme file is contained in:

/usr/share/netbook-launcher-efl/data/themes/default.edj

Theme files use the edje declarative layout format. By changing this file you can completely change the way the UI looks. For example, see the alternate UI screenshot above, both are based on the same code, the only difference is that they have a different theme file.

So if you have ARM based hardware but no 3D acceleration, fear not, you can get the same great user experience that your i386 cousins have in Ubuntu Netbook Remix.

How do you boot a live cd session?

The first thing to do was find out why it was slow. There are a few ways to do this but I chose to first use simple time-stamping methods and afterwards, the much prettier bootchart package.

A bit of background on how the live cd session is booted. There are two broad steps in the process of booting a live cd, the first is setting up the environment ready for the session, and the second, you guessed it, is actually booting into the session. Initial hunches were that the first step, setting up the session, was the major cause of slowness so investigations started there.

Casper

Setting up the session is the responsibility of a project called casper. Casper is a set of scripts that are run on boot to do such things as unpack the initial filesystem, add a dummy user, setup languages and keyboard layouts and so forth. Its mainly written in perl.

The time-stamping stage of investigates confirmed that casper was indeed slower than it should be. I’ll skip ahead to the bootchart part of the story as thats much more visually interesting.

Casper before

Above you can see the casper section of the live cd boot process. I’ve highlighted the bits that immediately stand out. The overall boot was 3 minutes and 15 seconds with casper responsible for around 2 minutes of that.

The highlighted bits seemed to have one thing in common, they all interacted with the debconf database. From the debconf programmers tutorial document:

“debconf is a backend database, with a frontend that talks to it and presents an interface to the user. There can be many different types of frontends, from plain text to a web frontend. The frontend also talks to a special config script in the control section of a debian package, and it can talk to postinst scripts and other scripts as well, all using a special protocol. These scripts tell the frontend what values they need from the database, and the frontend asks the user questions to get those values if they aren’t set.”

So communicating with the debconf database was slowing the boot. Initiating the call, sending the data and receiving a response, was taking up to 4 seconds at a time and when there are many of these calls, they all soon add up.

After many a head-scratching moment it was decided that the best way to solve this would be to initiate the communication once and keep it open so that when debconf was needed, the overhead in setting it up was removed. The implementation details are all in the code history but the results are much easier to show.

Casper after

With a couple of other tweaks besides the debconf one, the boot is now down to 1 minute 53 seconds and casper takes just under 50 seconds of that. There is more room for improvement, pre-generating a default locale (although which locale that would be is a tough choice), pre-generating fonts and looking into SSL, but for now, this is a big win.

The bootcharts above are for the ARM based iMX51 device made by Freescale which is only just beginning to proliferate onto the market. Intel/AMD based machines show an equivalent speed-up.

Conclusion

So there you have it. The next time you boot into a Lucid live cd session it shouldn’t take quite so long, and now you know why.