The 2.6.32 Linux kernel
Last week I released the 220.127.116.11 kernel
and said it would be the last one of this series that I was releasing.
Given that this was one of the most successful kernel series out there,
by number of users, I figured it was worth a brief history of how this
came to be, and what I have learned from it.
Stable kernel releases
The first stable kernel release, under the "new" model of development,
happened with the 18.104.22.168 release, way back on March 4, 2005, almost 7
years ago to today day. Since then, the stable series has been very
successfully released for every kernel that Linus releases, following
the rules outlined in the file Documentation/stable _ kernel _ rules.txt
in the kernel tree.
For more details about how the stable kernel series came to be about,
and how it all works, see the presentation I gave a few years ago at the
Tokyo LinuxCon conference, 2010.
Originally both me and Chris Wright started releasing the stable
kernels, but a few years back, Chris retired from this, and it's been
just me ever since.
One important part of the stable releases was the idea of "throw it on
the floor" when Linus would release a new kernel. This worked really
well for the community users of the stable kernel, but how about the
"enterprise" Linux distros?
My day job at the time was at Novell/SUSE, and we were about to release
a product based on the 2.6.16 kernel, SLE10. As engineers, we were
staring down the long hallway of pain we were going to have to endure in
order to maintain this specific kernel version for the next 5-7 years
for our customers. A large part of maintaining an enterprise kernel is
digging through the upstream kernel.org bugfixes and backporting them to
the kernel where needed. As this was essentially the same thing that I
was already doing as part of my upstream stable kernel work, and I had
all of the scripts and workflow already created, I decided to try to see
how well I could maintain the 2.6.16 kernel for a longer period of time
than just the 2-3 months I currently was.
So, with the 22.214.171.124 kernel release, done on March 27, 2006, I started
the 2.6.16 kernel series that would go on to live longer than any kernel
release I had ever managed. In the end, I put up with that kernel for
855 days, longer than I had ever imagined possible.
I don't remember if I publicly announced this anywhere, but as time
went on, I just kept the 2.6.16 kernel alive, backporting patches to it,
and doing releases. This base on which to do the SLE10 releases
proved invaluable to me and to the users of the SLE10 product. They
gained a lot more bugfixes than we had ever found previously, and it
made managing the kernel easier in that we now had a base that was
"known good" to constantly build upon. To me, this experiment paid off
very well, and others noticed, with community users of the 2.6.16 kernel
relying on it for their systems as well, for they too wanted a longterm
kernel for some machines that could be supported by the community.
Over time, I noticed that as the kernel.org releases moved on, the
amount of patches that actually applied to the older kernel dropped off
more and more, with a steep drop-off somewhere around 2 years.
The work that went into keeping this kernel alive, and the experience
gained from keeping it working for such a long time for enterprise
customers, made me write up a proposal about The Future of Enterprise
in June of 2007. Also, at the same time, I gave a talk at the Linux
Kernel summit (I think it was that year) about this same topic. I
pushed hard at that meeting, saying something like "Living at one kernel
version for the lifetime of an enterprise Linux release is wrong."
Despite my goal of getting rid of enterprise Linux distro kernels
sticking at a single release, my job went on, and work started at Novell
to plan for the SLE11 SP1 release.
The Cabal meets in secret
The kernel developer community is a very tight-knit one. Despite
working for companies that compete with each other, we work together
daily through email, make fun of each other on IRC, and drink beer
together in different cities around the world every few months at
During a few of these meetings, in mid to late 2009, the kernel
developers working for all of the various distros quickly figured out
that the timeline for the next major releases of a number of products
appeared to be lining up to happen all near the same timeframe.
Because of the success of the 2.6.16 kernel, and how it worked to
provide a solid base for a distro to work off of for a long time, we all
agreed, informally, to push for a specific kernel release within our
communities/companies that I would then maintain in the kernel.org
community in the same way I had done for the 2.6.16 kernel release.
We all drifted back to our companies, and planted the seeds that maybe
something like the 2.6.32 kernel would be a nice one to do our product
on. This planting worked so well, I had to refrain from fits of
laughter in one meeting where a project manager got up and said, "We
decided that the 2.6.32 kernel would be the best for our product, what
does engineering think about this?"
This successfully cumulated in the release of SLE11 SP1, Debian
"Squeeze", RHEL 6, Oracle Linux 6, and Ubuntu 10.4 LTS, all based on the
Hacking the business models of these different and competing groups, to
coordinate on this specific kernel, was one of the (previously) unsung
successes of how the community really can achieve remarkable things if
they decide to do it.
We did it, now to get down to work keeping this kernel alive.
Success is almost the death of us
Because it was a widespread "secret" that the 2.6.32 kernel was going to
be the base of all of these different enterprise releases, a lot of
development groups all started rushing to complete things in time for
this kernel release. In the end, when 2.6.32 was released on December
2, 2009, it was the third largest kernel ever developed, by number of
changes, with 10,998 of them resulting in the rate of 5.46 patches per
hour being applied to it over its release cycle (2.6.29 and 2.6.30 had
beaten it with 11,718 and 11,989 patches respectively.)
Because of this fixed deadline, it seemed that a number of areas of the
kernel were a bit more "unstable" than normal, but the release and
testing process of the enterprise distros soon shook all of that mess
out by the time they were finally released a number of months later.
The worries that we had chosen wrong and rushed things, was now gone.
How long does it live?
With the 2.6.32 kernel being the base of these longterm enterprise
distros, it was originally guessed that it would be hanging around for
many many years to come. But, my old argument about how moving a kernel
forward for an enterprise distro finally sunk in for 2 of the 3 major
players. Both Oracle Linux and SLES 11, in their latest releases these
few months, have moved to the 3.0 kernel as the base of them, despite
leaving almost all other parts of the distro alone. They did this to
take advantage of the better support for hardware, newer features, newer
filesystems, and the hundreds of thousands of different changes that has
happened in the kernel.org releases since way back in 2009.
Debian is planning on their next stable release, and it will be on a
newer kernel, and Ubuntu of course has long moved off of the
2.6.32-based kernel for their (monthly?) releases.
So, with the big users of the 2.6.32 kernel moving on, I've decided to
no longer support the 2.6.32 kernel. That's not to say it's going to be
dead now. The ever capable Willy Tarreau (the 2.4 kernel maintainer)
is going to be keeping it alive, slowly, with a very reduced release
schedule as time sees fit.
What about RHEL?
Yes, I know, what about RHEL? It turned out that Red Hat wasn't
interested in taking advantage of the stable kernel releases of the
2.6.32 kernel that I made. They have their reasons, and they are valid,
I'm not trying to say they aren't, but it turned out that these releases
I did really didn't provide them much value from their normal operation
of how they maintain their kernel. They are sticking with the 2.6.32
kernel for their RHEL 6 product for the near future as far as I can
tell, and it works quite well for them and their customers, with one of
the largest installed base of any distro out there. I think they pick
and choose pieces of the 2.6.32-stable releases as needed, but due to
them only releasing a large "one giant patch" for their kernel package,
it's a bit hard to tell what they are really doing there.
So, how did the 2.6.32 kernel stack up?
It lived (in my hands) for 823 days (only the 2.6.16 kernel lived longer
at 855 days). It ended up containing 3,349 patches that matched up with
patches in Linus's kernel tree, the most of any stable kernel release by
far (2.6.16 only had 991 patches, the next closest was 2.6.27 with 1,596
patches, 3.0 already has 1448 patches after 133 days, so it might end up
being the largest eventually.)
The 2.6.32 kernel was the basis of all of the enterprise distros of the
time, still running, and will still supported by the major enterprise
Linux vendors for many years in the future, so it will live on.
The lessons learned in maintaining the 2.6.32 kernel have cumulated in
the proposal I made last year for the longterm kernel.
With the proposed longterm kernel releases being chosen once a year,
it's obvious that I'm not giving up on the model of maintaining specific
kernel versions for longer than a few months. But the Linux user and
developer community has spread way beyond the enterprise Linux
distributions over the past few years. They are no longer the primary
consumer of the kernel, and it's obvious that the embedded market also
needs this type of support. Because of that, I'm working with the LTSI
project, to provide a base that a
wider range of distros and products can base themselves on.
Will we ever see all of the major distros ever end up basing themselves
on the same kernel version? The odds are now less than before, given
their shifting release cycles (some faster, some slower than before),
and the move forward to newer kernel releases instead of trying to patch
together a single kernel for many many years, a move I strongly support.
But, you never know. If you see a group of kernel developers sitting in
the corner at a conference, laughing and drinking beer, perhaps they are
really plotting how to convince their managers that the idea was really
theirs on what kernel they should be picking for the next product...
I would personally like to thank the Debian kernel developers,
specifically Ben Hutchings, Maximilian Attems, Dann Frazier, Bastian
Blank, and Moritz Muehlenhoff. They went above and beyond what any
"normal" developer would have done, ferreting patches out of the
kernel.org releases and the different vendor kernels and bug tracking
systems, backporting them to the 2.6.32 kernel, testing, and then
forwarding them on to me. Their dedication to their user community is
amazing for such a "volunteer" group of developers.
I firmly believe that without their help, the 2.6.32 kernel would not
have been the success that it was. The users of Red Hat and SuSE
products owe them a great debt.
Buy them a beer the next time you see them, they more than deserve it.
[Update May 17, 2013]
Above I mentioned the 2.6.16 kernel, while forgetting to acknowledge the
wonderful work that Adrian Bunk did in maintaining it, well after I let
it go. I gave it up on the 126.96.36.199 release, on July 17, 2006, and
Adrian took it and ran with it for much longer, releasing 35 kernels,
for two more years, until July of 2008 when it was retired.
My apologies, I never ment to ignore Adrian's work.