Swapnil Bhartiya | 26 April 2018
Get insights from Jon Corbet on the state of Linux kernel development.
At the recent Embedded Linux Conference + OpenIoT Summit, I sat down with Jonathan Corbet, the founder and editor-in-chief of LWN to discuss a wide range of topics, including the annual Linux kernel report.
The annual Linux Kernel Development Report, released by The Linux Foundation is the evolution of work Corbet and Greg Kroah-Hartman had been doing independently for years. The goal of the report is to document various facets of kernel development, such as who is doing the work, what is the pace of the work, and which companies are supporting the work.
Linux kernel contributors
To learn more about the companies supporting Linux kernel development in particular, Corbet wrote a set of scripts with the release of kernel 2.6.20, to pull the information out of the kernel repository. That information helped Corbet associate contributions with employers, whenever possible.
When Corbet published a report based on these findings in LWN, it created a bit of a stir. “It was a surprise to everyone, including me, because there was still this image of free software in general and Linux in particular as being something produced by kids who haven’t moved out of their parents basements,” said Corbet.
He found that more than 70 percent of the code going into the kernel was coming from professional developers who were getting paid to do that work. “Since then things have changed and our numbers have gotten better. Today, over 90 percent of the code is coming from professional developers who are employed by some company to work on the kernel,” he said.
Corbet has been involved with the Linux kernel from a very early stage, so connecting the dots was not too difficult, even though not all developers use official company email accounts,
“In most cases, we know who is working for which company. Sometimes people contact us and say that their employer wants to ensure that they do get credit for the work they are doing in the kernel. Sometimes we just ask who they are working for,” said Corbet.
Corbet not only gathers valuable data about the Linux kernel, he also analyzes the data to see some patterns and trends. The biggest trend, over the years, has been a decline in the number of contributions coming from volunteers, which has decreased from 15 percent to 6 percent since the 2.6.20 release.
“There are times when we have worried about it because volunteers are often the people who are in the next round of paid developers. That’s often how you get into the community — by doing a little bit of stuff on your own time,” he said. Corbet did a bit of digging to see the employment status of people when their very first patch merged and their latest status. He found that at this point most of those people were already working for some company.
While it’s true there are fewer volunteer developers now, it could also be said that people don’t remain volunteers for very long because when their code gets merged into the kernel, companies tend to approach these developers and offer jobs. So, if your code shows up in the kernel, that’s a good resume to have.
What keeps Corbet awake at night
There has been a growing concern of late that the Linux kernel community is getting older. Looking at the top maintainers, for example, you can see a lot of people who have been involved since the 1990s.
“The upper cadre is definitely getting a little bit older, a little bit grayer. There is some truth to that and I think the concerns of that are not entirely overblown,” said Corbet. “A whole bunch of us managed to stumble into something good back in the ’90s, and we have stuck with it ever since because it’s been a great ride.”
That doesn’t mean new people are not coming in. A new kernel is released every 9 to 10 weeks. And, every new release sees contributions from more than 200 developers submitting their very first patch.
“We are bringing a lot of new people into the community,” Corbet said. “Maybe half of those 200 contributors will never contribute anything again. They had one thing they wanted to fix and then they moved on. But there are a lot many others who stick around and become long-term members of the community. Some of these worked their way into the subsystem maintainer positions. They will be replacing the older members as they retire.”
Corbet is not at all worried about the aging community as it has evolved into an “organic” body with continuous flow of fresh blood. It’s true that becoming a kernel developer is more demanding; you do have to work your way into it a little bit, but plenty of people are doing it.
“I’m not really worried about the future of our community because we are doing so well at attracting bright new developers,” said Corbet, “We have an influx rate that any other project would just love to have.”
However, he did admit that the community is showing increasing signs of stress at the maintainer level. “The number of maintainers is not scaling with a number of developers,” he said. However, he said, this problem is not unique to the kernel community; the whole free software community is facing this challenge.
Another concern for Corbet is the emergence of other kernels, such as Google’s Fuchsia. These kernels are being developed specifically to be permissively licensed, which allows them to be controlled by one or a very small number of companies. “Some of those kernels could push Linux aside in various subfields,” said Corbet. “I think some of the corporate community members have lost sight of what made Linux great and so successful. It could be useful for some companies in the short term, but I don’t think it’s going to be a good thing for anyone in the long term.”
Core needs
Corbet also noted another worrisome trend. Although many companies contribute to every kernel release, if you look closely you will see that a lot of these contributions are toward making their own hardware work great with Linux.
“It’s a great thing. We have been asking them to do it for years, but there is a whole lot of the kernel that everyone needs,” he said. There is the memory management subsystem. There’s the virtual filesystem layer. There are components of the kernel that are not tied to any single company’s hardware, and it’s harder to find companies willing to support them.
“Some of the companies that contribute to the most code to the kernel do not contribute to the core kernel at all,” said Corbet.
Corbet also worries about the lack of quality documentation and has himself initiated some efforts to improve the situation. “Nobody wants to pay for documentation,” he said. “There is nobody whose job it is to write documentation for the kernel, and it really shows in the quality. So, some of those areas I think are really going to hurt us going forward. We need to get better investment there.”
You can hear more from Jon Corbet, including insights on the recent Spectre and Meltdown issues, in his presentation from Embedded Linux Conference:
You can learn more about the Linux kernel development process in the complete annual report. Download the 2017 Linux Kernel Development Report now.