The one-millionth commit: The search for the lucky Linux kernel contributor

This week has been “a week of millions” for the Linux Foundation, with our announcement that over 1 million people have taken our free Introduction to Linux course. As part of the research for our recently published 2020 Linux Kernel History Report, the Kernel Project itself determined that it had surpassed one million code commits. Here is how we established the identity of this lucky Kernel Project contributor. 

Methodology:

The historical repo of BitKeeper (converted to Git) has 63,428 commits. We then found the merge at which Linus Torvalds’ repo has at least 936,572 commits (his repo has at least this many commits).

At commit 92c59e126b21fd212195358a0d296e787e444087 the repo had 936,456 commits (116 shy of the million)

>git checkout 92c59e126b21fd212195358a0d296e787e444087

>git log --oneline | wc

 936456 7483489 62991540


The next merge 2f3fbfdaf77f3ac417d0511fac221f76af79f6fc passed that number, with 937,105

> git checkout 2f3fbfdaf77f3ac417d0511fac221f76af79f6fc

> git log --oneline | wc

 937105 7489456 63037625

So on merge 2f3fbfdaf77f3ac417d0511fac221f76af79f6fc Linus’ repo passed the 1M mark (to be precise, 1,000,533 including BitKeeper commits):

commit 2f3fbfdaf77f3ac417d0511fac221f76af79f6fc 92c59e126b21fd212195358a0d296e787e444087 f510ca05271b6f71bd532fe743b39f628110223f (HEAD)

Merge: 92c59e126b21 f510ca05271b

Author: Linus Torvalds <torvalds@linux-foundation.org>

Date:   Mon Aug 3 19:19:34 2020 -0700


Merge tag 'arm-dt-5.9' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc

At this point, we can simply list the 936,572nd commit in the log:

>git log --oneline | tail -936572 | head -1

85b23fbc7d88 x86/cpufeatures: Add enumeration for SERIALIZE instruction

And the committer is…

git log -1 85b23fbc7d88

commit 85b23fbc7d88f8c6e3951721802d7845bc39663d

Author: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>

Date:   Sun Jul 26 21:31:29 2020 -0700

    x86/cpufeatures: Add enumeration for SERIALIZE instruction

Ricardo’s momentous commit to the Kernel was to add enumeration support for the SERIALIZE instruction, supported in Intel’s forthcoming Sapphire Rapids and Alder Lake microarchitectures for their 10-nanometer server and workstation chips. Ricardo is a software engineer who has been working on Linux feature support for Intel’s microprocessors for 12 years as part of the company’s CPU enabling team.

For more about Intel Corporation’s Ricardo Neri, the one-millionth Linux Kernel code committer, please read and watch our interview, conducted by Swapnil Bhartiya on Linux.com.

All: I want to formally congratulate the Linux kernel project for earning a gold badge!! You can see their details here:

https://bestpractices.coreinfrastructure.org/en/projects/34

The Linux kernel has been close for a while. The final one they completed was to add some HTTP hardening headers to key websites.

Of course, a gold badge doesn’t mean that there are no vulnerabilities, or that it’s impossible to improve their development processes. Perfection is rare in this life. But it *does* mean that they’ve implemented a large number of good practices to keep the project sustainable, to counter vulnerabilities from entering their software, and to address vulnerabilities when they are found. The Linux kernel project takes many steps to do this, and it’s good to see.

The Linux kernel joins some of the few other gold applications, such as the Zephyr project, who have been at gold for a while. You can see the current gold holders here:

https://bestpractices.coreinfrastructure.org/en/projects?gteq=300

My thanks to Greg Kroah-Hartman, who spearheaded getting the badge “over the finish line.” Thank you for your effort.

I hope that this result will help inspire other projects to pursue — and earn — a gold badge. Of course, the real goal isn’t a badge — the real goal is to make our software much more secure. But I think it’s clear that good practices can help make our software more secure, and we want to praise & encourage projects to have good practices.

David A. Wheeler

Director of Open Source Supply Chain Security, The Linux Foundation

FINAL 

FOR IMMEDIATE RELEASE

SAN FRANCISCO, October 15, 2019The Linux Foundation, the nonprofit organization enabling mass innovation through open source, today announced enrollment is open for a new, free, online course – A Beginner’s Guide to Linux Kernel Development

Linux, created by Linus Torvalds 26 years ago as a “hobby project”, has become the world’s largest and most pervasive open source software project in the history of computing. The Linux kernel is the largest component of the Linux ecosystem, and is charged with managing the hardware, running user programs, and maintaining the security and integrity of the whole system. Over 13,000 kernel developers from around the world have contributed to the Linux kernel. It is a 24 hour a day, seven days a week, 365 day a year development process that results in a new release once every 9-10 weeks, along with several stable and extended stable releases. At all times, new development and current release integration cycles run in parallel.

New developers often struggle to find a way to productively engage with the Linux community. Developed by Shuah Khan, a Linux Foundation Fellow and an experienced Linux kernel developer, maintainer, and contributor, A Beginner’s Guide to Linux Kernel Development is designed for anyone interested in becoming a Linux Kernel developer and contributor. The course aims to ease the Linux Kernel Mentorship application process. It also serves as a resource for developers from companies and communities that might not be able to take advantage of the mentorship program, and want to learn kernel development on their own; as well as a resource for experienced engineers new to open source and upstream kernel development that are tasked with working with the kernel community.

“In a nutshell, my motivation is to empower new and experienced engineers to learn to work with the kernel community and become productive members of the community. I am hoping this course will help demystify the kernel development process by making it easily accessible to developers from diverse backgrounds”, says Shuah Khan.

According to Greg Kroah-Hartman, Linux Foundation Fellow and Linux Kernel Maintainer, “Shuah has created a wonderful asset for new developers interested in contributing to the Linux kernel.  This course is unique in that it covers both the technical aspects of submitting code as well as how the community works in order to have your code accepted easier”.

The course introduces developers to the Linux kernel development process and teaches the explicit and implicit “rules of the road”. It covers configuring a development system, git basics, writing kernel patches, testing patches, writing commit logs, sending patches, and working on feedback from the kernel community. 

The course will teach the following:

  • Select and configure your development system
  • Overview of Linux Kernel repositories and releases
  • Git basics – checking out kernel repositories and working with them
  • Build your first kernel and install it
  • Linux kernel Contributor Covenant Code of Conduct
  • Linux Kernel Enforcement Statement
  • Write kernel patches and test them
  • How to communicate with the kernel community (do’s and don’ts)
  • Who and how to send patches (checkpatch.pl and get_maintainers.pl)
  • Re-work patches and act on feedback from reviewers.

A Beginner’s Guide to Linux Kernel Development is available at no cost, for up to one year. 

About The Linux Foundation

The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and industry adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage.

Linux is a registered trademark of Linus Torvalds.

# # #

Media Contact:

Clyde Seepersad

The Linux Foundation

404-964-6973

cseepersad@linuxfoundation.org

The New CIP SLTS Kernel Expands the Support Architecture to include ARM64

SAN FRANCISCO –  February 25, 2019 – The Civil Infrastructure Platform (CIP) Project, which enables long-term management of infrastructure systems through a base layer of industrial grade open source software components, tools and methods, today announced the release of the Super Long Term Support (SLTS) Kernel. The new kernel expands architectural support for the 64-bit Arm® Cortex, which enables developers to use it in a variety of use cases including building automation, machine learning and artificial intelligence.

As requirements for reliability, connectivity and feature-richness increase, the amount of software needed to implement and maintain civil infrastructure systems has grown to unprecedented levels. These systems are the foundation for modern society and are ubiquitously responsible for supervision, control, and management of infrastructure for communities and industries across the globe. With these demands, there are unique challenges for safety, security and reliability requirements as updates are needed on an ongoing basis.

Hosted by the Linux Foundation, CIP aims to speed implementation of Linux-based civil infrastructure systems through industrial grade software and a universal operating system, build upon existing open source foundations and expertise, establish de facto standards by providing a base layer reference implementation, and contribute to and influence upstream projects regarding industrial needs.

“We depend on technical systems on a daily basis to keep us safe. Often times these are Linux-based systems that have to be maintained for more than ten years,” said Yoshitake Kobayashi, CIP Chair of the Technical Steering Committee and Senior Manager of The Open-Source Technology Department, Toshiba Corporation. “It is critical for us to better prepare our civil infrastructure systems, and the SLTS CIP kernel gets us one step closer to sustainability for up to multiple decades. With the new support for Arm64, the kernel can be applied to broader applications that are the future backbone of our lives.”

CROSS-INDUSTRY COLLABORATION AND DEVELOPMENT

Real-time Linux is a critical component for industrial grade systems. In addition to real-time management and data,  industrial systems require safety, security and reliability, which is why CIP plans to collaborate with the new Enabling Linux in Safety Applications (ELISA) project at the Linux Foundation. ELISA is an open source project to create a shared set of tools and processes to help companies build and certify Linux-based safety-critical applications and systems whose failure could result in loss of human life, significant property damage or environmental damage. Building off the work being done by SIL2LinuxMP project and Real-Time Linux project, ELISA will make it easier for companies to build safety-critical systems such as robotic devices, medical devices, smart factories, transportation systems and autonomous driving using Linux.

“Long-term maintenance and support is essential for the safety, security, and reliability required by embedded systems operating in industrial and infrastructure environments,” said Kate Stewart, Senior Director of Strategic Programs at the Linux Foundation. “With ELISA, we are collaborating with the broader Linux Foundation community like CIP to make this initiative successful. We look forward to working with CIP and its members on establishing processes and tooling to support certification of Linux-based safety-critical applications.”

CIP has also launched two new working groups to help manage specific aspects of the development process.

The Security Working Group will work with various security standards that help to address cyber security issues. Led by Renesas Electronics, the focus of the workgroup is for suppliers to certify using IEC 62443-4-x standards, which is one of the most important  security specification  for industrial products. They will keep the CIP platform up to date by certifying against various available standards and minimize the development time and cost for suppliers by creating a well-defined process for certification.

The Software Update Working Group will provide a robust software update tool that integrates and strengthens the industrial-grade open source base layer. Led by the Toshiba Corporation, the working group will focus on the software architecture, integrating chosen software into the Linux image build tools used by CIP Core and implementing the software update reference boards.

CIP is driven by some of the world’s most innovative industry leaders such as Codethink, Cybertrust, Hitachi, Moxa, Plat’Home, Renesas, Siemens and Toshiba and closely collaborates with other open source projects, such as Linux Kernel LTS, Debian Project, KernelCI. Many members plan to support the SLTS CIP kernel including Renesas, which recently announce RZ/G2 MPUs that will serve as a reference hardware for Arm64 for the certification and release of CIP Linux packages.

The source files for the CIP SLTS kernel can be found here: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/log/?h=linux-4.19.y.

Additional CIP Resources:

About CIP

The Civil Infrastructure Platform (CIP) is an open source project hosted by The Linux Foundation. The project is focused on establishing an open source base layer of industrial grade software to enable the use and implementation of reusable software building blocks that meet the safety, reliability and other requirements of industrial and civil infrastructure. For additional information, visit https://www.cip-project.org/.

About The Linux Foundation

Founded in 2000, the Linux Foundation is supported by more than 1,000 members and is the world’s leading home for collaboration on open source software, open standards, open data, and open hardware. Linux Foundation’s projects are critical to the world’s infrastructure including Linux, Kubernetes, Node.js, and more.  The Linux Foundation’s methodology focuses on leveraging best practices and addressing the needs of contributors, users and solution providers to create sustainable models for open collaboration. For more information, please visit us at linuxfoundation.org.

###

 

UEK

UEK is a Linux kernel that Oracle created to address the needs of customers running demanding software such as Oracle Database on large scale systems.

Oracle caused quite a stir in 2010 when it announced its Unbreakable Enterprise Kernel for Oracle Linux. We’ve checked in with Sergio Leunissen, Vice President, Linux and VM Development at Oracle, for an update on the ABCs of this important introduction as well as the company’s latest take on Linux.

Linux Foundation: First, please remind us what exactly is the Unbreakable Enterprise Kernel (UEK)?

Leunissen: UEK is a Linux kernel Oracle created to address the needs of customers running demanding software such as Oracle Database on large scale systems. Its focus is performance, stability, and minimal backports by tracking the upstream kernel code as closely as is practical. UEK is well-tested and used to run Oracle Engineered Systems, Oracle Cloud Infrastructure, and large enterprise deployments for Oracle customers. The source for UEK is published on GitHub: https://github.com/oracle/linux-uek

LF: Who is seeing the biggest benefit from the UEK?

Leunissen: First, it goes without saying that customers running Oracle software benefit from our development and testing of Oracle Database and middleware on tens of thousands of systems running Linux with UEK on a daily basis. Add to that the demands of running the infrastructure for Oracle Cloud running SaaS applications, databases, containers and Kubernetes clusters, etc. Our customers can take comfort in knowing that the kernel they run is the same one we run.

But, by no means do only Oracle customers benefit. Our kernel team adds features (to) and fixes bugs in subsystems that span the Linux kernel, including networking, block storage, filesystems, etc. This development will benefit any workload relying on the kernel’s overall ability to handle lots of memory, network, and I/O.

Finally, because UEK tracks upstream kernels so closely, the bugs we find and fix are relevant to the mainline Linux kernel.

LF: The Unbreakable Enterprise Kernel is touted as fast, modern and reliable. Can you elaborate on these benefits?

  • Fast – Optimized to run well on large systems with lots of memory and large storage systems. Works well with modern solid state storage.
  • Modern – tracking mainline Linux closely to incorporate the latest innovations
  • Reliable – Extensively tested by Oracle with real world workloads

Most customers we work with must stick to specific releases of core userspace components but do want to exploit innovations from upstream development efforts. With UEK, we are able to balance an enterprise support model with a Linux kernel that syncs up with mainline more frequently.

LF: How much emphasis do you put on security in your kernel development work?

Leunissen: Oracle is a cloud provider that contributes to Linux. Oracle Linux is the host OS for most of the Oracle application and infrastructure offerings.  As such, we work closely with our cloud development team to build a scalable Linux platform with virtualization and container services without compromising on security. UEK is a key part of this.

LF: The Linux kernel developers at Oracle work both on mainline directly and UEK, can you explain how this works?

Leunissen: As mentioned above, we publish the source for UEK on GitHub. Keeping our changes open source enables us to integrate with upstream Linux kernels quickly, which also means we have state-of-the-art drivers and filesystems, hardware support, and security fixes from the community. And, again, because UEK tracks upstream kernels so closely, we don’t spend a lot of time addressing bugs that are unique to the kernel as it relates to Oracle’s efforts. Rather, our fixes are relevant to upstream kernels as well.

LF: Are there particular development projects you are working on that you’d like to highlight?

Leunissen: As a cloud provider, containers and virtualization are important to Oracle. We do a lot of work on KVM and QEMU. For example, we are doing work to make sure Xen VM guests can run as is on a KVM host. Recently we’ve been working on Kata containers (previously Clear Containers), which is a deployment model for applications that combines the isolation of VMs with the speed, footprint, and interaction model of containers. Also, with UEK powering tens of thousands of systems in our cloud, we are doing work to improve the startup time performance for Linux systems by parallelizing kernel boot-time tasks, shaving precious seconds off the startup time for bare metal and virtualized workloads. Finally, it would be remiss not to point out that we are actively working on Linux for ARM with the focus to provide a high-quality Linux OS for 64-bit ARM (aarch64)-based servers.

New 4-course specialization prepares users for working productively with open source communities

SAN FRANCISCO, December 6, 2018The Linux Foundation, the nonprofit organization enabling mass innovation through open source, announced today that enrollment for a new 4-course specialization, Open Source Software Development, Linux and Git is now open. Offered through the world’s largest online platform for higher education, Coursera, students will attain the skills and knowledge needed to work comfortably and productively in open source development communities; have a good understanding of the Linux environment, as well as methods and tools required to successfully use it; and know how to use Git, the distributed version control system. This is the first time The Linux Foundation and Coursera have partnered to provide training opportunities.

Developed by the Linux Foundation’s Director of Training, Jerry Cooperstein, The Open Source Software Development, Linux and Git specialization is a remote learning program designed to give students a strong foundation of skills for working in open source development communities. It is designed for experienced computer users and developers who are looking to enter the world of open source development.

The specialization provides up to 70 hours of lectures and features labs for a hands-on learning experience. It requires that students have some familiarity with other operating systems, and experience working in the command line is also helpful, though not required.

A Coursera specialization is a series of related courses designed to help learners master a specific topic. To complete the specialization, learners must complete with a passing grade every course in the specialization.

”Coursera has a fantastic track record of making quality training programs accessible to individuals all around the world, and we are thrilled to partner with them on this new specialization,” said Linux Foundation General Manager, Training & Certification Clyde Seepersad. “Open source continues its dominance in software development and technological innovation, and providing opportunities to access this type of training will be hugely beneficial to individuals who have previously only worked on proprietary products.”

“At Coursera, we partner with leading organizations to help learners acquire the high-demand skills for the jobs of the future. As more professionals seek to utilize innovative technologies like cloud, artificial intelligence, and machine learning, open source software development skills become critical,” said Kevin McFarland, Head of Industry Partnerships, Coursera. “We are excited to partner with The Linux Foundation, the authority in Linux open source development, to bring these invaluable skills to developers around the world.”

Course Details:

Open Source Software Development MethodsOpen Source Software Development Methods is designed to introduce open source software to students as well as cover its history and the benefits it has provided to the global technological infrastructure. Participants will learn how to work productively while using open source projects, and learn the best practices involving collaboration. The class will also discuss the methods that are commonly used for open source, the use of hosting providers such as GitHub, and provide examples of previous open source projects.

Linux for Developers

Linux For Developers is designed to introduce students to the Linux operating system and teach them how to work comfortably at the command line. Students will be introduced to the Linux platform, and acquire the essential skills necessary for working with the operating system. Students will learn important skills such as learning how to separate the kernel from the Linux operating system, and how to make contributions to it.

Linux Tools for Developers

Linux Tools For Developers will focus on utilizing the tools needed for everyday work involving Linux. The course will cover the command line tools that are used daily by developers. The course will also focus on bash scripting where students will learn how to construct scripts and how to successfully complete complicated tasks in an automated fashion. Students will also learn about files and filesystems, and learn how to compile programs in Linux using compilers such as GCC.

Using Git for Distributed Development

Using Git for Distributed Development will focus on using Git for open source software development. The course will prepare students for using Git efficiently and help them make contributions to projects in a short timespan. Throughout the course, students will learn about Git workflows, how to identify problems in workflows, and how to find solutions to solve such problems.

Registration for the specialization is now open, at a cost of $49 per month until completion. To register for the specialization, visit https://coursera.org/specializations/oss-development-linux-git

About The Linux Foundation

The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and industry adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

About Coursera

Coursera is an online education company that partners with 170 of the world’s top universities and industry leaders to offer courses, Specializations, and degrees that empower learners around the world to achieve their career, educational, and personal enrichment goals throughout their lives. Since launching in 2012, the company has grown to 35 million registered learners who can choose among 2,900 courses and 300 Specializations. Leading universities are now also working with Coursera to offer online degrees in areas like business, computer science, data science, and public health. Coursera is backed by leading venture capital firms such as Kleiner Perkins Caufield & Byers, New Enterprise Associates, GSV Capital, International Finance Corporation, Laureate Education Inc., and Learn Capital.

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage.

Linux is a registered trademark of Linus Torvalds.

# # #

Course designed to help developers with expertise in other operating systems to gain more Linux, Git and general open source knowledge and experience

SAN FRANCISCO, June 6, 2018The Linux Foundation, the nonprofit organization enabling mass innovation through open source, today announced the availability of LFD201 – Introduction to Open Source Development, Git, and Linux, a training course focused on open source software, an introduction to Linux systems and the use of Git, the revision control system.

The 2018 Open Source Jobs Report from The Linux Foundation and Dice, scheduled to be released later this month, will reveal that Linux is back on top as the most in demand skill by hiring managers seeking open source talent. After 65% of hiring managers reported seeking Linux talent in 2017, already down from 71% in the 2016 report, it has rebounded to take the top spot in 2018 at 80%.

“Open source software development practices lead to better code and faster development, which is why open source has become the dominant model for how the world’s technology infrastructure is built and operates,” said Linux Foundation General Manager, Training & Certification Clyde Seepersad. “Git has also become the de facto standard for collaborative development, with tens of millions of projects using it. This is why it is imperative to make it easier for developers to master these systems, and this new course is a great first step in that journey.”

LFD201 covers how open source software works, including advantages of using it, methods of working in OSS communities, governance models and licensing choices. It then examines Linux systems and a wide set of topics, including installation, desktop environments, text editors, important commands and utilities, command shells and scripts, file systems and compiling software. The final module gives a thorough introduction to Git, the source control system that arose out of the Linux kernel community, that enables widely distributed development to operate efficiently.

The online course, accessible from anywhere in the world and only requiring a physical or virtual Linux environment – running any Linux distribution – contains 43 hands-on lab exercises that will allow students to practice their skills, as well as a similar number of knowledge check quizzes and more than 20 videos demonstrating accomplishing important tasks.

The course objectives are to:

  • Obtain a strong foundation for working comfortably and productively in open source development communities
  • Learn to work comfortably and productively in a Linux environment
  • Master important Linux methods and requisite tools
  • Learn to use Git to create new repositories or clone existing ones
  • Learn to use Git to commit new changes, review revision histories, and examine differences with older versions
  • Learn to use Git to work with different branches, merge repositories, and work with a distributed development team.

This course is addressed to those who are already experienced computer users and developers on another operating system, but have limited or no experience working in a Linux environment; and/or those who have already done some work on Linux systems and are looking to gain a good working grasp of Git.

LFD201 was developed by Linux Foundation Director of Training Jerry Cooperstein, who has worked with Linux since 1994, developing and delivering training in both the kernel and user space. For the better part of two decades, Cooperstein worked on problems in nuclear astrophysics including supernova explosions, nuclear matter and neutron stars, general relativity, neutrinos and hydrodynamics at various national laboratories and universities in the United States and Europe. During that time, he developed state-of-the-art simulation software on many kinds of supercomputers and taught at both the undergraduate and graduate level. Cooperstein joined the Linux Foundation in 2009 as the Training Program Director.

LFD201 is available to take now for $299. Students can register at https://training.linuxfoundation.org/linux-courses/development-training/introduction-to-open-source-development-git-and-linux.

About The Linux Foundation

The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and industry adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage.

Linux is a registered trademark of Linus Torvalds.

# # #

Linux kernel

Get insights from Jon Corbet on the state of Linux kernel development.

At the recent Embedded Linux Conference + OpenIoT Summit, I sat down with Jonathan Corbet, the founder and editor-in-chief of LWN to discuss a wide range of topics, including the annual Linux kernel report.

The annual Linux Kernel Development Report, released by The Linux Foundation is the evolution of work Corbet and Greg Kroah-Hartman had been doing independently for years. The goal of the report is to document various facets of kernel development, such as who is doing the work, what is the pace of the work, and which companies are supporting the work.

Linux kernel contributors

To learn more about the companies supporting Linux kernel development in particular, Corbet wrote a set of scripts with the release of kernel 2.6.20, to pull the information out of the kernel repository. That information helped Corbet associate contributions with employers, whenever possible.

When Corbet published a report based on these findings in LWN, it created a bit of a stir. “It was a surprise to everyone, including me, because there was still this image of free software in general and Linux in particular as being something produced by kids who haven’t moved out of their parents basements,” said Corbet.

He found that more than 70 percent of the code going into the kernel was coming from professional developers who were getting paid to do that work. “Since then things have changed and our numbers have gotten better. Today, over 90 percent of the code is coming from professional developers who are employed by some company to work on the kernel,” he said.

Corbet has been involved with the Linux kernel from a very early stage, so connecting the dots was not too difficult, even though not all developers use official company email accounts,

“In most cases, we know who is working for which company. Sometimes people contact us and say that their employer wants to ensure that they do get credit for the work they are doing in the kernel. Sometimes we just ask who they are working for,” said Corbet.

Corbet not only gathers valuable data about the Linux kernel, he also analyzes the data to see some patterns and trends. The biggest trend, over the years, has been a decline in the number of contributions coming from volunteers, which has decreased from 15 percent to 6 percent since the 2.6.20 release.

“There are times when we have worried about it because volunteers are often the people who are in the next round of paid developers. That’s often how you get into the community — by doing a little bit of stuff on your own time,” he said. Corbet did a bit of digging to see the employment status of people when their very first patch merged and their latest status. He found that at this point most of those people were already working for some company.

While it’s true there are fewer volunteer developers now, it could also be said that people don’t remain volunteers for very long because when their code gets merged into the kernel, companies tend to approach these developers and offer jobs. So, if your code shows up in the kernel, that’s a good resume to have.

What keeps Corbet awake at night

There has been a growing concern of late that the Linux kernel community is getting older. Looking at the top maintainers, for example, you can see a lot of people who have been involved since the 1990s.

“The upper cadre is definitely getting a little bit older, a little bit grayer. There is some truth to that and I think the concerns of that are not entirely overblown,” said Corbet. “A whole bunch of us managed to stumble into something good back in the ’90s, and we have stuck with it ever since because it’s been a great ride.”

That doesn’t mean new people are not coming in. A new kernel is released every 9 to 10 weeks. And, every new release sees contributions from more than 200 developers submitting their very first patch.

“We are bringing a lot of new people into the community,” Corbet said. “Maybe half of those 200 contributors will never contribute anything again. They had one thing they wanted to fix and then they moved on. But there are a lot many others who stick around and become long-term members of the community. Some of these worked their way into the subsystem maintainer positions. They will be replacing the older members as they retire.”

Corbet is not at all worried about the aging community as it has evolved into an “organic” body with continuous flow of fresh blood. It’s true that becoming a kernel developer is more demanding; you do have to work your way into it a little bit, but plenty of people are doing it.

“I’m not really worried about the future of our community because we are doing so well at attracting bright new developers,” said Corbet, “We have an influx rate that any other project would just love to have.”

However, he did admit that the community is showing increasing signs of stress at the maintainer level. “The number of maintainers is not scaling with a number of developers,” he said. However, he said, this problem is not unique to the kernel community; the whole free software community is facing this challenge.

Another concern for Corbet is the emergence of other kernels, such as Google’s Fuchsia. These kernels are being developed specifically to be permissively licensed, which allows them to  be controlled by one or a very small number of companies. “Some of those kernels could push Linux aside in various subfields,” said Corbet. “I think some of the corporate community members have lost sight of what made Linux great and so successful. It could be useful for some companies in the short term, but I don’t think it’s going to be a good thing for anyone in the long term.”

Core needs

Corbet also noted another worrisome trend. Although many companies contribute to every kernel release, if you look closely you will see that a lot of these contributions are toward making their own hardware work great with Linux.

“It’s a great thing. We have been asking them to do it for years, but there is a whole lot of the kernel that everyone needs,” he said. There is the memory management subsystem. There’s the virtual filesystem layer. There are components of the kernel that are not tied to any single company’s hardware, and it’s harder to find companies willing to support them.

“Some of the companies that contribute to the most code to the kernel do not contribute to the core kernel at all,” said Corbet.

Corbet also worries about the lack of quality documentation and has himself initiated some efforts to improve the situation. “Nobody wants to pay for documentation,” he said. “There is nobody whose job it is to write documentation for the kernel, and it really shows in the quality. So, some of those areas I think are really going to hurt us going forward. We need to get better investment there.”

You can hear more from Jon Corbet, including insights on the recent Spectre and Meltdown issues, in his presentation from Embedded Linux Conference:

You can learn more about the Linux kernel development process in the complete annual report. Download the 2017 Linux Kernel Development Report now.

linux kernel developer

Linux kernel developer Steven Rostedt maintains the Real Time Stable releases of the Linux kernel.

Linus Torvalds recently released version 4.16 of the Linux kernel. These releases typically occur every nine to ten weeks, and each one contains the work of more than 1,600 developers representing over 200 corporations, according to the 2017 Linux Kernel Development Report, written by Jonathan Corbet and Greg Kroah-Hartman. In this series, we’re highlighting some of the developers who contribute to the kernel.

Steven Rostedt, Open Source Programmer at VMware, maintains the Real Time Stable releases of the Linux kernel, among other things. Rostedt is one of the original developers of the PREEMPT_RT patch and began working on it in 2004 with the goal of turning Linux into a real-time designed operating system. He is also the main author, developer, and maintainer of Ftrace, a tool designed to help developers find what is going on inside the kernel. According to the Ftrace wiki, the tool can be used for debugging or analyzing latencies and performance issues that take place outside of user-space.

Linux kernel dev

Steven Rostedt

Additionally, this past year, Rostedt found time to speak at various events and serve on The Linux Foundation’s technical advisory board. Here are Rostedt’s responses to our questions.

Linux Foundation: What role do you play in the community and what subsystem(s) do you work on?

Steven Rostedt: I partake in a lot of the Linux Foundation events as well as Kernel Recipes, Linux Plumbers, sometimes Linux Tag and other events. I’m on The Linux Foundation’s Technical Advisory Board (TAB) and was on the Linux Plumbers programming committee. I’m an Open Source advocate and try to communicate to people what that means. I maintain the Real Time Stable releases, and the Ftrace (Linux kernel tracer) subsystem, as well as ktest, localmodconfig, and Ftrace tools like trace-cmd and KernelShark.

Linux Foundation: What have you been working on this year? / What’s one way you have contributed to the 4.8 to 4.13 releases?

Rostedt: I’ve been working on having ftrace trace init functions in both the main kernel core as well as in modules. Between 4.8 and 4.13, I rewrote the function tracing trigger code to be able to be expanded and used to enable function filtering for tracing on modules before they are loaded.

Linux Foundation:  What do you think the kernel community needs to work on in the upcoming year?

Rostedt: I think more focus should be on eBPF and helping it be easier to use as well as having an eye on security. Running a VM within the kernel can be very dangerous, and people need to use caution and be extra careful during development.

Linux Foundation: Why do you contribute to the Linux kernel?

Rostedt: Because it is the one place that you have total control over your computer.

At the recent Embedded Linux Conference, Rostedt presented a session on “Maintaining a Real Time Stable Kernel,” in which he explained what’s required to maintain a stable RT tree, which is a bit different from maintaining a normal stable tree. In this talk, he covered various tools that can be used and described the current tests performed to ensure that the RT stable kernel is fully functional.

You can learn more about the Linux kernel development process and read more developer profiles in the full report. Download the 2017 Linux Kernel Development Report now.

To find and report bugs, Linux kernel developers depend on a wide community of testers.

A kernel that has had nearly 83,000 patches applied will certainly have a few bugs introduced along with the new features, states the 2017 Linux Kernel Development Report, written by Jonathan Corbet and Greg Kroah-Hartman.Linux kernel

To find and report those bugs, Linux kernel developers depend on a wide community of testers. And, according to convention, when a bug-fixing patch is applied to the kernel, it should contain a “Reported-by” tag to credit the tester who found the problem. During the period covered by the most recent report, more than 4,100 patches carried such tags, and the top 21 bug reporters are shown in the table at right.

Julia Lawall, Senior Researcher at Inria, is one of the developers involved in the bug reporting process, as she works on the Coccinelle tool that is used to find bugs in the kernel. Here, Lawall answers a few questions about her contributions to the development process.

Linux kernel

Julia Lawall

The Linux Foundation: What role do you play in the community and what subsystem(s) do you work on?

Julia Lawall: I work on the tool Coccinelle that is used to find bugs in the Linux kernel and perform large-scale evolutions.

The Linux Foundation: What have you been working on this year?

Lawall: This year I have been working with Bhumika Goyal on making various kernel structures read-only. We have constified over 1500 structures this year. This work has also motivated various bug fixes and performance improvements in Coccinelle.

I have also been working on automatically identifying patches that should be considered for backporting to stable kernels, in collaboration with Greg K-H, Sasha Levin, and colleagues at Singapore Management University. Our approach is still work in progress, but several hundred commits that were not originally tagged for stable have been identified and applied to stable versions.

The Linux Foundation: What do you think the kernel community needs to work on in the upcoming year?

Lawall: Initial experiments suggest that the rate of propagation of commits to stable is rather uneven across the kernel. This can be due to the different properties of different subsystems, but there can also be room for maintainers to annotate commits for stable kernels more frequently and consistently.

The Linux Foundation: Why do you contribute to the Linux kernel?

Lawall: Many reasons: the potential impact, the challenge of understanding a huge code base of low-level code, the chance to interact with a community with a very high level of technical skill.

You can learn more about the Linux kernel development process and read more developer profiles in the full report. Download the 2017 Linux Kernel Development Report now.