Lukas Bulwahn | 06 September 2018
The Real-Time Linux project team continues to prepare the remaining patches for inclusion into the mainline kernel.
Long ago in 2009, a small team of kernel developers had finished consolidating previous prototypic developments to make Linux real-time capable into a single out-of-tree patch set, called the PREEMPT_RT patch set. This patch set can be applied to turn a vanilla mainline Linux kernel without real-time capabilities into a real-time capable Linux kernel. Many companies use this patch set to build various industrial systems that required to implement hard real-time properties at comparatively relaxed time bounds of about one millisecond precision.
BMW Car IT also used this patch set to build real-time capable prototypes for complex functions in the area of autonomous driving. However, from the beginning with the development of those prototypes, it was clear that any product with high-quality demands requires to get the PREEMPT_RT patch set in the main-line development for increased compatibility of features, stronger quality assurance and reduced maintenance. Hence, BMW Car IT started driving efforts to make Linux real-time capable in 2014.
First, BMW Car IT joined OSADL, the Open Source Automation Development Lab, as a Gold member to support real-time Linux development activities, which was collaboratively funded by the OSADL member at that time.
Second, our former colleague Daniel Wagner started to get acquainted with the existing PREEMPT_RT patch in 2014 and made a number of contributions to the Linux kernel related to real-time capabilities from 2015 until end of 2016. Due to his experience with the PREEMPT_RT patch, he is now the maintainer of the Linux 4.4 real-time stable branch, and one of the three maintainers for the real-time stable patch branches.
Since 2016, the Real-time Linux project has been a collaborative project under the umbrella of the Linux Foundation. The project’s goal is to make the mainline Linux real-time capable. The project ensures that the Linux kernel developers have the ability to continue development work, long-term support and future research for a real-time-capable Linux.
Rewriting and Refactoring
In the last two years, 2016 and 2017, the Real-time Linux development team rewrote the CPU hotplug infrastructure and refactored the timer wheel and high-resolution timers. This already reduced the out-of-tree PREEMPT_RT patch set significantly.
Due to a funding decrease that became apparent at the beginning of 2018, the development in the Real-time Linux project would have reduced its workforce. Fortunately, Intel and BMW Car IT could close this funding gap. Intel increased their membership from Gold to Platinum and BMW Car IT joined Linux Foundation and the collaborative project as Gold member in the Real-time Linux Project. So now after those project adjustments, the Real-time Linux Project team is back on track and continues to prepare the remaining patches for inclusion into the mainline development with full speed.
In 2018, the Real-time Linux kernel team will be refactoring, rewriting and generally improving the printk and soft interrupt infrastructure and other smaller other parts. This work will prepare the Linux kernel source code so that all further real-time specific changes can smoothly be merged into the mainline kernel.
The real-time functionality touches the core kernel parts (i.e., it requires significant changes in timers, schedulers, locking mechanisms, interrupt handling and more), and it also is a cross-cutting concern for all drivers (i.e., every driver has to follow a certain discipline to make the overall kernel real-time capable). Hence, it is difficult to predict the exact date when the Real-time Linux Project will finally have all its patches merged into the main-line development. However, there is no doubt that the Linux kernel will eventually become real-time capable.
“The Linux kernel is a software development project of huge invest to us. Obviously, BMW Car IT has a high interest of making best possible use of this software asset. The automotive industry has particular requirements, such as higher real-time requirements and the need for longer maintenance periods, than the general IT and consumer electronics industry. With our investments in initiatives addressing these requirements, we can ensure that Linux fits to our needs,” says Kai-Uwe Balszuweit, CEO of BMW Car IT.
Reviewing and Testing
Once the real-time capabilities have been integrated in the main-line development, the project work is of course not just finished and the Real-time Linux project cannot just be abandoned. After the final integration into the main-line development, the development activities will slowly shift its focus:
The core system will not require further changes for the real-time capability, but the Real-Time Linux development team will need to review, test and adjust new incoming features from other kernel development teams to keep the kernel real-time capable when these new features are included.
Furthermore, the already existing real-time stable trees must be further continued to be maintained until the end of life of the corresponding kernel LTS version, so commonly two years for most LTS versions, but possibly even longer. Slowly over the years, the real-time stable trees for older kernel versions will reach their end of life, while for younger LTS kernel versions, which have the real-time capabilities fully included, have no need to maintain a separate real-time stable branch. This will decrease the working effort on the current real-time stable maintainers and they can focus their work to assist in the quality assurance of the continuous main-line development.
Of course, all users and stakeholders of the real-time capability must continue to support all these activities over the next years.
This is well understood at BMW Car IT, and we expect that other companies that require the real-time capability in Linux will also follow and express this general common understanding. Beyond software development until start of production, operations and maintenance is an important software development activity that is not underestimated at BMW Car IT.
Christian Salzmann, the CEO of BMW Car IT, states it clearly: “Providing good software solutions to BMW for many years, BMW Car IT knows that continuous operations and maintenance is one of the major cornerstones for providing a great experience to our customers. The continuous activity of development and operations of software going hand-in-hand, in short DevOps, is part of BMW Car IT’s company mindset. BMW Car IT’s support for further development and operations in the Real-time Linux Project is no exception to this rule.”