Posts

The Xen Project is now one of the most popular open source hypervisors and amasses more than 10 million users, and this October marks our 15th anniversary.

In the 1990s, Xen was a part of a research project to build a public computing infrastructure on the Internet led by Ian Pratt and Keir Fraser at The University of Cambridge Computer Laboratory. The Xen Project is now one of the most popular open source hypervisors and amasses more than 10 million users, and this October marks our 15th anniversary.

From its beginnings, Xen technology focused on building a modular and flexible architecture, a high degree of customizability, and security. This security mindset from the outset led to inclusion of non-core security technologies, which eventually allowed the Xen Project to excel outside of the data center and be a trusted source for security and embedded vendors (ex. Qubes, Bromium, Bitdefender, Star Labs, Zentific, Dornerworks, Bosch, BAE systems), and also a leading hypervisor contender for the automotive space.

As the Xen Project looks to a future of virtualization everywhere, we reflect back on some of our major achievements over the last 15 years. To celebrate, we’ve created an infographic that captures some of our key milestones share it on social.

A few community members also weighed in on some of their favorite Xen Project moments and what’s to come:

“Xen offers best-in-class isolation and separation while preserving nearly bare-metal performance on x86 and ARM platforms. The growing market for a secure hypervisor ensures Xen will continue to grow in multiple markets to meet users demands.”

  • Doug Goldstein, Software Developer V, Hypervisors at Rackspace

“Xen started life at the University of Cambridge Computer Laboratory, as part of the XenoServers research project to build a public computing infrastructure on the Internet. It’s been fantastic to see the impact of Xen, and the role it’s played at the heart of what we now call Infrastructure as a Service Cloud Computing. It’s been an incredible journey from Xen’s early beginnings in the University, to making our first open source release in 2003, to building a strong community of contributors around the project, and then Xen’s growth beyond server virtualization into end-user systems and now embedded devices. Xen is a great example of the power of open source to enable cooperation and drive technological progress.”

  • Ian Pratt, Founder and President at Bromium, and Xen Project Founder

“From its beginnings as a research project, able to run just a handful of Linux VMs, through being the foundation of many of the world’s largest clouds, to being the open-source hypervisor of choice for many next-generation industrial, automotive and aeronautical applications, Xen Project has shown its adaptability, flexibility and pioneering spirit for 15 years. Today, at Citrix, Xen remains the core of our Citrix Hypervisor platform, powering the secure delivery of applications and data to organizations across the globe. Xen Project Hypervisor allows our customers to run thousands of virtual desktops per server, many of them using Xen’s ground-breaking GPU virtualization capabilities. Happy birthday, Xen!”

  • James Bulpin, Senior Director of Technology at Citrix

“The Xen open source community is a vibrant and diverse platform for collaboration, something which is important to Arm and vital to the ongoing success of our ecosystem. We’ve contributed to the Xen open source hypervisor across a range of markets starting with mobile, moving into the strategic enablement that allowed the deployment of Arm-based cloud servers, and more recently focusing on the embedded space, exploring computing in safety-sensitive environments such as connected vehicles.”

  • Mark Hambleton, Vice President of Open Source Software, Arm

“I – like many others – associate cloud computing with Xen. All my cloud-related projects are tied to companies running large deployments of Xen. These days even my weekend binge-watching needs are satisfied by a Xen instance somewhere. With Xen making its way into cars, rocket launch operations and satellites, it’s safe to say the industry at large recognizes it as a solid foundation for building the future, and I’m excited to be a part of it.”

  • Mihai Dontu, Chief Linux Officer at Bitdefender

“Xen was the first open source hypervisor for the data center, the very foundation of the cloud as we know it. Later, it pioneered virtualization for embedded and IoT, making its way into set-top boxes and smaller ARM devices. Now, we are discussing automotive, medical and industrial devices. It is incredibly exciting to be part of a ground-breaking project that has been at the forefront of open source innovation since its inception.”

  • Stefano Stabellini, Principal Engineer, Tech Lead at Xilinx and Xen on ARM Committer and Maintainer

“Congratulations to the Xen Project on this milestone anniversary. As the first open source data center hypervisor, Xen played a key role in defining what virtualization technology could deliver and has been the foundation for many advancements in the modern data center and cloud computing. Intel has been involved with Xen development since the early days and enjoys strong collaboration with the Xen community, which helped make Xen the first hypervisor to include Intel® Virtualization Technology (VT-x) support, providing a more secure, efficient platform for server workload consolidation and the growth of cloud computing.”

  • Susie Li, Director of Open Source Virtualization Engineering, Intel Corp.

“It is amazing how a project that started 15 years ago has not lost any of its original appeal, despite the constant evolution of hardware architectures and new applications that were unimaginable when the Xen Project started. In certain segments, e.g. power management, the pace of innovation in Xen is just accelerating and serves as the ultimate reference for all other virtualization efforts. Happy quinceañera (sweet 15) Xen!”

  • Vojin Zivojnovic, CEO and Co-Founder of Aggios

Building the Journey Towards the Next 15 Years; Sneak Peek into Xen Project 4.12

The next Xen Project release is set for March 2019. The release continues to support the Xen Project’s efforts around security with cloud environments and rich features and architectural changes for automotive and embedded use cases. Expect:

  • Deprivileged Device Model: Under tech preview in QEMU 3.0, the feature adds extra restrictions to a device model running in domain 0 in order to prevent a compromised device model to attack the rest of the system.  
  • Capability to compile a PV-only version of Xen giving cloud providers simplified management, reducing the surface of attack, and the ability to build a Xen Project hypervisor configuration with no “classic” PV support at all.
  • Xen to boot multiple domains in parallel on Arm, in addition to dom0 enabling booting of domains in less than 1 second. This is the first step towards a dom0-less Xen, which impacts statically configured embedded systems that require very fast boot times.  
  • Reduction of codesize to 46 KSLOC for safety certification and the first phase of making the codebase MISRA C compliant.
    • MISRA C is a set of software development guidelines for the C programming language developed by the Motor Industry Software Reliability Association with the aim to facilitate code safety, security, portability, and reliability in the context of embedded systems.

Thank you for the last 15 years and for the next 15+ to come!

Lars Kurth, Chairperson of the Xen Project

ACRN is a flexible, lightweight reference hypervisor, built with real-time and safety-criticality in mind.

This article was produced by The Linux Foundation with contributions from Eddie Dong, Principle Engineer of Intel Open Source Center.

As the Internet of Things has grown in scale, IoT developers are increasingly expected to support a range of hardware resources, operating systems, and software tools/applications. This is a challenge given many connected devices are size-constrained. Virtualization can help meet these broad needs, but existing options don’t offer the right mix of size, flexibility, and functionality for IoT development.
ACRN

ACRN is different by design. Launched at Embedded Linux Conference 2018, ACRN is a flexible, lightweight reference hypervisor, built with real-time and safety-criticality in mind and optimized to streamline embedded development through an open source platform.

One of ACRN’s biggest advantages is its small size — roughly only 25K lines of code at launch.

“The idea for ACRN came from our work enabling virtualization technology for customers,” said Imad Sousou, Corporate Vice President and General Manager of the Open Source Technology Center at Intel, which seeded the source code to launch the project. “There’s strong workload consolidation in embedded IoT development. Using hypervisor technology, workloads with mixed-criticality can be consolidated on a single platform, lowering development and deployment costs and allowing for a more streamlined system architecture.”

And about the name: ACRN is not an acronym. Pronounced “acorn,” the name symbolizes something that starts small and grows into something big, similar to how the project hopes to grow through community participation.

There’re two key components of ACRN: the hypervisor itself and the ACRN device model. The ACRN Hypervisor is a Type 1 reference hypervisor stack, running directly on bare-metal. The ACRN Device Model is a reference framework implementation for virtual device emulation that provides rich I/O virtualization support currently planned for audio, video, graphics, and USB. More mediator features are expected as the community grows.

How it works

ACRN features a Linux-based Service operating system (OS) running on the hypervisor and can simultaneously run multiple guest operating systems for workload consolidation. The ACRN hypervisor creates the first virtual environment for the Service OS and then launches Guest OSes. The Service OS runs the native device drivers to manage the hardware and provides I/O mediation to the Guest OS.ACRN

The Service OS runs with the system’s highest virtual machine priority to meet time-sensitive requirements and system quality of service (QoS). The Service OS runs Clear Linux* today, but ACRN can support other Linux* distros or proprietary RTOS as either the Service OS or Guest OS. The community is invited to help enable other Service OS options, and use the reference stack to enable Guest OSes such as other Linux* distributions, Android*, Windows* or proprietary RTOSes.

To keep the ACRN hypervisor code base as small and efficient as possible, the bulk of device model implementation resides in the Service OS to provide sharing and other capabilities. The result is a small footprint, low-latency code base optimized for resource constrained devices, built with virtualization functions specific to IoT development, such as graphics, media, audio, imaging, and other I/O mediators that require sharing of resources. In this way ACRN fills the gap between large datacenter hypervisors and hard partitioning hypervisors, and is ideal for a wide variety of IoT development.

One example is the Software Defined Cockpit (SDC) in vehicles. Using ACRN as the reference implementation, vendors can build solutions including the instrument cluster, in-vehicle infotainment (IVI) system, and one or more rear-seat entertainment (RSE) systems. The IVI and RSE systems can run as an isolated Virtual Machine (VM) for overall system safety considerations.

Software Defined Industrial Systems (SDIS) are further examples, including cyber-physical systems, IoT, cloud computing and cognitive computing. ACRN can help SDIS consolidate industrial workloads and can be orchestrated flexibly across systems. This helps provide substantial benefits to customers including lower costs, simplified security, increased reliability, and easier system management, among others.

Early endorsement of ACRN includes Intel, ADLINK Technology, Aptiv, LG Electronics, and Neusoft. Community members are invited to download the code and participate at the ACRN GitHub site. More detailed use case information and participation information can be found on the ACRN website.

Join us at Open Source Summit + Embedded Linux Conference Europe in Edinburgh, UK on October 22-24, 2018, for 100+ sessions on Linux, Cloud, Containers, AI, Community, and more.

Also, check out the ACRN Hypervisor Meetup in Shanghai – Q2 2018 (Minhang, China):

2018年3月Linux Foundation 发布了 ACRN hypervisor项目。随后陆续收到了很多来自社区和行业伙伴的反馈。这次的Meetup希望给大家一次面对面交流的机会。英特尔公司作为ACRN项目的发起者之一 将会介绍一下项目的体系架构,ACRN 未来的roadmap (draft)讨论,也将演示一些应用场景。各行业伙伴也将会分享各自的关心的话题。ACRN作为一个Linux Foundation的开源项目热情欢迎大家的参与与反馈。

Xen Project technology supports more than 10 million users and is a staple in some of the largest clouds in production today, including Amazon Web Service, Tencent, and Alibaba’s Aliyun. Recently, the project announced the arrival of Xen Project Hypervisor 4.7. This new release focuses on improving code quality, security hardening and features, and support for the latest hardware. It is also the first release of the project’s fixed-term June – December release cycles. The fixed-term release cycles provide more predictability making it easier for consumers of Xen to plan ahead.  

We recently sat down with the Xen Project chairperson, Lars Kurth, to talk about some of the key features of the release and the future of Xen Project technology. Lars will be discussing this topic and more during Xen Project’s Developer Summit in Toronto, CA from August 25-26 — the conference is directly after LinuxCon North America.

Q: What was the focus on this release?

Lars Kurth: There were five areas that we focused on for this release (full details are in our blog). In summary, we focused on security features, migration support, performance and workloads, support for new hardware features, and drivers and devices (Linux, FreeBSD and other).

Security is consistently something that we focus on in all of our releases. There are a lot of people that rely on Xen Project technology and security is our top concern in any release as well as how we organize our process around security disclosures.

Q: What was the biggest feature coming out of this release?

Lars: The biggest feature for us is live patching, which is a technology that enables re-boot free deployment for security patches to minimize disruption and downtime during security upgrades for cloud admins. It essentially eliminates all cloud reboots, making cloud providers and their users much more safe. It also eliminates a lot of headaches for system and DevOps admins of the world.

Q: Xen is often associated with the cloud, but are there additional use cases that you see growing around this technology, if so why?

Lars: We are seeing a lot of growth in terms of contributions, as well as many different use cases emerging, including automotive, aviation, embedded scenarios, security, and also IoT. In addition, we continue to grow within the public cloud sector and traditional server virtualization.

On the security front, for example, a number of vendors such as A1Logic, Bitdefender, Star Lab and Zentific have released or are working on new Xen Project-based security solutions. In addition, the security focused and Xen-based OpenXT project has started to work more closely with the Xen Project community.

Long-time contributors to the Xen Project, such as DornerWorks – a premier provider of electronic engineering services for the aerospace, medical, automotive, and industrial markets – have expanded their scope and are now providing support for the Xen Xilinx Zynq Distribution targeting embedded use-cases. We have also seen an increasing number of POCs and demos of automotive solutions, which include Xen as a virtualization solution.

Growth in these sectors is largely due to the Xen Project’s flexibility, extensibility, customisability and a clear lead when it comes to security-related technologies. Over the last year, we have also seen contributions increase from developers with strong security and embedded backgrounds. In fact, this totaled nearly 17 percent of the overall contributions in this release cycle, up from 9 percent in the previous release.

Q: How did you address these uses cases in this latest release?

Lars: We introduced the ability to remove core Xen Project Hypervisor features at compile via KCONFIG. This creates a more lightweight hypervisor and eliminates extra attack surfaces that are beneficial in security-first environments and microservice architectures. Users will still be able to get the core hypervisor functions, but they won’t receive all the drivers, schedulers, components or features that might not fit their use case.

Essentially it gives people an “a la carte” feature set. They can decide what they need for compliance, safety or performance reasons.

Q: Were there any new contributors for this release that surprised you?

Lars: We had three new companies contributing to the project: Star Lab, Bosch and Netflix. I met engineers from Star Lab for the first time at the 2015 Developer Summit less than a year ago, and helped introduce them to the Project’s culture. In that short period of time, Doug Goldstein from Star Lab has moved into the top five contributors and top 10 code reviewers for the Project.

I was surprised about Netflix’s contributions; I didn’t even know the company used Xen. Netflix improved and secured the VPMU feature, which is incredibly useful for system tuning and performance monitoring. Bosch Car Multimedia GmbH added some new ARM functionality. In addition, we have seen quite a bit of Xen related development in upstream and downstream projects such as Linux, FreeBSD, NetBSD, OpenBSD, QEMU and Libvirt.  

Q: What’s next for Xen Project? Where do you think the technology is heading in the future and why?

Lars: In the last three releases, we introduced several major new features such as PVH, COLO, new schedulers, VMI, Live Patching, Graphics Virtualization, etc. and significant re-work of existing features such as Migration and the Xen Security Modules (XSM). Looking at trends within the community, I expect that stepwise evolution of large new features to continue.

Some new capabilities, such as restartable Dom0’s, and additional techniques to provide more isolation and security, are also likely to appear. In addition, it looks likely that we will see some GPU virtualization capabilities for GPUs that target the ARM ecosystem, although it is not yet clear whether these will be available as open source. I also expect that both Intel and ARM hardware features will be closely tracked.

Some areas, such as new schedulers, XSM, PVH and Live Patching, will see significant efforts to harden and improve existing functionality. The goal is to ensure their swift adoption in commercial products and Linux and BSD distributions. Some features, which are not enabled by default are likely to become part of the Xen Project Hypervisor’s default configuration.

The Xen Project’s code contributions have grown more than 10 percent each year. Although growth is extremely healthy to the project as a whole, it has its growing pains. For the Xen Project, it led to issues with its code review process: maintainers believed that their review workload increased and a number of vendors claimed that it took significantly longer for contributions to be upstreamed, compared to the past.

The project developed some basic scripts that correlated development list traffic with Git commits, which showed indeed that it took longer for patches to be committed. In order to identify possible root causes, the project initially ran a number of surveys to identify possible causes for the slow down. Unfortunately, many of the observations made by community members contradicted each other, and were thus not actionable. To solve this problem, the Xen Project worked with Bitergia, a company that focuses on analyzing community software development processes, to better understand and address the issues at hand.

We recently sat down with Lars Kurth, who is the chairperson for the Xen Project, to discuss the overall growth of the Xen Project community as well as how the community was able to improve its code review process through software development analytics.

Like many FOSS projects, the Xen project code review process uses a mailing list-based review process, and this could be a good blueprint for projects that are finding themselves in the same predicament.

Linux.com: Why has there been so much growth in the Xen Project Community?

Lars Kurth: The Xen Project hypervisor powers some of the biggest cloud computing companies in the world, including Alibaba’s Aliyun Cloud Services, Amazon Web Services, IBM Softlayer, Tencent, Rackspace and Oracle (to name a few).

It is also being increasingly used in new market segments such as the automotive industry, embedded and mobile as well as IoT. It is a platform of innovation that is consistently being updated to fit the new needs of computing with commits coming from developers across the world. We’ve experienced incredible community growth of 100 percent in the last five years. A lot of this growth has come from new geographic locations — most of the growth is from China and Ukraine.

Linux.com: How did the project notice that there might be an issue and how did people respond to this?

Lars Kurth: In mid 2014, maintainers started to notice that their review workload increased. At the same time, some contributors noticed that it took longer to get their changes upstreamed. We first developed some basic scripts to prove that the total elapsed time from first code review to commit had indeed increased. I then ran a number of surveys, to be able to form a working thesis on the root causes.

In terms of response, there were a lot of differing opinions on what exactly was causing the process to slow down. Some thought that we did not have enough maintainers, some thought we did not have enough committers, others felt that the maintainers were not coordinating reviews well enough, while others felt that newcomers wrote lower quality code or there could be cultural and language issues.

Community members made a lot of assumptions based on their own worst experiences, without facts to support them. There were so many contradictions among the group that we couldn’t identify a clear root cause for what we saw.

Linux.com: What were some of your initial ideas on how to improve this and why did you eventually choose to work with Bitergia for open analytics of the review process?

Lars Kurth: We first took a step back and looked at some things we could do that made sense without a ton of data. For example, I developed a training course for new contributors. I then did a road tour (primarily to Asia) to build personal relationships with new contributors and to deliver the new training.

In the year before, we started experimenting with design and architecture reviews for complex features. We decided to encourage these more without being overly prescriptive. We highlighted positive examples in the training material.

I also kicked off a number of surveys around our governance, to see whether maybe we have scalability issues. Unfortunately, we didn’t have any data to support this, and as expected different community members had different views. We did change our release cadence from 9-12 months to 6 months, to make it less painful for contributors if a feature missed a release.

It became increasingly clear that to make true progress, we would need reliable data. And to get that we needed to work with a software development analytics specialist. I had watched Bitergia for a while and made a proposal to the Xen Project Advisory Board to fund development of metrics collection tools for our code review process.  

Linux.com: How did you collect the data (including what tools you used) to get what you needed from the mailing list and Git repositories?

Lars Kurth: We used existing tools such as MLStats and CVSAnalY to collect mailing list and Git data. The challenge was to identify the different stages of a code review in the database that was generated by MLStats and to link it to the Git activity database generated by CVSAnalY. After that step we ended up with a combined code review database and ran statistical analysis over the combined database. Quite a bit of plumbing and filtering had to be developed from scratch for that to take place.

Linux.com: Were there any challenges that you experienced along the way?

Lars Kurth: First we had to develop a reasonably accurate model of the code review process. This was rather challenging, as e-mail is essentially unstructured. Also, I had to act as bridge between Bitergia, which implemented the tools and the community. This took a significant portion of time. However, without spending that time, it would have been quite likely that the project would fail.

To de-risk the project, we designed it in two phases: the first phase focused on statistical analysis that allowed us to test some theories; the second phase focused on improving accuracy of the tools and making the data accessible to community stakeholders.

Linux.com: What were your initial results from the analysis?

Lars Kurth: There were three key areas that we found were causing the slow down:

  • Huge growth in comment activity from 2013 to 2015

  • The time it took to merge patches (=time to merge) increased significantly from 2012 to the first half of 2014. However, from the second half of 2014  time to merge moved back to its long-term average. This was a strong indicator that the measures we took actually had an effect.

  • Complex patches were taking significantly longer to merge than small patches. As it turns out, a significant number of new features were actually rather complex. At the same time, the demands on the project to deliver better quality and security had also raised the bar for what could be accepted.  

Linux.com: How did the community respond to your data? How did you use it to help you make decisions about what was best to improve the process?

Lars Kurth: Most people were receptive to the data, but some were concerned that we were only able to match 60 percent of the code reviews to Git commits. For the statistical analysis, this was a big enough sample.

Further investigation showed that the main factor for this low match rate was caused by cross-posting of patches across FOSS communities. For example, some QEMU and Linux patches cross-posted for review on the Xen Project mailing lists, but the code did not end up in Xen. Once this was understood, a few key people in the community started to see the potential value of the new tools.

This is where stage two of the project came in. We defined a set of use cases and supporting data that broadly covered three areas:

  • Community use cases to encourage desired behavior: this would be metrics such as real review contributions (not justed  ACKed-by and Reviewed-by flags), comparing review activity against contributions.

  • Performance use cases that would allow us to spot issues early: these would allow us to filter time-related metrics by a number of different criteria such as complexity of a patch series

  • Backlog use cases to optimize process and focus: the intention here was to give contributors and maintainers tools to see what reviews are active, nearly complete, complete or stale.

Linux.com: How have you made improvements based on your findings and what have been the end results for you?

Lars Kurth: We had to iterate the use cases, the data supporting them and how the data is shown. I expect that that process will continue, as more community members use the tools. For example, we realized that the code review process dashboard that was developed as part of the project is also useful for vendors to estimate how long it will take to get something upstreamed based on past performance.

Overall, I am very excited about this project, and although the initial contract with Bitergia has ended, we have an Outreachy intern working with Bitergia and me on the tools over the summer.

Linux.com: How can this analysis support other projects with the similar code review processes?

Lars Kurth: I believe that projects like the Linux kernel and others that use e-mail based code review processes and Git should be able to use and build on our work. Hopefully, we will be able to create a basis for collaboration that helps different projects become more efficient and ultimately improve what we build.

Resources: