Today the Linux Foundation announced that it would be hosting seven projects that originated at Call for Code for Racial Justice, an initiative driven by IBM and Creator David Clark Cause to urge the global developer ecosystem and open source community to contribute to solutions that can help confront racial inequalities. 

Launched by IBM in October 2020, Call for Code for Racial Justice facilitates the adoption and innovation of open source projects by developers, ecosystem partners, and communities across the world to promote racial justice across three distinct focus areas: Police & Judicial Reform and Accountability; Diverse Representation; and Policy & Legislation Reform. 

The initiative builds upon Call for Code, created by IBM in 2018 and has grown to over 400,000 developers and problem solvers in 179 countries, in partnership with Creator David Clark Cause, Founding Partner IBM, Charitable Partner United Nations Human Rights, and the Linux Foundation.

As part of today’s announcement, the Linux Foundation and IBM unveiled two new solution starters, Fair Change and TakeTwo: 

Fair Change is a platform to help record, catalog, and access evidence of potentially racially charged incidents to enable transparency, reeducation, and reform as a matter of public interest and safety. For example, real-world video footage related to routine traffic stops, stop and search, or other scenarios may be recorded and accessed by the involved parties and authorities to determine whether the incidents were handled in a biased manner. Fair Change consists of a mobile application for iOS and Android built using React Native, an API for capturing data from various sources built using Node JS. It also includes a website with a geospatial map view of incidents built using Google Maps and React. Data can be stored in a cloud-hosted database and object-store. Visit the tutorial or project page to learn more. 

TakeTwo aims to help mitigate digital content bias, whether overt or subtle, focusing on text across news articles, headlines, web pages, blogs, and even code. The solution is designed to leverage directories of inclusive terms compiled by trusted sources like the Inclusive Naming Initiative, which the Linux Foundation and CNCF co-founded. The terminology is categorized to train an AI model to enhance its accuracy over time. TakeTwo is built using open source technologies, including Python, FastAPI, and Docker. The API can be run locally with a CouchDB backend database or IBM Cloudant database. IBM has already deployed TakeTwo within its existing IBM Developer tools that are used to publish new content produced by hundreds of IBMers each week. IBM is trialing TakeTwo for IBM Developer website content. Visit the tutorial or project page to learn more.

In addition to the two new solution starters, The Linux Foundation will now host five existing and evolving open source projects from Call for Code for Racial Justice:

  • Five-Fifths Voter: This web app empowers minorities to exercise their right to vote and ensures their voice is heard by determining optimal voting strategies and limiting suppression issues.
  • Legit-Info: Local legislation can significantly impact areas as far-reaching as jobs, the environment, and safety. Legit-Info helps individuals understand the legislation that shapes their lives.
  • Incident Accuracy Reporting System: This platform allows witnesses and victims to corroborate evidence or provide additional information from multiple sources against an official police report.
  • Open Sentencing: To help public defenders better serve their clients and make a stronger case, Open Sentencing shows racial bias in data such as demographics.
  • Truth Loop: This app helps communities simply understand the policies, regulations, and legislation that will impact them the most.  

These projects were built using open source technologies that include Red Hat OpenShift, IBM Cloud, IBM Watson, Blockchain ledger, Node.js, Vu.js, Docker, Kubernetes, and Tekton. The Linux Foundation and IBM ask developers and ecosystem partners to contribute to these solutions by testing, extending, implementing them, and adding their own diverse perspectives and expertise to make them even stronger. 

For more information and to begin contributing, please visit: 

https://developer.ibm.com/callforcode/racial-justice/get-started/

https://developer.ibm.com/callforcode/racial-justice/projects/  

https://www.linuxfoundation.org/projects/call-for-code/  

https://github.com/Call-for-Code-for-Racial-Justice/

Linux Foundation Logo

Today, the Linux Foundation announced that Renesas’ Hisao Munakata has been re-elected to its board, representing the Gold Member community. GitLab’s Eric Johnson has been elected to represent the Silver Member community. Linux Foundation elected board directors serve 2-year terms.

Directors elected to the Linux Foundation’s board are committed to building sustainable ecosystems around open collaboration to accelerate technology development and industry adoption. The Linux Foundation expands the open collaboration communities it supports with community efforts focused on building open standards, open hardware, and open data. It is dedicated to improving diversity in open source communities and working on processes, tools, and best security practices in open development communities. 

Hisao Munakata, Renesas (Gold Member)

Renesas is a global semiconductor manufacturer that provides cutting-edge SoC (system-on-chip) devices for the automotive, industry, and infrastructure. As open source support became essential for the company, Munakata-san encouraged Renesas developers to follow an “upstream-first” scheme to minimize gaps from the mainline community codebase. The industry has now accepted this as standard practice, following Renesas’ direction and pioneering work. 

Hisao Munakata

Munakata-san has served as an LF board director since 2019 and has reflected the voice from the embedded industry. 

Renesas, which joined the Linux Foundation in 2011, has ranked in the top twelve kernel development contributor firms in the past 14 years. Munakata-san serves pivotal roles in various LF projects such as the AGL (Automotive Grade Linux) Advisory Board, Yocto Project Advisory Board, Core Embedded Linux Project, and OpenSSF. In these roles, Munakata-san has supported many industry participants in these projects to achieve harmony. 

As cloud-native trends break barriers between enterprise and embedded systems, Munakata-san seeks to improve close collaboration across the industry and increase contribution from participants in the embedded systems space, focusing on safety in a post-COVID world.

Eric Johnson, GitLab (Silver Member)

Eric Johnson is the Chief Technology Officer at GitLab, Inc. — the first single application for the DevSecOps lifecycle. GitLab is a free, open core software used by more than 30 million registered users to collaborate, author, test, secure, and release software quickly and efficiently. 

Eric Johnson

At GitLab, Eric is responsible for the organization that integrates the work of over a hundred external open source contributors into GitLab’s codebase every month. During his tenure Eric has contributed to a 10x+ increase in annual recurring revenue and has scaled Engineering from 100 to more than 550 people while dramatically increasing team diversity in gender, ethnicity, and country-of-residence. He’s also helped turn GitLab, Inc. into one of the most productive engineering organizations in the world, as evidenced by their substantial monthly on-premise releases.

Eric is also a veteran of 4 previous enterprise technology startups in fields as varied as marketing technology, localization software, streaming video, and commercial drone hardware/software. He currently advises two startups in the medical trial software and recycling robotics industries. 

Eric brings his open source and Linux background to the Foundation. In his professional work, he has spent 17 years hands-on or managing teams that develop software that runs on Linux systems, administrating server clusters, orchestrating containers, open-sourcing privately built software, and contributing back to open source projects. Personally, he’s also administered a Linux home server for the past ten years.

As a Linux Foundation board member, Eric looks forward to using his execution-focused executive experience to turn ideas into results. Collaboration with the Linux Foundation has already begun with Distributed Developer ID and Digital Bill of Materials (DBoM). As a remote work expert with years of experience developing best practices, Eric will use his expertise to help the board, the Foundation, and its partners become more efficient in a remote, asynchronous, and geographically distributed way.

Click here to read the February 2021 Linux Foundation Newsletter

Throughout the modern business era, industries and commercial operations have shifted substantially to digital processes. Whether you look at EDI as a means to exchange invoices or cloud-based billing and payment solutions today, businesses have steadily been moving towards increasing digital operations. In the last few years, we’ve seen the promises of digital transformation come alive, particularly in industries that have shifted to software-defined models. The next step of this journey will involve enabling digital transactions through decentralized networks. 

A fundamental adoption issue will be figuring out who controls and decides how a decentralized network is governed. It may seem oxymoronic at first, but decentralized networks still need governance. A future may hold autonomously self-governing decentralized networks, but this model is not accepted in industries today. The governance challenge with a decentralized network technology lies in who and how participants in a network will establish and maintain policies, network operations, on/offboarding of participants, setting fees, configurations, and software changes and are among the issues that will have to be decided to achieve a successful network. No company wants to participate or take a dependency on a network that is controlled or run by a competitor, potential competitor, or any single stakeholder at all for that matter. 

Earlier this year, we presented a solution for Open Governance Networks that enable an industry or ecosystem to govern itself in an open, inclusive, neutral, and participatory model. You may be surprised to learn that it’s based on best practices in open governance we’ve developed over decades of facilitating the world’s most successful and competitive open source projects.

The Challenge

For the last few years, a running technology joke has been “describe your problem, and someone will tell you blockchain is the solution.” There have been many other concerns raised and confusion created, as overnight headlines hyped cryptocurrency schemes. Despite all this, behind the scenes, and all along, sophisticated companies understood a distributed ledger technology would be a powerful enabler for tackling complex challenges in an industry, or even a section of an industry. 

At the Linux Foundation, we focused on enabling those organizations to collaborate on open source enterprise blockchain technologies within our Hyperledger community. That community has driven collaboration on every aspect of enterprise blockchain technology, including identity, security, and transparency. Like other Linux Foundation projects, these enterprise blockchain communities are open, collaborative efforts. We have had many vertical industry participants engage, from retail, automotive, aerospace, banking, and others participate with real industry challenges they needed to solve. And in this subset of cases, enterprise blockchain is the answer.

The technology is ready. Enterprise blockchain has been through many proof-of-concept implementations, and we’ve already seen that many organizations have shifted to production deployments. A few notable examples are:

  • Trust Your Supplier Network 25 major corporate members from Anheuser-Busch InBev to UPS In production since September 2019. 
  • Foodtrust Launched Aug 2017 with ten members, now being used by all major retailers. 
  • Honeywell 50 vendors with storefronts in the new marketplace. In its first year, GoDirect Trade processed more than $5 million in online transactions.

However, just because we have the technology doesn’t mean we have the appropriate conditions to solve adoption challenges. A certain set of challenges about networks’ governance have become a “last mile” problem for industry adoption. While there are many examples of successful production deployments and multi-stakeholder engagements for commercial enterprise blockchains already, specific adoption scenarios have been halted over uncertainty, or mistrust, over who and how a blockchain network will be governed. 

To precisely state the issue, in many situations, company A does not want to be dependent on, or trust, company B to control a network. For specific solutions that require broad industry participation to succeed, you can name any industry, and there will be company A and company B. 

We think the solution to this challenge will be Open Governance Networks.

The Linux Foundation vision of the Open Governance Network

An Open Governance Network is a distributed ledger service, composed of nodes, operated under the policies and directions of an inclusive set of industry stakeholders. 

Open Governance Networks will set the policies and rules for participation in a decentralized ledger network that acts as an industry utility for transactions and data sharing among participants that have permissions on the network. The Open Governance Network model allows any organization to participate. Those organizations that want to be active in sharing the operational costs will benefit from having a representative say in the policies and rules for the network itself. The software underlying the Open Governance Network will be open source software, including the configurations and build tools so that anyone can validate whether a network node complies with the appropriate policies.

Many who have worked with the Linux Foundation will realize an open, neutral, and participatory governance model under a nonprofit structure that has already been thriving for decades in successful open source software communities. All we’re doing here is taking the same core principles of what makes open governance work for open source software, open standards, and open collaboration and applying those principles to managing a distributed ledger. This is a model that the Linux Foundation has used successfully in other communities, such as the Let’s Encrypt certificate authority.

Our ecosystem members trust the Linux Foundation to help solve this last mile problem using open governance under a neutral nonprofit entity. This is one solution to the concerns about neutrality and distributed control. In pan-industry use cases, it is generally not acceptable for one participant in the network to have power in any way that could be used as an advantage over someone else in the industry.  The control of a ledger is a valuable asset, and competitive organizations generally have concerns in allowing one entity to control this asset. If not hosted in a neutral environment for the community’s benefit, network control can become a leverage point over network users.  

We see this neutrality of control challenge as the primary reason why some privately held networks have struggled to gain widespread adoption. In order to encourage participation, industry leaders are looking for a neutral governance structure, and the Linux Foundation has proven the open governance models accomplish that exceptionally well.

This neutrality of control issue is very similar to the rationale for public utilities. Because the economic model mirrors a public utility, we debated calling these “industry utility networks.” In our conversations, we have learned industry participants are open to sharing the cost burden to stand up and maintain a utility. Still, they want a low-cost, not profit-maximizing model. That is why our nonprofit model makes the most sense.

It’s also not a public utility in that each network we foresee today would be restricted in participation to those who have a stake in the network, not any random person in the world. There’s a layer of human trust that our communities have been enabling on top of distributed networks, which started with the Trust over IP Foundation

Unlike public cryptocurrency networks where anyone can view the ledger or submit proposed transactions, industries have a natural need to limit access to legitimate parties in their industry. With minor adjustments to address the need for policies for transactions on the network, we believe a similar governance model applied to distributed ledger ecosystems can resolve concerns about the neutrality of control. 

Understanding LF Open Governance Networks

Open Governance Networks can be reduced to the following building block components:

  • Business Governance: Networks need a decision-making body to establish core policies (e.g., network policies), make funding and budget decisions, contracting with a network manager, and other business matters necessary for the network’s success. The Linux Foundation establishes a governing board to manage the business governance.
  • Technical Governance: Networks will require software. A technical open source community will openly maintain the software, specifications, or configuration decisions implemented by the network nodes. The Linux Foundation establishes a technical steering committee to oversee technical projects, configurations, working groups, etc.
  • Transaction Entity: Networks will require a transaction entity that will a) act as counterparty to agreements with parties transacting on the network, b) collect fees from participants, and c) execute contracts for operational support (e.g., hiring a network manager).

Of these building blocks, the Linux Foundation already offers its communities the Business and Technical Governance needed for Open Governance Networks. The final component is the new, LF Open Governance Networks. 

LF Open Governance Networks will enable our communities to establish their own Open Governance Network and have an entity to process agreements and collect transaction fees. This new entity is a Delaware nonprofit, a nonstock corporation that will maximize utility and not profit. Through agreements with the Linux Foundation, LF Governance Networks will be available to Open Governance Networks hosted at the Linux Foundation. 

If you’re interested in learning more about hosting an Open Governance Network at the Linux Foundation, please contact us at governancenetworks@linuxfoundation.org

Jason Perlow, Director of Project Insights and Editorial Content at the Linux Foundation, had an opportunity to speak with Shuah Khan about her experiences as a woman in the technology industry. She discusses how mentorship can improve the overall diversity and makeup of open source projects, why software maintainers are important for the health of open source projects such as the Linux kernel, and how language inclusivity and codes of conduct can improve relationships and communication between software maintainers and individual contributors.

JP: So, Shuah, I know you wear many different hats at the Linux Foundation. What do you call yourself around here these days?

SK: <laughs> Well, I primarily call myself a Kernel Maintainer & Linux Fellow. In addition to that, I focus on two areas that are important to the continued health and sustainability of the open source projects in the Linux ecosystem. The first one is bringing more women into the Kernel community, and additionally, I am leading the mentorship program efforts overall at the Linux Foundation. And in that role, in addition to the Linux Kernel Mentorship, we are looking at how the Linux Foundation mentorship program is working overall, how it is scaling. I make sure the LFX Mentorship platform scales and serves diverse mentees and mentors’ needs in this role. 

The LF mentorships program includes several projects in the Linux kernel, LFN, HyperLedger, Open MainFrame, OpenHPC, and other technologies. The Linux Foundation’s Mentorship Programs are designed to help developers with the necessary skills–many of whom are first-time open source contributors–experiment, learn, and contribute effectively to open source communities. 

The mentorship program has been successful in its mission to train new developers and make these talented pools of prospective employees trained by experts to employers. Several graduated mentees have found jobs. New developers have improved the quality and security of various open source projects, including the Linux kernel. Several Linux kernel bugs were fixed, a new subsystem mentor was added, and a new driver maintainer is now part of the Linux kernel community. My sincere thanks to all our mentors for volunteering to share their expertise.

JP: How long have you been working on the Kernel?

SK: Since 2010, or 2011, I got involved in the Android Mainlining project. My first patch removed the Android pmem driver.

JP: Wow! Is there any particular subsystem that you specialize in?

SK: I am a self described generalist. I maintain the kernel self-test subsystem, the USB over IP driver, usbip tool, and the cpupower tool. I contributed to the media subsystem working on Media Controller Device Allocator API to resolve shared device resource management problems across device drivers from different subsystems.

JP: Hey, I’ve actually used the USB over IP driver when I worked at Microsoft on Azure. And also, when I’ve used AWS and Google Compute. 

SK: It’s a small niche driver used in cloud computing. Docker and other containers use that driver heavily. That’s how they provide remote access to USB devices on the server to export devices to be imported by other systems for use.

JP: I initially used it for IoT kinds of stuff in the embedded systems space. Were you the original lead developer on it, or was it one of those things you fell into because nobody else was maintaining it?

SK: Well, twofold. I was looking at USB over IP because I like that technology. it just so happened the driver was brought from the staging tree into the Mainline kernel, I volunteered at the time to maintain it. Over the last few years, we discovered some security issues with it, because it handles a lot of userspace data, so I had a lot of fun fixing all of those. <laugh>.

JP: What drew you into the Linux operating system, and what drew you into the kernel development community in the first place?

SK: Well, I have been doing kernel development for a very long time. I worked on the LynxOS RTOS, a while back, and then HP/UX, when I was working at HP, after which I transitioned into  doing open source development — the OpenHPI project, to support HP’s rack server hardware, and that allowed me to work much more closely with Linux on the back end. And at some point, I decided I wanted to work with the kernel and become part of the Linux kernel community. I started as an independent contributor.

JP: Maybe it just displays my own ignorance, but you are the first female, hardcore Linux kernel developer I have ever met. I mean, I had met female core OS developers before — such as when I was at Microsoft and IBM — but not for Linux. Why do you suppose we lack women and diversity in general when participating in open source and the technology industry overall?

SK: So I’ll answer this question from my perspective, from what I have seen and experienced, over the years. You are right; you probably don’t come across that many hardcore women Kernel developers. I’ve been working professionally in this industry since the early 1990s, and on every project I have been involved with, I am usually the only woman sitting at the table. Some of it, I think, is culture and society. There are some roles that we are told are acceptable to women — even me, when I was thinking about going into engineering as a profession. Some of it has to do with where we are guided, as a natural path. 

There’s a natural resistance to choosing certain professions that you have to overcome first within yourself and externally. This process is different for everybody based on their personality and their origin story. And once you go through the hurdle of getting your engineering degree and figuring out which industry you want to work in, there is a level of establishing credibility in those work environments you have to endure and persevere. Sometimes when I would walk into a room, I felt like people were looking at me and thinking, “why is she here?” You aren’t accepted right away, and you have to overcome that as well. You have to go in there and say, “I am here because I want to be here, and therefore, I belong here.” You have to have that mindset. Society sends you signals that “this profession is not for me” — and you have to be aware of that and resist it. I consider myself an engineer that happens to be a woman as opposed to a woman engineer.

JP: Are you from India, originally?

SK: Yes.

JP: It’s funny; my wife really likes this Netflix show about matchmaking in India. Are you familiar with it?

SK: <laughs> Yes I enjoyed the series, and A Suitable Girl documentary film that follows three women as they navigate making decisions about their careers and family obligations.

JP: For many Americans, this is our first introduction to what home life is like for Indian people. But many of the women featured on this show are professionals, such as doctors, lawyers, and engineers. And they are very ambitious, but of course, the family tries to set them up in a marriage to find a husband for them that is compatible. As a result, you get to learn about the traditional values and roles they still want women to play there — while at the same time, many women are coming out of higher learning institutions in that country that are seeking technical careers. 

SK: India is a very fascinatingly complex place. But generally speaking, in a global sense, having an environment at home where your parents tell you that you may choose any profession you want to choose is very encouraging. I was extremely fortunate to have parents like that. They never said to me that there was a role or a mold that I needed to fit into. They have always told me, “do what you want to do.” Which is different; I don’t find that even here, in the US. Having that support system, beginning in the home to tell you, “you are open to whatever profession you want to choose,” is essential. That’s where a lot of the change has to come from. 

JP: Women in technical and STEM professions are becoming much more prominent in other countries, such as China, Japan, and Korea. For some reason, in the US, I tend to see more women enter the medical profession than hard technology — and it might be a level of effort and perceived reward thing. You can spend eight years becoming a medical doctor or eight years becoming a scientist or an engineer, and it can be equally difficult, but the compensation at the end may not be the same. It’s expensive to get an education, and it takes a long time and hard work, regardless of the professional discipline.

SK: I have also heard that women also like to enter professions where they can make a difference in the world — a human touch, if you will. So that may translate to them choosing careers where they can make a larger impact on people — and they may view careers in technology as not having those same attributes. Maybe when we think about attracting women to technology fields, we might have to promote technology aspects that make a difference. That may be changing now, such as the LF Public Health (LFPH) project we kicked off last year. And with LF AI & Data Foundation, we are also making a difference in people’s lives, such as detecting earthquakes or analyzing climate change. If we were to promote projects such as these, we might draw more women in.

JP: So clearly, one of the areas of technology where you can make a difference is in open source, as the LF is hosting some very high-concept and existential types of projects such as LF Energy, for example — I had no idea what was involved in it and what its goals were until I spoke to Shuli Goodman in-depth about it. With the mentorship program, I assume we need this to attract fresh talent — because as folks like us get older and retire, and they exit the field, we need new people to replace them. So I assume mentorship, for the Linux Foundation, is an investment in our own technologies, correct?

SK: Correct. Bringing in new developers into the fold is the primary purpose, of course — and at the same time, I view the LF as taking on mentorship provides that neutral, level playing field across the industry for all open source projects. Secondly, we offer a self-service platform, LFX Mentorship, where anyone can come in and start their project. So when the COVID-19 pandemic began, we expanded this program to help displaced people — students, et cetera, and less visible projects. Not all projects typically get as much funding or attention as others do — such as a Kubernetes or  Linux kernel — among the COVID mentorship program projects we are funding. I am particularly proud of supporting a climate change-related project, Using Machine Learning to Predict Deforestation.

The self-service approach allows us to fund and add new developers to projects where they are needed. The LF mentorships are remote work opportunities that are accessible to developers around the globe. We see people sign up for mentorship projects from places we haven’t seen before, such as Africa, and so on, thus creating a level playing field. 

The other thing that we are trying to increase focus on is how do you get maintainers? Getting new developers is a starting point, but how do we get them to continue working on the projects they are mentored on? As you said, someday, you and I and others working on these things are going to retire, maybe five or ten years from now. This is a harder problem to solve than training and adding new developers to the project itself.

JP: And that is core to our software supply chain security mission. It’s one thing to have this new, flashy project, and then all these developers say, “oh wow, this is cool, I want to join that,” but then, you have to have a certain number of people maintaining it for it to have long-term viability. As we learned in our FOSS study with Harvard, there are components in the Linux operating system that are like this. Perhaps even modules within the kernel itself, I assume that maybe you might have only one or two people actively maintaining it for many years. And what happens if that person dies or can no longer work? What happens to that code? And if someone isn’t familiar with that code, it might become abandoned. That’s a serious problem in open source right now, isn’t it?

SK: Right. We have seen that with SSH and other security-critical areas. What if you don’t have the bandwidth to fix it? Or the money to fix it? I ended up volunteering to maintain a tool for a similar reason when the maintainer could no longer contribute regularly. It is true; we have many drivers where maintainer bandwidth is an issue in the kernel. So the question is, how do we grow that talent pool?

JP: Do we need a job board or something? We need X number of maintainers. So should we say, “Hey, we know you want to join the kernel project as a contributor, and we have other people working on this thing, but we really need your help working on something else, and if you do a good job, we know tons of companies willing to hire developers just like you?” 

SK: With the kernel, we are talking about organic growth; it is just like any other open source project. It’s not a traditional hire and talent placement scenario. Organically they have to have credibility, and they have to acquire it through experience and relationships with people on those projects. We just talked about it at the previous Linux Plumbers Conference, we do have areas where we really need maintainers, and the MAINTAINERS file does show areas where they need help. 

To answer your question, it’s not one of those things where we can seek people to fill that role, like LinkedIn or one of the other job sites. It has to be an organic fulfillment of that role, so the mentorship program is essential in creating those relationships. It is the double-edged sword of open source; it is both the strength and weakness. People need to have an interest in becoming a maintainer and also a commitment to being one, long term.

JP: So, what do you see as the future of your mentorship and diversity efforts at the Linux Foundation? What are you particularly excited about that is forthcoming that you are working on?

SK: I view the Linux Foundation mentoring as a three-pronged approach to provide unstructured webinars, training courses, and structured mentoring programs. All of these efforts combine to advance a diverse, healthy, and vibrant open source community. So over the past several months, we have been morphing our speed mentorship style format into an expanded webinar format — the LF Live Mentorship series. This will have the function of growing our next level of expertise. As a complement to our traditional mentorship programs, these are webinars and courses that are an hour and a half long that we hold a few times a month that tackle specific technical areas in software development. So it might cover how to write great commit logs, for example, for your patches to be accepted, or how to find bugs in C code. Commit logs are one of those things that are important to code maintenance, so promoting good documentation is a beneficial thing. Webinars provide a way for experts short on time to share their knowledge with a few hours of time commitment and offer a self-paced learning opportunity to new developers.

Additionally, I have started the Linux Kernel Mentorship forum for developers and their mentors to connect and interact with others participating in the Linux Kernel Mentorship program and graduated mentees to mentor new developers. We kicked off Linux Kernel mentorship Spring 2021 and are planning for Summer and Fall.

A big challenge is we are short on mentors to be able to scale the structured program. Solving the problem requires help from LF member companies and others to encourage their employees to mentor, “it takes a village,” they say.

JP: So this webinar series and the expanded mentorship program will help developers cultivate both hard and soft skills, then.

SK: Correct. The thing about doing webinars is that if we are talking about this from a diversity perspective, they might not have time for a full-length mentorship, typically like a three-month or six-month commitment. This might help them expand their resources for self-study. When we ask for developers’ feedback about what else they need to learn new skill sets, we hear that they don’t have resources, don’t have time to do self-study, and learn to become open source developers and software maintainers. This webinar series covers general open source software topics such as the Linux kernel and legal issues. It could also cover topics specific to other LF projects such as CNCF, Hyperledger, LF Networking, etc.

JP: Anything else we should know about the mentorship program in 2021?

SK: In my view,  attracting diversity and new people is two-fold. One of the things we are working on is inclusive language. Now, we’re not talking about curbing harsh words, although that is a component of what we are looking at. The English you and I use in North America isn’t the same English used elsewhere. As an example, when we use North American-centric terms in our email communications, such as when a maintainer is communicating on a list with people from South Korea, something like “where the rubber meets the road” may not make sense to them at all. So we have to be aware of that.

JP: I know that you are serving on the Linux kernel Code of Conduct Committee and actively developing the handbook. When I first joined the Linux Foundation, I learned what the Community Managers do and our governance model. I didn’t realize that we even needed to have codes of conduct for open source projects. I have been covering open source for 25 years, but I come out of the corporate world, such as IBM and Microsoft. Codes of Conduct are typically things that the Human Resources officer shows you during your initial onboarding, as part of reviewing your employee manual. You are expected to follow those rules as a condition of employment. 

So why do we need Codes of Conduct in an open source project? Is it because these are people who are coming from all sorts of different backgrounds, companies, and ways of life, and may not have interacted in this form of organized and distributed project before? Or is it about personalities, people interacting with each other over long distance, and email, which creates situations that may arise due to that separation?

SK: Yes, I come out of the corporate world as well, and of course, we had to practice those codes of conduct in that setting. But conduct situations arise that you have to deal with in the corporate world. There are always interpersonal scenarios that can be difficult or challenging to work with — the corporate world isn’t better than the open source world in that respect. It is just that all of that happens behind a closed setting.

But there is no accountability in the open source world because everyone participates out of their own free will. So on a small, traditional closed project, inside the corporate world, where you might have 20 people involved, you might get one or two people that could be difficult to work with. The same thing happens and is multiplied many times in the open source community, where you have hundreds of thousands of developers working across many different open source projects. 

The biggest problem with these types of projects when you encounter situations such as this is dealing with participation in public forums. In the corporate world, this can be addressed in private. But on a public mailing list, if you are being put down or talked down to, it can be extremely humiliating. 

These interactions are not always extreme cases; they could be simple as a maintainer or a lead developer providing negative feedback — so how do you give it? It has to be done constructively. And that is true for all of us.

JP: Anything else?

SK: In addition to bringing our learnings and applying this to the kernel project, I am also doing this on the ELISA project, where I chair the Technical Steering Committee, where I am bridging communication between experts from the kernel and the safety communities. To make sure we can use the kernel the best ways in safety-critical applications, in the automotive and medical industry, and so on. Many lessons can be learned in terms of connecting the dots, defining clearly what is essential to make Linux run effectively in these environments, in terms of dependability. How can we think more proactively instead of being engaged in fire-fighting in terms of security or kernel bugs? As a result of this, I am also working on any necessary kernel changes needed to support these safety-critical usage scenarios.

JP: Before we go, what are you passionate about besides all this software stuff? If you have any free time left, what else do you enjoy doing?

SK: I read a lot. COVID quarantine has given me plenty of opportunities to read. I like to go hiking, snowshoeing, and other outdoor activities. Living in Colorado gives me ample opportunities to be in nature. I also like backpacking — while I wasn’t able to do it last year because of COVID — I like to take backpacking trips with my son. I also love to go to conferences and travel, so I am looking forward to doing that again as soon as we are able.

Talking about backpacking reminded me of the two-day, 22-mile backpacking trip during the summer of 2019 with my son. You can see me in the picture above at the end of the road, carrying a bearbox, sleeping bag, and hammock. It was worth injuring my foot and hurting in places I didn’t even know I had.

JP: Awesome. I enjoyed talking to you today. So happy I finally got to meet you virtually.

The Open Source Security Foundation (OpenSSF) officially launched on August 3, 2020. In this article, we’ll look at why the OpenSSF was formed, what it’s accomplished in its first six months, and its plans for the future.

The world depends on open source software (OSS), so OSS security is vital. Various efforts have been created to help improve OSS security. These efforts include the Core Infrastructure Initiative (CII) in the Linux Foundation, the Open Source Security Coalition (OSSC) founded by the GitHub Security Lab, and the Joint Open Source Software Initiative (JOSSI) founded by Google and others.

It became apparent that progress would be easier if these efforts merged into a single effort. The OpenSSF was created in 2020 as a merging of these three groups into “a cross-industry collaboration that brings together leaders to improve the security of open source software (OSS).”

The OpenSSF has certainly gained that “cross-industry collaboration”; its dozens of members include (alphabetically) Canonical, GitHub, Google, IBM, Intel, Microsoft, and Red Hat. Its governing board also includes a Security Community Individual Representative to represent those not represented in other ways specifically. It’s also created some structures to help people work together: it’s established active working groups, identified (and posted) its values, and agreed on its technical vision.

But none of that matters unless they actually produce results. It’s still early, but they already have several accomplishments. They have released:

  • Secure Software Development Fundamentals courses. This set of 3 freely-available courses on the edX platform is for software developers to learn to develop secure software. It focuses on practical steps that any software developer can easily take, not theory or actions requiring unlimited resources.  Developers can also pay a fee to take tests to attempt to earn certificates to prove they understand the material.
  • Security Scorecards. This auto-generates a “security score” for open source projects to help users as they decide the trust, risk, and security posture for their use case.
  • Criticality Score. This project auto-generates a criticality score for open source projects based on a number of parameters. The goal is to better understand the most critical open source projects the world depends on.  
  • Security metrics dashboard. This early-release work provides a dashboard of security and sustainment information about OSS projects by combining the Security ScoreCards, CII Best Practices, and other data sources.
  • OpenSSF CVE Benchmark. This benchmark consists of vulnerable code and metadata for over 200 historical JavaScript/TypeScript vulnerabilities (CVEs). This will help security teams evaluate different security tools on the market by enabling teams to determine false positive and false negative rates with real codebases instead of synthetic test code.
  • OWASP Security Knowledge Framework (SKF). In collaboration with OWASP, this work is a knowledge base that includes projects with checklists and best practice code examples in multiple programming languages. It includes training materials for developers on how to write secure code in specific languages and security labs for hands-on work.
  • Report on the 2020 FOSS Contributor Survey, The OpenSSF and the Laboratory for Innovation Science at Harvard (LISH) released a report that details the findings of a contributor survey to study and identify ways to improve OSS security and sustainability. There were nearly 1,200 respondents.

The existing CII Best Practices badge project has also been folded into the OpenSSF and continues to be improved. The project now has more Chinese translators, a new ongoing Swahili translation, and various small refinements that clarify the badging requirements.

The November 2020 OpenSSF Town Hall discussed the OpenSSF’s ongoing work. The OpenSSF currently has the following working groups:

  • Vulnerability Disclosures
  • Security Tooling
  • Security Best Practices
  • Identifying Security Threats to Open Source Projects (focusing on a metrics dashboard)
  • Securing Critical Projects
  • Digital Identity Attestation

Future potential work, other than continuously improving work already released, includes:

  • Identifying overlapping and related security requirements in various specifications to reduce duplicate effort. This is to be developed in collaboration with OWASP as lead and is termed the Common Requirements Enumeration (CRE). The CRE is to “link sections of standard[s] and guidelines to each other, using a mutual topic identifier, enabling standard and scheme makers to work efficiently, enabling standard users to find the information they need, and attaining a shared understanding in the industry of what cyber security is.” [Source: “Common Requirements Enumeration”]
  • Establishing a website for no-install access to a security metrics OSS dashboard. Again, this will provide a single view of data from multiple data sources, including the Security Scorecards and CII Best Practices.
  • Developing improved identification of critical OSS projects. Harvard and the LF have previously worked to identify critical OSS projects. In the coming year, they will refine their approaches and add new data sources to identify critical OSS projects better.
  • Funding specific critical OSS projects to improve their security. The expectation is that this will focus on critical OSS projects that are not otherwise being adequately funded and will work to improve their overall sustainability.
  • Identifying and implementing improved, simplified techniques for digitally signing commits and verifying those identity attestations.

As with all Linux Foundation projects, the work by the OpenSSF is decided by its participants. If you are interested in the security of the OSS we all depend on, check out the OpenSSF and participate in some way. The best way to get involved is to attend the working group meetings — they are usually every other week and very casual. By working together we can make a difference. For more information, see https://openssf.org

David A. Wheeler, Director of Open Source Supply Chain Security at the Linux Foundation

Click here to read the January 2021 Linux Foundation Newsletter

Preventing Supply Chain Attacks like SolarWinds

In late 2020, it was revealed that the SolarWinds Orion software, which is in use by numerous US Government agencies and many private organizations, was severely compromised. This was an incredibly dangerous set of supply chain compromises that the information technology community (including the Open Source community) needs to learn from and take action on.

The US Cybersecurity and Infrastructure Security Agency (CISA) released an alert noting that the SolarWinds Orion software included malicious functionality in March 2020, but it was not detected until December 2020. CISA’s Emergency Directive 21-01 stated that it was being exploited, had a high potential of compromise, and a grave impact on entire organizations when compromised. Indeed, because Orion deployments typically control networks of whole organizations, this is a grave problem. The more people look, the worse it gets. As I write this, it appears that a second and third malware have been identified in Orion.

Why the SolarWinds Attack Is Particularly Noteworthy

What’s especially noteworthy is how the malicious code was inserted into Orion: the attackers subverted something called the build environment. When software is being developed it is converted (compiled) from source code (the text that software developers update) into an executable package using a “build process.” For example, the source code of many open source software projects is then used in software that is built, compiled, and redistributed by other organizations, so that it is ready to install and run on various computing platforms. In the case of SolarWinds’ Orion, CrowdStrike found a piece of malware called Sunspot that watched the build server for build commands and silently replaced source code files inside the Orion app with files that loaded the Sunburst malware. The SolarWinds Orion compromise by Sunspot isn’t the first example of these kinds of attacks, but it has demonstrated just how dangerous they can be when they compromise widely-used software.

Unfortunately, a lot of conventional security advice cannot counter this kind of attack: 

SolarWinds’ Orion is not open source software. Only the company’s developers can legally review, modify, or redistribute its source code or its build system and configurations. If we needed further evidence that obscurity of software source code doesn’t automatically provide security, this is it.

Recommendations from The Linux Foundation 

Organizations need to harden their build environments against attackers. SolarWinds followed some poor practices, such as using the insecure ftp protocol and publicly revealing passwords, which may have made these attacks especially easy. The build system is a critical production system, and it should be treated like one, with the same or higher security requirements as its production environments. This is an important short-term step that organizations should already be doing. However, it’s not clear that these particular weaknesses were exploited or that such hardening would have made any difference. Assuming a system can “never be broken into” is a failing strategy.

In the longer term, I know of only one strong countermeasure for this kind of attack: verified reproducible builds. A “reproducible build” is a build that always produces the same outputs given the same inputs so that the build results can be verified. A verified reproducible build is a process where independent organizations produce a build from source code and verify that the built results come from the claimed source code. Almost all software today is not reproducible, but there’s work to change this. The Linux Foundation and Civil Infrastructure Platform has been funding work, including the Reproducible Builds project, to make it possible to have verified reproducible builds.

The software industry needs to begin shifting towards implementing and requiring verified reproducible builds. This will not be easy. Most software is not designed to be reproducible in their build environments today, so it may take years to make software reproducible. Many changes must be made to make software reproducible, so resources (time and money) are often needed. And there’s a lot of software that needs to be reproducible, including operating system packages and library level packages. There are package distribution systems that would need to be reviewed and likely modified. I would expect some of the most critical software to become reproducible first, and then less critical software would increase over time as pressure increases to make more software verified reproducible. It would be wise to develop widely-applicable standards and best practices for creating reproducible builds. Once software is reproducible, others will need to verify the build results for given source code to counter these kinds of attacks. Reproducible builds are much easier for open source software (OSS) because there’s no legal impediment to having many verifiers. Closed source software developers will have added challenges; their business models often depend on hiding source code. It’s still possible to have “trusted rebuilders” worldwide to verify closed source software, even though it’s more challenging and the number of rebuilders would necessarily be smaller.

The information technology industry is generally moving away from “black boxes” that cannot be inspected and verified and towards components that can be reviewed. So this is part of a general industry trend; it’s a trend that needs to be accelerated.

This is not unprecedented. Auditors have access to the financial data and review the financial systems of most enterprises. Audits are an independent entity verifying the data and systems for the benefit of the ecosystem. There is a similar opportunity for organizations to become independent verifiers for both open source and closed source software and build systems. 

Attackers will always take the easiest path, so we can’t ignore other attacks. Today most attacks exploit unintentional vulnerabilities in code, so we need to continue to work to prevent these unintentional vulnerabilities. These mitigations include changing tools & interfaces so those problems won’t happen, educating developers on developing secure software (such as the free courses from OpenSSF on edX), and detecting residual vulnerabilities before deployment through various detection tools. The Open Source Security Foundation (OpenSSF) is working on improving the security of open source software (OSS), including all these points.

Applications are mostly reused software (with a small amount of custom code), so this reused software’s software supply chain is critical. Reused components are often extremely out-of-date. Thus, they have many publicly-known unintentional vulnerabilities; in fact, reused components with known vulnerabilities are among the topmost common problems in web applications. The LF’s LFX security tools, GitHub’s Dependabot, GitLab’s dependency analyzers, and many other tools & services can help detect reused components with known vulnerabilities.

Vulnerabilities in widely-reused OSS can cause widespread problems, so the LF is already working to identify such OSS so that it can be reviewed and hardened further (see Vulnerabilities in the Core Preliminary Report and Census II of Open Source Software).

The supply chain matters for malicious code, too; most malicious code gets into applications through library “typosquatting” (that is, by creating a malicious library with a name that looks like a legitimate library). 

That means that Users need to start asking for a software bill of materials (SBOM) so they will know what they are using. The US National Telecommunications and Information Administration (NTIA) has been encouraging the adoption of SBOMs throughout organizations and the software supply chain process. The Linux Foundation’s Software Package Data Exchange (SPDX) format is a SBOM format by many. Once you get SBOM information, examine the versions that are included. If the software has malicious components, or components with known vulnerabilities, start asking why. Some vulnerabilities may not be exploitable, but too many application developers simply don’t update dependencies even when they are exploitable. To be fair, there’s a chicken-and-egg problem here: specifications are in the process of being updated, tools are in development, and many software producers aren’t ready to provide SBOMs.  So users should not expect that most software producers will have SBOMs ready today. However, they do need to create a demand for SBOMs.

Similarly, software producers should work towards providing SBOM information. For many OSS projects this can typically be done, at least in part, by providing package management information that identifies their direct and indirect dependencies (e.g., in package.json, requirements.txt, Gemfile, Gemfile.lock, and similar files). Many tools can combine this information to create more complete SBOM information for larger systems.

Organizations should invest in OpenChain conformance and require their suppliers to implement a process designed to improve trust in a supply chain.  OpenChain’s conformance process reveals specifics about the components you depend on that are a critical first step to countering many supply chain attacks.

Conclusion

The attack on SolarWinds’ Orion will have devastating effects for years to come. But we can and should learn from it. 

We can:

  1. Harden software build environments
  2. Move towards verified reproducible builds 
  3. Change tools & interfaces so unintentional vulnerabilities are less likely
  4. Educate developers (such as the free courses from OpenSSF on edX)
  5. Use vulnerability detection tools when developing software
  6. Use tools to detect known-vulnerable components when developing software
  7. Improve widely-used OSS (the OpenSSF is working on this)
  8. Ask for a software bill of materials (SBOMs), e.g., in SPDX format. Many software producers aren’t ready to provide one yet, but creating the demand will speed progress
  9. Determine if subcomponents we use have known vulnerabilities 
  10. Work towards providing SBOM information if we produce software for others
  11. Implement OpenChain 

Let’s make it much harder to exploit the future systems we all depend on. Those who do not learn from history are often doomed to repeat it.

David A. Wheeler, Director of Open Source Supply Chain Security at the Linux Foundation

Centaurus today is becoming a Linux Foundation Project. The Centaurus Infrastructure Project is a cloud infrastructure platform for building distributed cloud as well as a platform for modern cloud native computing. It supports applications and workloads for 5G, Edge and AI and unifies the orchestration, network provisioning and management of cloud compute and network resources at a regional scale. 

Founding members include Click2cloud, Distributed Systems, Futurewei, GridGain Systems, Reinvent Labs, SODA Foundation and Tu Wien Informatics. Centaurus is an umbrella project for modern distributed computing and hosts both Arktos and Mizar. Arktos is a compute cluster management system designed for large scale clouds, while Mizar is the high-performance cloud-network powered by eXpress Data Path (XDP) and Geneve protocol for high scale cloud. More members and projects are expected to be accepted in the coming months. 

“The market is changing and customers require a new kind of cloud infrastructure that will cater to modern applications and workloads for 5G, AI and Edge,” said Mike Dolan, senior vice president and general manager for Linux Foundation Projects. “Centaurus is a technical project with strategic vision, and we’re looking forward to a deep collaboration that advances cloud native computing for generations to come.” 

Current cloud infrastructure technology needs are evolving, requiring companies to manage a larger scale of compute and network resources across data centers and more quickly provision those resources. Centaurus unifies management across bare metal, VMs, containers and serverless, while reducing operational costs and delivering on the low latency and data privacy requirements of edge networks. Centaurus offers a consistent API experience to provision and manage virtual machines, containers, serverless and other types of cloud resources by  combining traditional (Infrastructure as a Service) IaaS and Platform as a Service (PaaS) layers into one common infrastructure platform that can simplify cloud management.

“The Linux Foundation’s support in expanding the Centaurus community will accelerate cloud native infrastructure for the most pressing compute and networking demands,” said Dr. Xiong Ying, the current acting TSC chair, Centaurus Infrastructure Project. “It’s large network of open source developers and projects already supporting this future will enable mass collaboration and important integrations for 5G, AI and Edge workloads.” 

To contribute to Centaurus, please visit: https://www.centauruscloud.io/

Supporting Member Quotes

Click2cloud
“Click2cloud has been part of the development of Centaurus, which is world class software that will lead organizations to have a clear transition from IaaS to Cloud Native Infrastructure. Click2cloud has already started a development program to enable the journey from IaaS (Openstack) to Cloud Native migration, 5G cloud based on Centaurus reference architecture to support the partner ecosystem. We are very excited for Centaurus to be a part of Linux Foundation,” said Prashant Mishra, CEO, Click2cloud. 

Futurewei
“Distributed cloud architecture is a natural evolution for cloud computing infrastructure. Centaurus is a cloud native infrastructure platform aiming to unify management and orchestration of virtual machines, containers, and other forms of cloud resources natively at scale and at the edge. We have seen many enterprise users and partners wanting a unified solution to build their distributed cloud to manage virtual machines, containers or bare metal-based applications running at cloud as well as at edge sites. We are very pleased to see, today, the Centaurus Infrastructure project becomes a Linux Foundation open-source project, providing an option for community and enterprise users to build their cloud infrastructure to run and manage next generation applications such as AI, 5G and IoT. We look forward to working with the open-source community to realize the vision of Centaurus,” said Dr. Xiong Ying, Sr. Technical VP, Head of Cloud Lab, Futurewei. 

GridGain Systems
“Creating and managing a unified and scalable distributed cloud infrastructure that extends from cloud to edge is increasingly a challenge for organizations worldwide. GridGain Systems has been a proud sponsor and active participant in the development of in-memory computing solutions to support the Centaurus project. We look forward to helping organizations realize the benefits of Centaurus and continuing to help extend its scalability and adoption,” said Nikita Ivanov, Co-Founder and CTO, GridGain Systems. 

Reinvent Labs
“We are a young company, which specializes in cloud computing and delivering cloud-native solutions to our customers across various industries. As such, we are ever stronger witnessing the need to manage cloud services and applications that span across complex and heterogeneous infrastructures, which combine containers, VMs and serverless functions. What is more, such infrastructures are also starting to grow beyond traditional cloud platforms towards the edge on the network. Being part of the Centaurus project will not only allow us to innovate in this space and deliver a platform for unified management of infrastructure resources across both large Cloud platforms and the Edge, but it will also enable us to connect and collaborate with like-minded members for thought leadership and industry best practices,” said Dr. Stefan Nastic, founder and CEO of Reinvent Labs GmbH. 

The SODA Foundation
“The SODA Open Data Framework is an open source data and storage management framework that goes from the edge to the core to the cloud. Centaurus offers the opportunity for SODA to be deployed in the next generation cloud infrastructure for 5G, AI and Edge, and allows both communities to innovate together,” said Steven Tan, SODA Foundation Chairman and VP & CTO Cloud Solution, Storage at Futurewei. 

TU Wien
“We are very excited to be part of the Centaurus ecosystem and honored to be part of this open source movement and contributing in the fields of IoT, Edge intelligence, and Edge and Cloud Computing, including networking and communication aspects, as well as orchestration, resource allocation, and task scheduling,” said Prof. Schahram Dustdar, IEEE Fellow, Member Academia Europaea Professor of Distributed Systems, TU Wien, Austria.