kubernetes developer aws python microservices architecture api gateway

The article originally appeared on the Linux Foundation’s Training and Certification blog. The author is Marco Fioretti. If you are interested in learning more about microservices, consider some of our free training courses including Introduction to Cloud Infrastructure TechnologiesBuilding Microservice Platforms with TARS, and WebAssembly Actors: From Cloud to Edge.

Microservices allow software developers to design highly scalable, highly fault-tolerant internet-based applications. But how do the microservices of a platform actually communicate? How do they coordinate their activities or know who to work with in the first place? Here we present the main answers to these questions, and their most important features and drawbacks. Before digging into this topic, you may want to first read the earlier pieces in this series, Microservices: Definition and Main Applications, APIs in Microservices, and Introduction to Microservices Security.

Tight coupling, orchestration and choreography

When every microservice can and must talk directly with all its partner microservices, without intermediaries, we have what is called tight coupling. The result can be very efficient, but makes all microservices more complex, and harder to change or scale. Besides, if one of the microservices breaks, everything breaks.

The first way to overcome these drawbacks of tight coupling is to have one central controller of all, or at least some of the microservices of a platform, that makes them work synchronously, just like the conductor of an orchestra. In this orchestration – also called request/response pattern – it is the conductor that issues requests, receives their answers and then decides what to do next; that is whether to send further requests to other microservices, or pass the results of that work to external users or client applications.

The complementary approach of orchestration is the decentralized architecture called choreography. This consists of multiple microservices that work independently, each with its own responsibilities, but like dancers in the same ballet. In choreography, coordination happens without central supervision, via messages flowing among several microservices according to common, predefined rules.

That exchange of messages, as well as the discovery of which microservices are available and how to talk with them, happen via event buses. These are software components with well defined APIs to subscribe and unsubscribe to events and to publish events. These event buses can be implemented in several ways, to exchange messages using standards such as XML, SOAP or Web Services Description Language (WSDL).

When a microservice emits a message on a bus, all the microservices who subscribed to listen on the corresponding event bus see it, and know if and how to answer it asynchronously, each by its own, in no particular order. In this event-driven architecture, all a developer must code into a microservice to make it interact with the rest of the platform is the subscription commands for the event buses on which it should generate events, or wait for them.

Orchestration or Choreography? It depends

The two most popular coordination choices for microservices are choreography and orchestration, whose fundamental difference is in where they place control: one distributes it among peer microservices that communicate asynchronously, the other into one central conductor, who keeps everybody else always in line.

Which is better depends upon the characteristics, needs and patterns of real-world use of each platform, with maybe just two rules that apply in all cases. The first is that actual tight coupling should be almost always avoided, because it goes against the very idea of microservices. Loose coupling with asynchronous communication is a far better match with the fundamental advantages of microservices, that is independent deployment and maximum scalability. The real world, however, is a bit more complex, so let’s spend a few more words on the pros and cons of each approach.

As far as orchestration is concerned, its main disadvantage may be that centralized control often is, if not a synonym, at least a shortcut to a single point of failure. A much more frequent disadvantage of orchestration is that, since microservices and a conductor may be on different servers or clouds, only connected through the public Internet, performance may suffer, more or less unpredictably, unless connectivity is really excellent. At another level, with orchestration virtually any addition of microservices or change to their workflows may require changes to many parts of the platform, not just the conductor. The same applies to failures: when an orchestrated microservice fails, there will generally be cascading effects: such as other microservices waiting to receive orders, only because the conductor is temporarily stuck waiting for answers from the failed one. On the plus side, exactly because the “chain of command” and communication are well defined and not really flexible, it will be relatively easy to find out what broke and where. For the very same reason, orchestration facilitates independent testing of distinct functions. Consequently, orchestration may be the way to go whenever the communication flows inside a microservice-based platform are well defined, and relatively stable.

In many other cases, choreography may provide the best balance between independence of individual microservices, overall efficiency and simplicity of development.

With choreography, a service must only emit events, that is communications that something happened (e.g., a log-in request was received), and all its downstream microservices must only react to it, autonomously. Therefore, changing a microservice will have no impacts on the ones upstream. Even adding or removing microservices is simpler than it would be with orchestration. The flip side of this coin is that, at least if one goes for it without taking precautions, it creates more chances for things to go wrong, in more places, and in ways that are harder to predict, test or debug. Throwing messages into the Internet counting on everything to be fine, but without any way to know if all their recipients got them, and were all able to react in the right way can make life very hard for system integrators.

Conclusion

Certain workflows are by their own nature highly synchronous and predictable. Others aren’t. This means that many real-world microservice platforms could and probably should mix both approaches to obtain the best combination of performance and resistance to faults or peak loads. This is because temporary peak loads – that may  be best handled with choreography – may happen only in certain parts of a platform, and the faults with the most serious consequences, for which tighter orchestration could be safer, only in others (e.g. purchases of single products by end customers, vs orders to buy the same products in bulk, to restock the warehouse) . For system architects, maybe the worst that happens could be to design an architecture that is either orchestration or choreography, but without being really conscious (maybe because they are just porting to microservices a pre-existing, monolithic platform) of which one it is, thus getting nasty surprises when something goes wrong, or new requirements turn out to be much harder than expected to design or test. Which leads to the second of the two general rules mentioned above: don’t even start to choose between orchestration or choreography for your microservices, before having the best possible estimate of what their real world loads and communication needs will be.

peanut farm field with tech UI overlay

One of the first projects I noticed after starting at the Linux Foundation was AgStack. It caught my attention because I have a natural inclination towards farming and ranching, although, in reality, I really just want a reason to own and use a John Deere tractor (or more than one). The reality is the closest I will ever get to being a farmer is my backyard garden with, perhaps, some chickens one day. But I did work in agriculture policy for a number of years, including some time at USDA’s Natural Resources Conservation Service. So, AgStack piqued my interest. Most people don’t really understand where their food comes from, the challenges that exist across the globe, and the innovation that is still possible in agriculture. It is encouraging to see the passion and innovation coming from the folks at AgStack.

Speaking of that, I want to dig into (pun intended) one of AgStacks’ projects, Ag-Rec.

Backing up a bit, in the United States, the U.S. Department of Agriculture operates a vast network of cooperative extension offices to help farmers, ranchers, and even gardeners improve their practices. They have proven themselves to be invaluable resources and are credited with improving agriculture practices both here in the U.S. and around the globe through research, information sharing, and partnerships. Even if you aren’t a farmer, they can help you with your garden, lawn, and more. Give them a call – almost every county has an office.

The reality with extension education is that it is still heavily reliant on individuals going to offices and reading printed materials or PDFs. It could use an upgrade to help the data be more easily digestible, to make it quicker to update, to expand the information available, and to facilitate information sharing around the world. Enter Ag-Rec. 

I listened to Brandy Byrd and Gaurav Ramakrishna, both with IBM, present about Ag-Rec at the Open Source Summit 2022 in Austin, Texas. 

Brandy is a native of rural South Carolina, raised in an area where everyone farmed. She recalled some words of wisdom her granddaddy always said, “Never sell the goose that laid the golden egg.” He was referring to the value of the farmland – it was their livelihood. She grew up seeing firsthand the value of farms, and she was already familiar with the value of the information from the extension service and of information sharing among farmers and ranchers beyond mornings at the local coffee shop. But she also sees a better way. 

The vision of Ag-Rec is a framework where rural farmers from small SC towns to anywhere in the world have the same cooperative extension framework where they can get info, advice, and community. They don’t have to go to an office or have a physical manual. They can access a wealth of information and that can be shared anywhere, anytime. 

On top of that, by making it open source, anyone can use the framework so anyone can build applications and make the data available in new and useful ways. Ag-Rec is providing the base for even more innovation. Imagine the innovation we don’t know is possible. 

The Roadmap

Brandy and Gaurav shared about how Ag-Rec is being built and how developers, UI experts, agriculture practices experts, end users, and others can help contribute. When the recording of the presentation is available we will share that here. You can also go over to Ag-Rec’s GitHub for more information and to help. 

Here is the current roadmap: 

Immediate

  • Design and development of UI with Mojoe.net
  • Plant data validation and enhancements
  • Gather requirements to provision additional Extensive Service recommendation data
  • Integrate User Registry for authentication and authorization

Mid-term

  • Testing and feedback from stakeholders
  • Deploy the solution on AgStack Cloud
  • Add documentation for external contribution and self-deployment

Long-term

  • Invite other Extension Services and communities
  • Iterate and continuous improvement

I, for one, am excited about the possibility of this program to help improve crop production, agricultural-land conservation, pest management, and more around the world. Farms feed the world, fuel economies, and so much more. With even better practices, their positive impact can be even greater while helping conserve the earth’s resources. 

The Partners


In May 2021, the Linux Foundation launched the AgStack Foundation to “build and sustain the global data infrastructure for food and agriculture to help scale digital transformation and address climate change, rural engagement, and food and water security.”  Not long after, IBM, Call for Code and Clemson University Cooperative Extension “sought to digitize data that’s been collected over the years, making it accessible to anyone on their phone or computer to search data and find answers they need.” AgStack “way to collaborate with and gain insights from a community of people working on similar ideas, and this helped the team make progress quickly.” And Ag-Rec was born. 

A special thank you to the core team cultivating (pun intended) this innovation: 

Brandy Byrd, IBM

Gaurav Ramakrishna, IBM

Sumer Johal, AgStack

Kendall Kirk, Clemson University

Mallory Douglass, Clemson University

Mojoe.net

Resources

1 + 1 = 3

At last week’’s Open Source Summit North America, Robin Ginn, Executive Director of the OpenJS Foundation, relayed a principle her mentor taught: “1+1=3”. No, this isn’t ‘new math,’ it is demonstrating the principle that, working together, we are more impactful than working apart. Or, as my wife and I say all of the time, teamwork makes the dream work. 

This principle is really at the core of open source technology. Turns out it is also how I look at the Open Programmable Infrastructure project. 

Stepping back a bit, as “the new guy” around here, I am still constantly running across projects where I want to dig in more and understand what it does, how it does it, and why it is important. I had that very thought last week as we launched another new project, the Open Programmable Infrastructure Project. As I was reading up on it, they talked a lot about data processing units (DPUs) and infrastructure processing units (IPUs), and I thought, I need to know what these are and why they matter. In the timeless words of The Bobs, “What exactly is it you do here?” 

What are DPUs/IPUs? 

First – and this is important – they are basically the same thing, they just have different names. Here is my oversimplified explanation of what they do.

In most personal computers, you have a separate graphic processing unit(s) that helps the central 1 + 1 = 3 processing unit(s) (CPU) handle the tasks related to processing and displaying the graphics. They offload that work from the CPU, allowing it to spend more time on the tasks it does best. So, working together, they can achieve more than each can separately. 

Servers powering the cloud also have CPUs, but they have other tasks that can consume tremendous computing  power, say data encryption or network packet management. Offloading these tasks to separate processors enhances the performance of the whole system, as each processor focuses on what it does best. 

In order words, 1+1=3. 

DPUs/IPUs are highly customizable

While separate processing units have been around for some time, like your PC’s GPU, their functionally was primarily dedicated to a particular task. Instead, DPUs/IPUs combine multiple offload capabilities that are highly  customizable through software. That means a hardware manufacturer can ship these units out and each organization uses software to configure the units according to their specific needs. And, they can do this on the fly. 

Core to the cloud and its continued advancement and growth is the ability to quickly and easily create and dispose of the “hardware” you need. It wasn’t too long ago that if you wanted a server, you spent thousands of dollars on one and built all kinds of infrastructure around it and hoped it was what you needed for the time. Now, pretty much anyone can quickly setup a virtual server in a matter of minutes for virtually no initial cost. 

DPUs/IPUs bring this same type of flexibility to your own datacenter because they can be configured to be “specialized” with software rather than having to literally design and build a different server every time you need a different capability. 

What is Open Programmable Infrastructure (OPI)?

OPI is focused on utilizing  open software and standards, as well as frameworks and toolkits, to allow for the rapid adoption and use of DPUs/IPUs. The OPI Project is both hardware and software companies coming together to establish and nurture an ecosystem to support these solutions. It “seeks to help define the architecture and frameworks for the DPU and IPU software stacks that can be applied to any vendor’s hardware offerings. The OPI Project also aims to foster a rich open source application ecosystem, leveraging existing open source projects, such as DPDK, SPDK, OvS, P4, etc., as appropriate.”

In other words, competitors are coming together to agree on a common, open ecosystem they can build together and innovate, separately, on top of. The are living out 1+1=3.

I, for one, can’t wait to see the innovation.

A special thanks to Yan Fisher of Red Hat for helping me understand open programmable infrastructure concepts. He and his colleague, Kris Murphy, have a more technical blog post on Red Hat’s blog. Check it out. 

For more information on the OPI Project, visit their website and start contributing at https://github.com/opiproject/opi.  

Click here to add your own text

jetliner flying

In a new white paper, the Cardea Project at Linux Foundation Public Health demonstrates a complete, decentralized, open source system for sharing medical data in a privacy-preserving way with machine readable governance for establishing trust.

The Cardea Project began as a response to the global Covid-19 pandemic and the need for countries and airlines to admit travelers. As Covid shut down air travel and presented an existential threat to countries whose economies depended on tourism, SITA Aero, the largest provider of IT technology to the air transport sector, saw decentralized identity technology as the ideal solution to manage a proof of Covid test status for travel.

With a verifiable credential, a traveler could hold their health data and not only prove they had a specific test at a specific time, they could use it—or a derivative credential—to prove their test status to enter hotels and hospitality spaces without having to divulge any personal information. Entities that needed to verify a traveler’s test status could, in turn, avoid the complexity of direct integrations with healthcare providers and the challenge of complying with onerous health data privacy law.

Developed by Indicio with SITA and the government of Aruba, the technology was successfully trialed in 2021 and the code specifically developed for the project was donated to Linux Foundation Public Health (LFPH) as a way for any public health authority to implement an open source, privacy-preserving way to manage Covid test and vaccination data. The Cardea codebase continues to develop at LFPH as Indicio, SITA, and the Cardea Community Group extend its features and applications beyond Covid-related data.

On May 22, 2022 at the 15th KuppingerCole European Identity and Cloud Conference in Berlin, SITA won the Verifiable Credentials and Decentralized Identity Award for its implementation of decentralized identity in Aruba.

The new white paper from the Cardea Project provides an in-depth examination of the background to Cardea, the transformational power of decentralized identity technology, how it works, the implementation in Aruba, and how it can be deployed to authenticate and share multiple kinds of health data in privacy-preserving ways. As the white paper notes:

“…Cardea is more than a solution for managing COVID-19 testing; it is a way to manage any health-related process where critical and personal information needs to be shared and verified in a way that enables privacy and enhances security. It is able to meet the requirements of the 21st Century Cures Act and Europe’s General Data Protection Regulation, and in doing so enable use cases that range from simple proof of identity to interoperating ecosystems encompassing multiple cloud services, organizations, and sectors, where data needs to be, and can be, shared in immediately actionable ways.

Open source, interoperable decentralized identity technology is the only viable way to manage both the challenges of the present—where entire health systems can be held at ransom through identity-based breaches—and the opportunities presented by a digital future where digital twins, smart hospitals, and spatial web applications will reshape how healthcare is managed and delivered.”

The white paper is available here. The community development group meets weekly on Thursdays at 9:00am PST—please join us!

This article was originally published on the Linux Foundation Public Health project’s blog


patent trolls open source chart

So, I am old enough to remember when the U.S. Congress temporarily intervened in a patent dispute over the technology that powered BlackBerries. A U.S. Federal judge ordered the BlackBerry service to shutdown until the matter was resolved, and Congress determined that BlackBerry service was too integral to commerce to be allowed to be turned off. Eventually, RIM settled the patent dispute and the BlackBerry rode off into technology oblivion

I am not here to argue the merits of this nearly 20-year-old case (in fact, I coincidentally had friends on both legal teams), but it was when I was introduced to the idea of companies that purchase patents with the goal of using this purchased right to extract money from other companies. 

Patents are an important legal protection to foster innovation, but, like all systems, it isn’t perfect. 

At this week’s  Open Source Summit North America, we heard from Kevin Jakel with Unified Patents. Kevin is a patent attorney who saw the damage being done to innovation by patent trolls – more kindly known as non-practicing entities (NPEs). 

Kevin points out that patents are intellectual property designed to protect inventions, granting a time-bound legal monopoly, but they are only a sword, not a shield. You can use it to stop people, but it doesn’t give you a right to do anything. He emphasizes, “You are vulnerable even if you invented something. Someone can come at you with other patents.” 

Kevin has watched a whole industry develop where patents are purchased by other entities, who then three steps of stopping patent trolls go after successful individuals or companies who they claim are infringing on the patents they now legally own (but is not something they invented). In fact, 88% of all high-tech patent litigation is from an NPE.

NPEs are rational actors using the legal system to their advantage, and they are driven by the fact that almost all of the time the defendant decides to settle to avoid the costs of defending the litigation. This perpetuates the problem by both reducing the risk to the NPEs and also giving them funds to purchase additional patents for future campaigns. 

In regards to open source software, the problem is on the rise and is only going to get worse without strategic, consistent action to combat it.

patent trolls open source cases by year chart

Kevin started Unified Patents with the goal of solving this problem without incentivizing further NPE activity. He wants to increase the risk for NPEs so that they are incentivized to not pursue non-existent claims. Because NPEs are rational actors, they are going to weigh risks vs. rewards before making any decisions. 

How does Unified Patents do this? They use a three-step process: 

  • Detect – Patent Troll Campaigns
  • Disrupt – Patent Troll Assertions
  • Deter – Further Patent Troll Investment 

Unified Patents works on behalf of 11 technology areas (they call them Zones). They added an Open Source Zone in 2019 with the help of the Linux Foundation, Open Invention Network, and Microsoft. They look for demands being filed in court, and then they selectively pick patent trolls out of the group and challenge them, attempting to disrupt the process. They take the patent back to the U.S. Patent and Trademark Office and see if the patent should have ever existed in the first place. Typically, patent trolls look for broad patents so they can sue lots of companies, making their investment more profitable and less risky. This means it is so broad that it probably should never have been awarded in the first place. 

The result – they end up killing a lot of patents that should have never been issued but are being exploited by patent trolls, stifling innovation. The goal is to slow them down and eventually bring them to a stop as quickly as they can. Then, the next time they go to look for a patent, they look somewhere else.

And it is working. The image below shows some of the open source projects that Unified Patents has actively protected since 2019.

open source tech logos

The Linux Foundation participates in Unified Patents’ Open Source Zone to help protect the individuals and organizations innovating every day. We encourage you to join the fight and create a true deterrence for patent trolls. It is the only way to extinguish this threat. 

Learn more at unifiedpatents.com/join

And if you are a die-hard fan of the BlackBerry’s iconic keyboard, my apologies for dredging up the painful memory of your loss. 

dronecode featured image

If you are interested in online and in-person training and certifications in open source software development and key open source software, such as Linux and Kubernetes, see our special discount just for readers of this post. Scroll to the end.

dronecode foundation logo with animated drones

On Thursday, 6/23, over the night skies over Congress Bridge in Austin, Texas, 300 drones will work in concert to provide a light show to entertain but also inform about the power of open source software to drive innovation in our world, making an impact in every life, every day.


Backing up a bit, open source software often conjures up inaccurate visions and presumptions that just aren’t true. No need to conjure those up – we all know what they are. The reality is that open source software (OSS) has transformed our world and become the backbone of our digital economy and the foundation of our digital world. 

The reality is that open source software (OSS) has transformed our world and become the backbone of our digital economy and the foundation of our digital world. 

Some quick, fun facts

  • In vertical software stacks across industries, open source penetration ranges from 20 to 85 percent of the overall software used
  • Linux fuels 90%+ of web servers and Internet-connected devices
  • The Android mobile operating system is built on the Linux kernel
  • Immensely popular libraries and tools to build web applications, such as: AMP, Appium, Dojo, jQuery, Marko, Node.js and so many more are open source
  • The world’s top 100 supercomputers run Linux
  • 100% of mainframe customers use Linux
  • The major cloud-service providers – AWS, Google, and Microsoft – all utilize open-source software to run their services and host open-source solutions delivered through the cloud

Open source software is about organizations coming together to collectively solve common problems so they can separately innovate and differentiate on top of the common baseline. They see they are better off pooling resources to make the baseline better. Sometimes it is called “coopetition.” It generally means that while companies may be in competition with each other in certain areas, they can still cooperate on others.

I borrowed from a well-known tagline from my childhood in the headline – open source does bring good things to life. 

Fueling Drone Innovation 

Drones were introduced to the world through military applications and then toys we could all easily fly (well, my personal track record is abysmal). But the reality is that drones are seeing a variety of commercial applications, such as energy facility inspection for oil, gas, and solar, search and rescue, firefighting, and more, with new uses coming online all of the time. We aren’t at The Jetsons level yet, but they are making our lives easier and safer (and some really cool aerial shots).

Much of that innovation comes from open source coopetition. 

The Linux Foundation hosts the Dronecode Foundation, which fosters open source code and standards critical to the worldwide drone industry. In a recent blog post, the general manager, Ramón Roche, discusses some of the ways open source has created an ecosystem of interoperability,  which leads to users having more choice and flexibility. 

Building the Foundation

Ramón recounts how it all started with the creation of Pixhawk, open standards for drone hardware, with the goal to make drones fly autonomously using computer vision. Working to overcome the lack of computing power and technology in 2008, Lorenz Meier, then a student, set out to build the necessary flight control software and hardware. Realizing the task’s scale, he sought the help of fourteen fellow students, many of whom were more experienced than him, to make it happen. They built Pixhawk and kick started an open source community around various technologies. It, “enabled talented people worldwide to collaborate and create a full-scale solution that was reusable and standardized. By giving their technology a permissive open source license, they opened it to everyone for use and collaboration.”

Benefits of Openness in the Real World

The innovation and technological backbone we see in drones is thanks to open software, hardware, and standards. Dronecode’s blog has interviews with Max Tubman of Freefly Systems talks about how open standards are enabling interoperability of various payloads amongst partners in the Open Ecosystem. Also, Bobby Watts of Watts Innovation explains the power of standardization and how it has streamlined their interoperability with other ecosystem partners like Gremsy and Drone Rescue Systems.

The innovation and technological backbone we see in drones is thanks to open software, hardware, and standards

Check out both interviews here and read about what is next.

The story of open source driving innovation in the drone industry is just one of thousands of examples of how open source is driving global innovation. Whether you know it or not, you use open source software every minute of every hour of every day.



Training promo

Use promo code DRONE25 here to receive up to 25% off of Linux Foundation’s training, taken by millions of students around the world. Expires on June 30, 2022. View the whole catalog, from AI and blockchain to web and application development, we have something for you.  

LFX dashboard example

Open source communities are driven by a mutual interest in collaboration and sharing around a common solution. They are filled with passion and energy. As a result, today’s world is powered by open source software, powering the Internet, databases, programming languages, and so much more. It is revolutionizing industries and tackling the toughest challenges. Just check out the projects fostered here at the Linux Foundation for a peek into what is possible. 

What is the challenge? 

As the communities and the projects they support grow and mature, active community engagement to recruit, mentor, and enable an active community is critical. Organizations are now recognizing this as they are more and more dependent on open source communities. Yet, while the ethos of open source is transparency and collaboration, the tool chain to automate, visualize, analyze, and manage open source software production remains scattered, siloed, and of varying quality.

How do we address these challenges?

And now, involvement and engagement in open source communities goes beyond software developers and extends to engineers, architects, documentation writers, designers, Open Source Program Office professionals, lawyers, and more. To help everyone stay coordinated and engaged, a centralized source of information about their activities, tooling to simplify and streamline information from multiple sources, and a solution to visualize and analyze key parameters and indicators is critical. It can help: 

  • Organizations wishing to better understand how to coordinate internal participation in open source and measure outcomes
  • CTOs and engineering leads looking to build a cohesive open source strategy 
  • Project maintainers needing to wrangle the legal and operational sides of the project
  • Individual keeping track of their open source impacts

Enter the Linux Foundation’s LFX Platform – LFX operationalizes this approach, providing tools built to facilitate every aspect of open source development and empowers projects to standardize, automate, analyze, and self-manage while preserving their choice of tools and development workflows in a vendor-neutral platform.

LFX tools do not disrupt a project’s existing toolchain but rather integrate a project’s community tools and ecosystem to provide a common control plane with APIs from numerous distributed data sources and operations tools. It also adds intelligence to drive outcome-driven KPIs and utilizes a best practices-driven, vendor-agnostic tools chain. It is the place to go for active community engagement and open source activity, enabling the already powerful open source movement to be even more successful.

How does it work? 

Much of the data and information that makes up the open source universe is, not surprisingly, open to see. For instance, GitHub and GitLab both offer APIs that allow third-parties to track all activity on open projects. Social media and public chat channels, blog posts, documentation, and conference talks are also easily captured. For projects hosted at a foundation, such as the Linux Foundation, there is an opportunity to aggregate the public and semi-private data into a privacy respecting, opt-in unified data layer. 

More specifically to an organization or project, LFX is modular, extensible, and API-driven. It is pluggable and can easily integrate the data sources and tools that are already in use by organizations rather than force them to change their work processes. For instance:

  • Source control software (e.g. Git, GitHub, or GitLab)
  • CI/CD platforms (e.g. Jenkins, CircleCI, Travis CI, and GitHub Actions)
  • Project management (e.g. Jira, GitHub Issues)
  • Registries  (e.g. Docker Hub)
  • Documentation  (e.g. Confluence Wiki)
  • Marketing automation (e.g. social media and blogging platforms)
  • Event management platforms (e.g. physical event attendance, speaking engagements, sponsorships, webinar attendance, and webinar presentations)

This holistic and configurable view of projects, organizations, foundations, and more make it much easier to understand what is happening in open source, from the most granular to the universal. 

What do real-world users think? 

Part of LFX is a community forum to ask questions, share solutions, and more. Recently, Jessica Wagantall shared about the Open Network Automation Platform (ONAP). She notes:

ONAP is part of the LF Networking umbrella and consists of 30+ components working together towards the same goal since 2017. Since then, we have faced situations where we have to evaluate if the components are getting enough support during release schedules and if we are identifying our key contributors to the project.

In this time, we have learned a lot as we grow, and we have had the chance to have tools and resources that we can rely on every step of the way. One of these tools is LFX Insights.

We rely on LFX Insights tools to guide the internal decisions and keep the project growing and the contributions flowing.

LFX Insights has become a potent tool that gives us an overview of the project as well as statistics of where our project stands and the changes that we have encountered when we evaluate release content and contribution trends.

Read Jessica’s full post for some specific examples of how LFX Insights helps her and the whole team. 

John Mertic is a seasoned open source project manager. One of his jobs currently is helping to manage the Academy Software Foundation. John shares: 

The Academy Software Foundation was formed in 2018 in partnership with the Academy of Motion Pictures Arts and Sciences to provide a vendor-neutral home for open source software in the visual effects and motion picture industries.

A challenge this industry was having was that there were many key open source projects used in the industry, such as OpenVDB, OpenColorIO, and OpenEXR, that were cornerstones to production but lacked developers and resources to maintain them. These projects were predominantly single vendor owned and led, and my experience with other open source projects in other verticals and horizontal industries causes this situation, which leads to sustainability concerns, security issues, and lack of future development and innovation.

As the project hit its 3rd anniversary in 2021, the Governing Board was wanting to assess the impact the foundation has had on increasing the sustainability of these projects. There were three primary dimensions being assessed.

  • Contributor growth
  • Contribution growth
  • Contributor diversity

We at the LF know that seeing those metrics increasing is a good sign for a healthy, sustainable project.

Academy Software Foundation projects use LFX Insights as a tool for measuring community health. Using this tool enabled us to build some helpful charts which illustrated the impacts of being a part of the Academy Software Foundation.

We took the approach of looking at before and after data on the contributor, contribution, and contributor diversity.

Here is one of the charts that John shared. You can view all of them on his post


LFX dashboard example

Conclusion 

LFX will improve communication and collaboration, simplify management, surface the best projects and project leaders, and provide insightful guidance based on real data captured at scale, across the widest variety of projects ever collected into a single source of information. And it is available to you – all Linux Foundation members and projects have access to LFX. 

To learn more about what it can do for you and your organization and project(s), read our white paper (LINK), read posts in the LFX Community Forum, or just log in with your free LFID and give it a spin. And check back here on the LF Blog for more articles in the coming months on LFX – digging in deeper. 

If you would like to talk to someone at the Linux Foundation about LFX or membership, reach out to Jen Shelby at jshelby@linuxfoundation.org

OSPOlogy live workshops

As more and more organizations adopt open source initiatives and/or seek to mature their involvement in open source, they often face many challenges, such as educating developers on good open source practices, building policies and infrastructure, ensuring high-quality and frequent releases, engaging with developer communities, and contributing back to other projects effectively. They recognize that open source is a complex ecosystem that is a community of communities. It doesn’t follow traditional corporate rules, so guidance is needed to overcome cultural change. 

To help address these challenges and take advantage of the opportunities, organizations are turning to open source program offices (OSPOs). An OSPO is designed to be the center of competency for an organization’s open source operations and structure. This can include setting code use, distribution, selection, auditing, and other policies, as well as training developers, ensuring legal compliance, and promoting and building community engagement that benefits the organization strategically. 

The Linux Foundation’s TODO Group’s mission is to help foster the adoption and improvement of OSPOs around the world. They are a tremendous resource, with extensive guides, a new mind map, an online course, case studies, and more. Check out their resources, community, and join their efforts

Thanks in part to their efforts, the OSPO movement is expanding across industries and regions of all types and sizes. However, due to the wide range of responsibilities and ways to operate, OSPO professionals often find it difficult to implement OSPO best practices, policies, processes, or tools for their open source management efforts.

To help people with these challenges, the TODO Group is introducing a new framework for in-person OSPO workshops. The framework is publicly available in ospology. This repo encapsulates a set of open initiatives (including an OSPO Mind Map 2.0, virtual global & regional meetings, an OSPO discussion forum, monthly OSPO News, and now, in-person workshops) to work in collaboration that aims to study and discuss the status of OSPOs and, ultimately, make them even more effective. 

TODO is piloting these in Europe first, and they are currently seeking collaborators to bring together the various communities involved in OSPO-specific topics and help organizations effectively implement OSPO Programs based on the specific needs for the region.

Backing up a bit, let’s look at the OSPOlogy.live framework. 

OSPOlogy.live framework in a nutshell

  • Follows an “unconference style,” meaning it’s a participants-driven meeting
  • Adheres to the Chatham House Rule in order to share openly and learn from each other 
  • Connects OSPOs with various open source communities involved in the open source activities that matter to them (e.g. policies, tooling, standards, and community building)
  • Takes place over two days and is an in-person event
  • Consists of prepared presentations, hands-on workshops, and space for networking
  • Falls under the Linux Foundation’s policies and code of conduct
  • Held at a location provided by one of the participants for free
  • Each participant pays for their own food, travel, and lodging. Meals may be free if workshop organizers find sponsors.
  • Participants can register their interest to receive an invite via Linux Foundation’s community platform as seats are limited.

With that overview, let’s dig in a little on how the workshop is conducted.

Unconference style

Typically at an unconference, the agenda of the workshop portion is created by the attendees at the beginning of the meeting. Anyone who wants to initiate a discussion on a topic can claim a time and a space. OSPOlogy workshops are not fully an unconference as the first day is a series of prepared presentations, so you know what the sessions are before joining (1 or 2 will be chosen by the participants ahead of time). For Day 2, the workshops follow the unconference model. Participants vote on topics to be worked on that day. Participants may be asked to submit their topic before the workshop to accelerate/simplify the voting process.

Suggested workshop sections

  • OSPO USE CASES ➡️Expert-led panels or talks to share experiences and case studies from specific OSPOs
  • OSPO ACCELERATORS ➡️Presentation highlighting a specific activity within the specific project, such as outcomes of recent community activities. The aim of the presentation is to give people insights on various topics the communities are working on and get their feedback / to ask for contributions.
  • SHARED CHALLENGES ASSESSMENT ➡️ Description: Identify OSPO shared challenges / pain points on the OSPO Mind Map 2.0 and let the audience vote for the areas of interest (working groups) for the workshop breakout groups. For instance, focus areas can be specific activities within OSPO responsibilities.
  • BREAK OUT SESSIONS ➡️ Define goals and identify pain points. Each break out group aims to capture their challenges for the selected focus and if possible document their experiences/solutions.
  • NETWORKING

Interested in becoming a collaborator?

We can’t do this alone! If you are part of an open source community involved in OSPO-specific topics or an organization willing to help with the workshop planning, schedule and/or provide a space to kick off the first meet-up in Europe, we need your help! Please contact:

And check out the FAQs below. 

Don’t live in Europe? Pencil us in for when this is expanded. 

Not involved in an OSPO yet? Take time to check out the TODO Group and join the community to start your OSPOlogy journey.

Also, consider joining OSPONCon North America next week, June 21-24, 2022, either in Austin, Texas during the Open Source Summit or virtually. Register here.



Frequently Asked Questions

What do we mean by communities involved in OSPO-specific topics?

OSPO-specific topics range from safely using open source to license compliance, sustainability, contributing back to the community, and more. For the full list of OSPO topics please see https://ospomindmap.todogroup.org/:

  • Develop and Execute Open Source Strategy
  • Oversee Open Source Compliance
  • Establish and Improve Open Source Policies and Processes
  • Prioritize and Drive Open Source Upstream Development
  • Collaborate with Open Source Organizations
  • Track Performance Metrics
  • Implement InnerSource Practices
  • Grow and Retain Open Source Talent Inside the Organization
  • Give Advice on Open Source
  • Manage Open Source IT Infrastructure

Some examples of OS communities highly involved in these topics are:

What are the necessary roles to set up an OSPOlogy.live workshop?

There are two ways in which you can play your part in OSPOlogy.live set up: (1) the hosting party who makes available a meeting room; and, (2) the workshop organizer/facilitator in charge of workshop activities and planning. (1) and (2) may be the same entity/individual. Further details can be found in the framework documentation

Where can I register for the next OSPOlogy.live?

Efforts are already on the way to organize the OSPOlogy workshops in different European countries each quarter. Once collaborators and days are confirmed, registration details and schedules will be published via the OSPOlogy community platform.

For further updates, please subscribe to OSPONewsletter and join the TODO community.

SBOMs and food labels

Software Bill of Materials (SBOMs) are like ingredient labels on food. They are critical to keep consumers safe and healthy, they are somewhat standardized, but it is a lot more exciting to grow or make the food rather than the label. 

What is an SBOM?

What is an SBOM? In short, it is a way to tell another party all of the software that is used in the stack that makes up an application. One benefit of having a SBOM is you know what is in there when a vulnerability comes up. You can easily determine if you are vulnerable and where. 

As modern software is built utilizing a base of software already written (no sense in recreating the wheel), it is important that all of the components don’t get lost in the shuffle. It isn’t readily apparent what a particular piece of software utilizes. So, if a vulnerability for Software A arises, you need to know, do I have that piece of software somewhere in my ecosystem, and, if so, where. Then you can remediate if you need to.

I can’t take credit for the food label analogy used in my introduction. I heard it from Allan Friedman, a Senior Advisor and Strategist at the U.S. Cybersecurity and Infrastructure Security Agency (CISA) and a key SBOM advocate, when he presented about SBOMs at the RSA Conference 2022 with Kate Stewart, the VP of Dependable Embedded Systems here at the Linux Foundation. Allan made the point that food labels only provide information. The consumer needs to read and understand them and take appropriate action. For instance, if they are allergic to peanuts, they can look at an ingredient label and determine if they can safely eat the food. 

SBOMs are similar – they tell a person what software is used as an “ingredient” so someone can determine if they need to take action if a vulnerability arises. It isn’t a silver bullet, but it is a vital tool. Without SBOMs no one can track what component “ingredients” are in their software applications.

SBOMs and the Software Supply Chain

Supply chains are impacting our lives more than just restricting availability of consumer goods. Software supply chains are immensely more complicated now as software is built with pre-existing components. This makes software better, more effective, more powerful, etc. But it also introduces risk as more and more parties touch a particular piece of software. Much like our world has become so interdependent, so has our software. 

Understanding what is in the supply chain for our software helps us effectively secure it. When a new risk emerges, we know what we need to do. 

SBOMs and Software Security

SBOMs are increasingly being recognized as an important pillar in any comprehensive software security plan. A global survey conducted in 2021 Q3 by the Linux Foundation found that 78% of organizations responding plan to use SBOMs in 2022. Additionally, the recently published Open Source Software Security Mobilization Plan recommends SBOMs be universal and the U.S. Executive Order on Improving the Nation’s Cybersecurity requires SBOMs be provided for software purchased by the U.S. government. And, as Allan points out in his talk, “We buy everything.” The E.O. actually lays out a nice summary of SBOMs and their benefits: 

The term “Software Bill of Materials” or “SBOM” means a formal record containing the details and supply chain relationships of various components used in building software.  Software developers and vendors often create products by assembling existing open source and commercial software components.  The SBOM enumerates these components in a product.  It is analogous to a list of ingredients on food packaging.  An SBOM is useful to those who develop or manufacture software, those who select or purchase software, and those who operate software.  Developers often use available open source and third-party software components to create a product; an SBOM allows the builder to make sure those components are up to date and to respond quickly to new vulnerabilities.  Buyers can use an SBOM to perform vulnerability or license analysis, both of which can be used to evaluate risk in a product.  Those who operate software can use SBOMs to quickly and easily determine whether they are at potential risk of a newly discovered vulnerability.   A widely used, machine-readable SBOM format allows for greater benefits through automation and tool integration.  The SBOMs gain greater value when collectively stored in a repository that can be easily queried by other applications and systems.  Understanding the supply chain of software, obtaining an SBOM, and using it to analyze known vulnerabilities are crucial in managing risk.

Allan and Kate spent time in their talk going into the current state of SBOMs, challenges, benefits, tools available for creating and sharing SBOMs, what is a minimum SBOM, standards being developed, making them fully automated, and more. Look for some future LF Blog posts digging into these. 

But there are things you can do now. 

What can you and your organization do now?

Allan and Kate laid out several things you and your organization can do, starting now. Starting within your organization: 

Next week: Understand origins of software your organization is using

  • Commercial: can you ask for an SBOM?
  • Open source: do you have an SBOM for the binary or sources you’re importing? 

Three months: Understand what SBOMs your customers will require

  • Expectations: which standards, dependency depth, licensing info?

Six months: Prototype and deploy

  • Implement SBOM through using an OSS tool and/or starting a conversation with vendor

And participate in ongoing discussions to determine best practices for the ecosystem and contribute to open source project any code developed to support SBOMs. 

But there are also steps you can take as an individual:

Next week: Start playing with an open source SBOM tool and apply it to a repo

Three months: Have an SBOM strategy that explicitly identifies tooling needs

Six months

  • Begin SBOM implementation through using an OSS tool or starting a conversation with vendor
  • Participate in a plugfest and try to consume another’s SBOM

And make sure to share any open source and commercial tools you find helpful and work with the tools to help harden them, test and report bugs, and push them to scale.

How can you shape the future of SBOMs?

First, I want to highlight some upcoming opportunities they shared to help shape the future of SBOMs. CISA is running public Tooling & Implementation work stream discussions in July 2022. They are the same, but occur at different times to help accommodate more time zones: 

  • July 13, 2022 – 3:00-4:30 PM ET
  • July 21, 2022 – 9:30-11:00 AM ET 

If you want to participate, please email SBOM@cisa.dhs.gov

Additionally, there will be “plugfests” to be announced soon, and they suggested organizations already adopting SBOMs publish case studies and reference tooling workflows to help others. 

Conclusion

SBOMs are here to stay. If you aren’t already, get on the train now. It is pulling out of the station, but you still have an opportunity to help shape where it is going and how well the journey goes. 

Allan’s and Kate’s slides are available here. If you registered to attend the RSA Conference, you can now watch their full presentation on demand here.



The Software Package Data ExchangeⓇ (SPDXⓇ)

The Linux Foundation hosts SPDX, which is an open standard for communicating software bill of material information, including components, licenses, copyrights, and security references. SPDX reduces redundant work by providing a common format for companies and communities to share important data, thereby streamlining and improving compliance. The SPDX specification is an international open standard (ISO/IEC 5962:2021). Learn more at spdx.dev

A brief about my experience with the Linux Foundation Mentorship.

The post originally appeared on deprov477’s blog. The author, Anubhav Choudhary, particpated in the Linux Foundation’s Mentorship Program in 2022. The program is designed to help developers — many of whom are first-time open source contributors — with necessary skills and resources to learn, experiment, and contribute effectively to open source communities. By participating in a mentorship program, mentees have the opportunity to learn from experienced open source contributors as a segue to get internship and job opportunities upon graduation. If you are interested, we invite you to learn more and apply today here.


Hi everyone, I recently completed my LFX Mentorship project. I was a mentee for the LFXM summer term of 2022 at Pixie, a CNCF sandbox project donated by The New Relic.

In this blog, I will be sharing my experience of mentorship. (TLDR; just awesome, one-of-a-kind experience <3) If you're also applying for this (which every open-source newbie should), or have a doubt, feel free to drop me a message. I’d be more than happy to help.

What is LFX Mentorship?

Let’s start this by knowing about The Linux Foundation. The Linux Foundation (LF) is a non-profit organization, that standardizes the development of the Linux kernel and also promotes open source projects such as Kubernetes, GraphQL, Hyperledger, RISC-V, Xen project, etc.

The Linux Foundation Mentorship is a program run by LF, which helps developers with the necessary skills and resources to learn and contribute to open source projects, through 3 or 6 months of internship. During this period, the mentee is guided through the development workflow and methodologies used by open source organizations, through a project.

Selection procedure

I’ve been involved in open source for some time and have been applying for the mentorship, but got rejected every time.

This time also I was going through the projects and found a particularly interesting project. It was about parsing a protocol. This took my eye as at that time I was learning networking and experimenting a lot with communications. So naturally, I got interested. After reading the project details, I went to the project’s slack channel to find a mentor. Omid, one of Pixie’s founding engineers, was kind enough to reply to my message and asked for a quick call.

I talked to him and told him about my interest and how I made a preliminary Mongo wire protocol parser using Node.js as preparation. He seemed satisfied with this and told me about further steps and time commitment.

Other formalities included submitting a cover letter, and my resume.

A few days later got this:

LFX Hi Anubhav

Finally, after applying so many times, got selected !!!

Month 1

Started, and was introduced to my mentor Yaxiong Zhao, another founding engineer at Pixie. He told me about what we were going to do in the next 3 months. He demoed me the Pixie UI and explained to me the working of it, and how pixie catches packets (hint: eBPF). And then sent me the AMQP spec sheet, and how it needs to be implemented using C++.

Yes, the protocol changed from Mongo to AMQP, and the language from Node.js to C++. But I guess a very important survival quality of industry is being flexible.

So, in the first month, I got a theoretical knowledge about AMQP wire spec and experimented with it by deploying a local RabbitMQ server, and monitoring packets using Wireshark. My mentor also tried helping me build Pixie on my local machine, but we failed, even after switching distros. At last, we were able to set up my dev environment inside a container.

…quite a month

Month 2

In the first half of this month, I continued my research on AMQP (apparently implementing a protocol required a lot of extensive reading) and found analogies of it with protocols I was already familiar with, and kept on manually experimenting with packet translation.

3rd week of the month, It was finally time for me to start writing some code. Okay, so this was the difficult part. Having very limited knowledge of C++, continued forward. But my mentor was being an angel at this point, very patiently explaining to me, and pointing me in the right direction, making me understand every lex required. I started with implementing a data structure for storing and creating relations between packets. After some effort, finally got my PR merged.

AMQP types header file

Month 3

Continuing my code work, I started building a parser code. Yaxiong was very patient and helpful during this time, sending me blogs, and guides and explaining to me every little doubt I had. Thanks to him I was able to finally submit my preliminary code for parsing the code.

And a final thing for this was to write tests. Learned google’s C++ testing library. Wrote code, pushed.

Concluding the program

Like every good thing, this also came to an end. 12 weeks just fly by — faster than you can think — The program opened up a new world of open source and got me introduced to a lot of professional tools and etiquette. I appreciate the time and efforts my mentor put into this program.

Completing this internship was a dream come true, dodging tonnes of problems: internet, college, placement preparation, exams, everything. At many points in the internship, I was very certain I won’t be able to complete the project. but:

At some point, everything’s gonna go south on you… everything’s going to go south and you’re going to say, this is it. This is how I end. Now you can either accept that, or you can get to work. That’s all it is. You just begin. You do the math. You solve one problem… and you solve the next one… and then the next. And If you solve enough problems, you get to come home.

— Tail ender, The Martian.