sigstore logo

This post is authored by Hayden Blauzvern and originally appeared on Sigstore’s blog. Sigstore is a new standard for signing, verifying, and protecting software. It is a project of the Linux Foundation. 

Developers, package maintainers, and enterprises that would like to sigstore logo adopt Sigstore may already sign published artifacts. Signers may have existing procedures to securely store and use signing keys. Sigstore can be used to sign artifacts with existing self-managed, long-lived signing keys. Sigstore provides a simple user experience for signing, verification, and generating structured signature metadata for artifacts and container signatures. Sigstore also offers a community-operated, free-to-use transparency log for auditing signature generation.

Sigstore additionally has the ability to use code signing certificates with short-lived signing keys bound to OpenID Connect identities. This signing approach offers simplicity due to the lack of key management; however, this may be too drastic of a change for enterprises that have existing infrastructure for signing. This blog post outlines strategies to ease adoption of Sigstore while still using existing signing approaches.

Signing with self-managed, long-lived keys

Developers that maintain their own signing keys but want to migrate to Sigstore can first switch to using Cosign to generate a signature over an artifact. Cosign supports importing an existing RSA, ECDSA, or ED25519 PEM-encoded PKCS#1 or PKCS#8 key with cosign import-key-pair –key key.pem, and can sign and verify with cosign sign-blob –key cosign.key artifact-path and cosign verify-blob –key cosign.pub artifact-path.

Benefits

  • Developers can get accustomed to Sigstore tooling to sign and verify artifacts.
  • Sigstore tooling can be integrated into CI/CD pipelines.
  • For signing containers, signature metadata is published with the OCI image in an OCI registry.

Signing with self-managed keys with auditability

While maintaining their own signing keys, developers can increase auditability of signing events by publishing signatures to the Sigstore transparency log, Rekor. This allows developers to audit when signatures are generated for artifacts they maintain, and also monitor when their signing key is used to create a signature.

Developers can upload a signature to the transparency log during signing with COSIGN_EXPERIMENTAL=1 cosign sign-blob –key cosign.key artifact-path. If developers would like to use their own signing infrastructure while still publishing to a transparency log, developers can use the Rekor CLI or API. To upload an artifact and cryptographically verify its inclusion in the log using the Rekor CLI:

rekor-cli upload --rekor_server https://rekor.sigstore.dev \
  --signature <artifact_signature> \
  --public-key <your_public_key> \
  --artifact <url_to_artifact|local_path>

rekor-cli verify --rekor_server https://rekor.sigstore.dev \
  --signature <artifact-signature> \
  --public-key <your_public_key> \
  --artifact <url_to_artifact|local_path>

In addition to PEM-encoded certificates and public keys, Sigstore supports uploading many different key formats, including PGP, Minisign, SSH, PKCS#7, and TUF. When uploading using the Rekor CLI, specify the –pki-format flag. For example, to upload an artifact signed with a PGP key:

gpg --armor -u user@example.com --output signature.asc --detach-sig package.tar.gz

gpg --export --armor "user@example.com" > public.key

rekor-cli upload --rekor_server https://rekor.sigstore.dev \
  --signature signature.asc \
  --public-key public.key \
  --pki-format=pgp \
  --artifact package.tar.gz

Benefits

  • Developers begin to publish signing events for auditability.
  • Artifact consumers can create a verification policy that requires a signature be published to a transparency log.

Self-managed keys in identity-based code signing certificate with auditability

When requesting a code signing certificate from the Sigstore certificate authority Fulcio, Fulcio binds an OpenID Connect identity to a key, allowing for a verification policy based on identity rather than a key. Developers can request a code signing certificate from Fulcio with a self-managed long-lived key, sign an artifact with Cosign, and upload the artifact signature to the transparency log.

However, artifact consumers can still fail-open with verification (allow the artifact, while logging the failure) if they do not want to take a hard dependency on Sigstore (require that Sigstore services be used for signature generation). A developer can use their self-managed key to generate a signature. A verifier can simply extract the verification key from the certificate without verification of the certificate’s signature. (Note that verification can occur offline, since inclusion in a transparency log can be verified using a persisted signed bundle from Rekor and code signing certificates can be verified with the CA root certificate. See Cosign’s verification code for an example of verifying the Rekor bundle.)

Once a consumer takes a hard dependency on Sigstore, a CI/CD pipeline can move to fail-closed (forbid the artifact if verification fails).

Benefits

  • A stronger verification policy that enforces both the presence of the signature in a transparency log and the identity of the signer.
  • Verification policies can be enforced fail-closed.

Identity-based (“keyless”) signing

This final step is added for completeness. Signing is done using code signing certificates, and signatures must be published to a transparency log for verification. With identity-based signing, fail-closed is the only option, since Sigstore services must be online to retrieve code signing certificates and append entries to the transparency log. Developers will no longer need to maintain signing keys.

Conclusion

The Sigstore tooling and infrastructure can be used as a whole or modularly. Each separate integration can help to improve the security of artifact distribution while allowing for incremental updates and verifying each step of the integration.

CRob on open source software security education on TechStrong TV

In the Open Source Software Security Mobilization Plan released this past May, the very first stream – of the 10 recommended – is to “Deliver baseline secure software development education and certification to all.”

As the plan states, it is rare to find a software developer who receives formal training in writing software securely. The plan advocates that a modest amount of training – from 10 to ideally 40-50 hours – could make a significant difference in developer contributions to more secure software from the beginning of the software development life cycle. The Linux Foundation now offers a free course, Developing Secure Software, which is 15 hours of training across 3 modules (security principles, implementation considerations & software verification).

The plan proposes, “bringing together a small team to iterate and improve such training materials so they can be considered industry standard, and then driving demand for those courses and certifications through partnerships with educational institutions of all kinds, coding academies and accelerators, and major employers to both train their own employees and require certification for job applicants.”

Also in the plan is Stream 5 to, “Establish the OpenSSF Open Source Security Incident Response Team, security experts who can step in to assist open source projects during critical times when responding to a vulnerability.” They are a small team of professional software developers, vetted for security and trained on the specifics of language and frameworks being used by that OSS project. 30-40 experts would be available to go out in teams of 2-3 for any given crisis.

Christopher “CRob” Robinson is instrumental to the concepts behind, and the implementation of, both of these recommendations. He is the Director of Security Communications at Intel Product Assurance and also serves on the OpenSSF Technical Advisory Committee. At Open Source Summit North America, he sat down with TechStrong TV host Alan Shimel to talk about the origin of his nickname and, more importantly, software security education and the Open Source Product Security Incident Response Team (PSIRT) – streams 1 and 5 in the Plan.  Here are some key takeaways:

  • I’ve been with the OpenSSF for over two years, almost from the beginning. And currently I am the working group lead for the Developer Best Practices Working Group and the Vulnerability Disclosures Working Group. I sit on the Technical Advisory Committee. We help kind of shape, steer the strategy for the Foundation. I’m on the Public Policy and Government Affairs Committee. And I’m just now the owner of two brand new SIGs, special interest groups, underneath the working group. So I’m in charge of the Education SIG and the Open Source Cert SIG. We’re going to create a PSIRT for open source.
  • The idea is to try to find a collection of experts from around the industry that understand how to do incident response and also understand how to get things fixed within open source communities. . . I think, ultimately, it’s going to be kind of a mentorship program for upstream communities to teach them how to do incident response. We know and help them work with security researchers and reporters and also help make sure that they’ve got tools and processes in place so they can be successful.
  • A lot of the conference this week is talking about how we need to get more training and certification and education into the hands of developers. We’ve created another kind of Tiger team, and  we’re gonna be focusing on this. And my friend, Dr. David Wheeler, he had a big announcement where we have existing body of material, the secure coding fundamentals class, and he was able to transform that into SCORM. So now anybody who has a SCORM learning management system has the ability to leverage this free developer secure software training on their internal learning management systems.
  • We have a lot of different learners. We have brand new students, we have people in the middle of their careers, people are making career changes. We have to kind of serve all these different constituents.

Of course, he had a lot more to say. You can watch the full interview, including how CRob got his nickname, and read the transcript below.

Alan Shimel 00:06
Hey, everyone back here live in Austin at the Linux Foundation Open Source Summit. You know, we’ve had a very security-heavy lineup this past week. And for good reason, security is top of mind to everyone. The OpenSSF. Of course, Monday was OpenSSF day, but it’s more than that. More than Monday, we really talked a lot about software supply chains and SBOMs and just securing open source software. My next guest is CGrove or CRbn? No, no, you know, I had CRob in my mind, and that’s what messed me up. Let’s go back to Crob. Excuse me. Now check this out a little thing myself. So Crob was actually the emcee of OpenSSF day on Monday.

CRob 01:01
I had an amazing hat. You did. And you didn’t wear it here. I came from outside with tacos, and it was all sweaty.

Alan Shimel 01:08
We just have two bald guys here. Anyway,

CRob 01:14
safety in numbers.

Alan Shimel 01:15
Well, yeah, that’s true. It’s true. Wear the hat next time. But anyway, first of all, welcome, man. Thank you.

CRob 01:21
It’s wonderful to be here. I’m excited to have this little chat.

Alan Shimel 01:24
We are excited to have you on here. So before we jump into Monday, and OpenSSF day, in that whole thing, you’re with Intel, full disclosure, what do you do in your day job.

CRob 01:36
So my day job, I am the Director of Security Communications. So primarily our function is as incidents happen, so there’s a new vulnerability discovered, or researchers find some report on our portfolio, I help kind of evaluate that and kind of determine how we’re going to communicate it.

Alan Shimel 01:56
Love it, and your role within OpenSSF?

CRob 02:01
So I’ve been with the OpenSSF for over two years, almost from the beginning. And currently I am the working group lead for the developer best practices working group and the vulnerability disclosures working group. I sit on the technical advisory committee, so we help kind of shape, steer the strategy for the foundation. I’m on the Public Policy and Government Affairs Committee. And I’m just now the owner of two brand new SIGs, special interest groups underneath the working group. So I’m in charge of the education SIG, and the open source cert SIG. So we’re going to create a PSIRT for open source.

Alan Shimel 02:38
That’s beautiful man. That is really and let’s talk about that SIRT. Yeah, it’ll be through Linux Foundation.

Unknown Speaker 02:47
Yeah, we are still. So back in May the foundation and some contributors created the mobilization plan. I’m sure people have talked about it this week. 10 point plan addressing trying to help respond to things like the White House executive order. And it’s a plan that says these 10 different work streams we feel we can improve the security posture of open source software. And the open source SIRT was stream five. And the idea is to try to find a collection of experts from around the industry that understand how to do incident response, and also understand how to get things fixed within open source communities.

CRob 03:27
So we’re we have our first meeting for the SIG the first week of July. And we’re going to try to refine the initial plan and kind of spec it out and see how we want to react. But I think ultimately, it’s going to be kind of a mentorship program for upstream communities to teach them how to do incident response. We know and help them work with security researchers and reporters, and also help make sure that they’ve got tools and processes in place so they can be successful.

Alan Shimel 03:56
I love it. Yeah. Let’s be honest, this is a piece of work you cut out for yourself.

Unknown Speaker 04:04
Yes, one of my other groups I work with is a group called First, the Form of Incident Response and Security Teams. And I’m one of the authors of the PSIRT services framework. So I have a little help. So I understand that you got a vendor back on that, right? Yeah, we’re gonna lean into that as kind of a model to start with, and kind of see what we need to change to make it work for open source communities.

Alan Shimel 04:27
I actually love that good thing. When do you think we might see something on this? No pressure.

Unknown Speaker 04:32
No pressure? Oh, definitely. The meetings will be public. So all of that will go up into YouTube. So you’ll be able to observe kind of the progress of the group. I expect we’re going to take probably at least a month to refine the current plan and submit a proposal back to the governing board. We think this is actionable. So hopefully before the end of the year, maybe late fall, we’ll actually be able to start taking action.

Alan Shimel 04:57
All right. Love it. Love it. Gotta ask you, Where does the name come from?

Unknown Speaker 05:03
So the name comes from Novell GroupWise. So back in the day, our network was run by an HP VAX. But our email system plugged into the VAX and you were limited by the characters of your name. So my name Chris Robinson. So his first little first letter, first name, next seven of your last, so I ended up being Crobinsoe. And we hired a developer that walked in, he looked at it, and he’s like, ah, Crobinso the chromosome, right? Got shortened to Crob.

Alan Shimel 05:36
Okay, not very cool. So thank you. Not Crob. That’s right. Thank you Novell is right. That was very interesting days. Remember.

Unknown Speaker 05:45
I love that stuff. I was Novell engineer for many years.

Alan Shimel 05:49
That’s when certs really meant something certified Novell. You are? Yeah. Where are they now? See, I think the last time I was out in Utah. Now I was I think it was 2005. I was out in Utah, they would do if there was something they were working on.

Unknown Speaker 06:14
They bought SUSE. And we thought that that would be pretty amazing to kind of incorporate this Novell had some amazing tools. Absolutely. So we thought that would be really awesome than the NDS was the best. But we were hoping that through SUSE they be able to channel these tools and get broader adoption.

Alan Shimel 06:30
No, I think for whatever reason. There’s a lot of companies from back in those days, right, that we think about, indeed, Yeah. Anyway,

Unknown Speaker 06:45
My other working group. So we have more, but wait, there’s more, we have more. So the developer best practices working group is spinning off and education sake. So a lot of the conference this week is talking about how we need to get more training and certification and education into the hands of developers. So again, we’ve created another kind of Tiger team, are we’re gonna be focusing on this. And my friend, Dr. David Wheeler, David A. Wheeler, he had a big announcement where we have existing body of material, the secure coding fundamentals class, and he was able to transform that into SCORM. So now that anybody who has a SCORM learning management system has the ability to leverage this free developer secure software training, really, yes.

Alan Shimel 07:35
And that’s the SCORM. system. If you have SCORM, you can leverage this.

Unknown Speaker 07:39
free, there’s some rules behind it. But yeah, absolutely. It’s plugged in, we’re looking to get that donated to higher education, historically black colleges and universities (HBCU), trade schools like DeVry, wherever

Alan Shimel 07:52
Get it into people’s hands. That’s the thing to do. So that get that kind of stuff gets me really excited. I’ll be honest with you, you know, all too often, we’re good in the tech industry for forming a foundation and, and a SIG and an advisory board. But rubber meets the road, when you can teach people coming up. Right, so they come in with the right habits, because you know, it’s harder to teach the old dogs, the new tricks, right.

CRob 08:23
I can’t take the class. I know the brains full.

Alan Shimel 08:26
Yeah, no, I hear you. But no, but not only that, look, if you’ve been developing software for 25 years, and I’m gonna come and tell you, Well, what you doing is wrong. And I need you to start doing it this way. Now, I’m gonna make some progress. Because no one wants to say I know everything. And I’m not changing. People don’t just say that. But it’s just almost subconsciously, it’s a lot harder.

Unknown Speaker 08:51
It definitely is. And that’s kind of informing our approach. So we have a traditional, about 20 hours worth of traditional class material. So we’re looking at how we can transform that material into things like webinars and podcasts, and maybe a boot camp. So maybe next year, at the Open Source Summit, we might be able to offer a training class where you walk in, take the class, and walk out with a certification.

CRob 09:17
And then thinking about, you know, we have a lot of different learners. We have, you know, brand new students, we have people in the middle of their careers, people are making career changes. So we have to kind of serve all these different constituents. And that’s absolutely true. And that is one of the problems. Kind of the user journeys we’re trying to fulfill is this. I’m an existing developer, how do I gain new skills or refine what I have?

Alan Shimel 09:40
Let me ask you a question. So, I come from the security side of that. Nothing the matter with putting the emphasis on developers developing more secure software. But shouldn’t we also be developing for security people to better secure open source software.

CRob 10:02
And the foundation itself does have many, it’s multipronged. And so to help like a practitioner, we have things like our scorecard and all stars. And then we have a project criticality score. And actually, we just I, there was a great session just a couple hours ago, by one of my peers, Jacque Chester, and it was kind of a, if you’re a risk guy, it was kind of based off of Open Fair, which is a risk management methodology, kind of explaining how we can evaluate open source projects, share that information with downstream consumers and risk management teams or procurement teams, and kind of give them a quantitative assessment of this is what risks you could incur by these projects.

CRob 10:44
So if you have two projects that do the same thing, one might have a higher or lower score will provide you the data that you could make your own assessment off of that and make your own judgment. So that the foundation is also looking at just many different avenues to get this out there, focused on practitioners and developers, and hopefully by this kind of hydraulic approach, it will be successful. It’ll stick.

Alan Shimel 11:07
you know what you just put as much stuff on the wall and whatever sticks sticks man up. So anyway, hey Crob. Right. I got it right. Yep. All right. Thank you for stopping by. So thank you for all you do, right. I mean, it’s a community thing. These are not paid type of gigs, right. Sure. Yeah. No, and I thank you for your for your time and efforts on that.

CRob 11:30
Thank you very much. All right.

Alan Shimel 11:31
Hey, keep up the great work. We’re gonna take a break. I think we’ve got another interview coming up in a moment. And we’re here live in Austin.

white house at dusk

Today the White House convened the White House Cyber Workforce and Education Summit to gather government and private-sector leaders to discuss how to address the labor shortage and other challenges for U.S. cybersecurity. The meeting included the nation’s top cybersecurity and workforce policy decision makers, including the National Cyber Director and the Cabinet secretaries from the Departments of Commerce, Homeland Security, and Labor and the Under Secretary of Education. 

Jim Zemlin, Executive Director of the Linux Foundation, was invited to participate.

During the meeting, Jim emphasized the need to “shift left” security training and best practices as much as possible. Addressing security at the beginning of the technology supply chain is more efficient and effective – it is being proactive rather than reactive. This begins with providing open source practitioners with the knowledge and skills to build security into the development of the software we all depend on.  

Addressing security at the beginning of the technology supply chain is more efficient and effective – it is being proactive rather than reactive.

He emphasized the commitment of the Linux Foundation to partner with industry leaders to provide no cost or low cost training and certification in cybersecurity beginning with our Developing Secure Software course, which is 15 hours of training across 3 modules (security principles, implementation considerations & software verification). The goal is to teach software developers how to develop more secure software from the beginning because that is much more efficient than finding and remediating vulnerabilities.

Since launching it this spring, over 10,000 students have started the course and over 1,000 completed it and received their verifiable certification. But this is just the beginning. Over the next few months, the Linux Foundation will launch new courses and certification exams on topics such as: 

Addressing cybersecurity challenges through investments in the workforce is about more than hiring and training more cybersecurity professionals. Providing effective training for individuals involved at all points in the software development lifecycle is key to success – kind of like building security into a building at the beginning rather than just hiring security guards to protect it. 

Providing effective training for individuals involved at all points in the software development lifecycle is key to success – kind of like building security into a building at the beginning rather than just hiring security guards to protect it. 

The goal of building a more robust cyber workforce is part of the recommendations developed earlier this year after the White House-convened Open Source Software Security Summit in February and a follow-up Summit in May. You can read about the recommended 10 streams of investment and the entire Open Source Software Security Mobilization Plan here. And consider joining the OpenSSF to help make our software supply chain more secure by building an expert community, targeted initiatives, and best practices.

We encourage you to  enroll in the Developing Secure Software training from the OpenSSF. It is free for everyone through Linux Foundation Training & Certification. You can also enroll through edX for free in audit mode or with a verified certificate of completion for an additional fee.

For more information on the Summit, the White House’s fact sheet summarizing the Summit is here.

Jamie Thomas on TechStrong TV

Jamie Thomas is the General Manager, Systems Strategy and Development at IBM and is also the OpenSSF Board chair. She sat down with Alan Shimel of TechStrong TV during OpenSSF Day in Austin to share about OpenSSF and how the open source community is rallying together to increase the resilience of open source software. 

You can watch the full interview or read the transcript below. But, since we are all busy, I have pulled together some of the key points Jamie made from the interview:

OpenSSF is focused on a proactive posture. How do we prevent these kinds of events? And so to do that, we think there’s a number of things we have to do: 

  • First and foremost is education, of course, in terms of basic security education for developers.
  • Another key tenant is how do you put automation on steroids? So the automation and best practices that are reflected in that automation that open source projects can consume? How do you get that out to the most critical projects, and then provide some support for the long tail projects
  • It’s also about working, frankly, with other industry consortia as well as the government. In Particular, we’ve been working with the US government in the OpenSSF to define what are some actions that are really going to make a difference. 
  • And I think critical to all of this is getting collaboration across the different insights from the governing body, which includes a lot of technology firms, as well as commercial firms. Like there’s a lot of financial firms actually involved in the governing body. What are the key elements that we really need to address first. So getting those priorities set, and then having an execution agenda and really getting something done in the short term, I think is really going to be important for this group.

In the world of cybersecurity, you often learn that no one pays attention to a lot of things unless there’s a huge compelling event. And that’s what log4j was. So while it was not desired, it was helpful in that vein. . . So coming out of all of the meetings that we’ve had, the collaboration that we’ve had across the industry, it is going to be imperative that we execute, and that the things that we have identified as top priorities that we make measurable progress on those projects this year. That’s the importance of this OpenSSF day here today in Austin, which is allowing us, with a key set of stakeholders, to start to share perspectives of the projects that are underway, and how others can engage in those projects. And how, once again, working together, we can actually make a difference. 

 Working together, we can actually make a difference. 

We are turning the corner on a new level of commitment around security, there’s always been a commitment in open source around innovation, around feature function. I mean, that’s what’s driven open source and allowed it to be so successful. And for others, other corporations like IBM, we take an enormous advantage out of that, right, we’ve all gotten a huge advantage in productivity out of that. But now, it’s really about turning the focus a little bit more, getting that focus on security, so that we can use open source and continue to have that productivity, but with confidence as we go forward.

How do we make it easy for the maintainers of these open source projects? How do we make it easy for the contributors, because without doing that, it will not have the consumption by developers at large.

Alan Shimel 0:06
Hey, everyone, we’re back here live at the Linux Foundation’s Open Source Summit here in Austin, Texas. And as we mentioned earlier, today is is a day of, I don’t know if you want to call it daughter-sister foumdations or satellite conferences, the main event really starts tomorrow. But there’s several important foundations who are holding conferences today. One of which, and kind of the one kind of the nearest to me is the Open Source Security Foundation. OpenSSF. And we are really happy to be joined by Jamie Thomas, who is the governing chair or the chair of the governing board. Jamie, welcome to our show. Thanks for joining us. So, look, when you’re not busy running, or being Chair of the Board for OpenSSF you have a day job as well. If you want to share with our audience feel free.

Jamie Thomas 1:05
Well, first of all, Alan, thanks for having me. I’m really pleased to be here to talk about OpenSSF. But I am a general manager at IBM responsible for systems development and delivery as well as IBM’s enterprise security program. And enterprise security, of course, is how I got involved in this particular topic.

Alan Shimel 1:22
Absolutely. And that look, that is a world and job unto itself. And we could probably do a few hours on that. But we’re going to focus on on OpenSSF today. So, you know, for most of our audience is familiar, we’ve covered,we’ve had the pleasure of speaking with Brian from OpenSSF a few times. It was a nice idea I think when it was first conceived about yes, we need to do something about security, about the security of open source tools specifically.

Alan Shimel 1:53
And then kind of all hell broke loose. You know, sometimes, sometimes things just work like that. Right? History runs in currents. So we started the OpenSSF. And then we had this spate of supply chain security issues and the whole SBOM thing with the White House. And then like kind of the cherry on top was log4j, it was around when January or December of last year. And that’s really, I guess, accelerated has it accelerated. Maybe you had big plants to begin with. Talk to us a little bit about kind of the whole OpenSSF and how it all came together. And what’s happened?

Jamie Thomas 2:33
Well, I think it was very fortuitous that the industry did come together last year with the Linux Foundation to create a new governing body around open source security called the OpenSSF. Because as you say, not long after that we had this industry compelling event log4j and realized the industry had already had we’d already had Solarwinds that year before, which also ruined our holiday in December. We had Kaseya, we had a number of these big supply chain attacks.

Jamie Thomas 3:00
But the difference I would say in log4j is just the predominance of the asset in code. It had been out there for over 20 years, it was a very utilized, a very popular piece of code. And so it affected a lot of software.

Jamie Thomas 3:14
So one of the things that you realize when this kind of thing happens, it’s not just about your fidelity of being able to identify it and get it patched. But for all those downstream consuming organizations, how fast do they roll out these patches, because we’re talking about a huge amount of affected software. So I think that there’s nothing like a true test of your governing body. And this was actually a real test run of what we needed to do in OpenSSF. And, of course, it garnered a lot of tension from the US government and other entities that we can we can talk more about.

Alan Shimel 3:48
Sure. Okay. So let’s talk a little bit about the charter or the mission of the OpenSSF. And it’s something I brought up to you off camera, which is okay. Log4J, let’s make that the poster child for a second. So log4j is basically this open source component, if you will, right, that many, many, many, many, many applications have incorporated into their package, if you will, into their source code. And it’s not, look, I’m not blaming the log4j developers or anything. There was a defect, I don’t even want to you know, it became a vulnerability but there’s a defect, while software has defects that we haven’t even found yet. But nevertheless, this one kind of went public and then we saw exploits with it and in the wild and such as the world of security we both live in. What is the chart is that what OpenSSS is about to prevent, or not prevent, but deal with future log4j kinds of events?

Jamie Thomas 4:55
Well, I think first and foremost, OpenSSF is focused on a proactive posture. Right. So how do we prevent these kinds of events? And so to do that, we think there’s a number of things we have to do. First and foremost is education, of course, in terms of basic security education for developers.

Jamie Thomas 5:14
Another key tenant is how do you put automation on steroids? So the automation and best practices that are reflected in that automation that open source projects can consume? How do you get that out to the most critical projects, and then provide some support for the long tail projects, if you will?

Jamie Thomas 5:31
It’s also about working, frankly, with other industry consortia as well as the government. Particularly we’ve been working with the US government in the OpenSSF to define what are some actions that are really going to make a difference. And I think critical to all of this is getting collaboration across the different insights from the governing body, which includes a lot of technology firms, as well as commercial firms. Like there’s a lot of financial firms actually involved in the governing body. What are the key elements that we really need to address first. So getting those priorities set, and then having an execution agenda and really getting something done in the short term, I think is really going to be important for this group?

Alan Shimel 6:14
Well, look, a lot of people look at what you guys have done, and you’ve gotten stuff done, right? There’s been a tremendous groundswell of support. And granted, log4j didn’t hurt you in that regard. But there are others. But there’s been a tremendous groundswell, right, there’s been a I think, about $30 million raised right, between some of the biggest names in tech kicking in here. There’s been the White House and CISA involvement. So it’s certainly, for a relatively new foundation, it has really garnered a lot of, I don’t want to say market share, but a lot of publicity, a lot of attention.

Alan Shimel 7:02
Now, of course, the question is, okay, how does this translate to rubber meets the road? How do we prevent the next slide? I don’t know if we can prevent the next log4j. But how do we minimize that?

Jamie Thomas 7:14
minimize the impact Exactly. Because I would say, if you look at what happened with log4j, the level of preparedness was not there. So how do you get it remediated fast? And how do you identify it? How do you help the open source projects be more effective. In this case, it was of course tied to the Apache Foundation. But not only that, how did the commercial entities then take advantage of that patch and act expeditiously to benefit the clients?

Jamie Thomas 7:45
So I think there’s a real opportunity here. In the world of cybersecurity, you often learn that no one pays attention to a lot of things unless there’s a huge compelling event. And that’s what this was. So while it was not desired, it was helpful in that in that vein, so coming out of all of the meetings that we’ve had, the collaboration that we’ve had across the industry, is going to be imperative that we execute, and that the things that we have identified as top priorities that we make measurable progress on those projects this year. And I think that’s the importance of this OpenSSF day here today in Austin, which is allowing us, with a key set of stakeholders, to start to share perspectives of the projects that are underway, and how others can engage in those projects. And how, once again, working together, we can actually make a difference.

Jamie Thomas 8:36
I think this on this ongoing level of engagement, making sure that we have the right stakeholders engaged, is going to be important to make progress. And as you know, in the world of open source, the nice thing about OepnSSF is we do have the ability to hire critical roles that can focus on this full time. Because the nature of open source typically is that it’s a it’s a volunteer army. Right? And there’s 1000s and 1000s of volunteers out there. But then how do we help with these resources, enable those volunteers to be more effective.

Alan Shimel 9:10
And frankly, that’s been one of, I think, the key ingredients to the Linux Foundation’s formula for success is, you know, herding. It’s a bit like herding cats herding the open source community, it’s vast, the 1000s, hundreds of 1000s, millions, but you need a few full timers who are, this is their day job, right? This is their this is what they do.

Alan Shimel 9:36
Jamie, I want to talk a little bit for people who are watching this now at home. Or maybe, you know, recorded later on. They weren’t here. They didn’t get what was happening, especially today, which is kind of you know, the OpenSSF’s day. Give them if you don’t mind a little bit of maybe a synopsis of what they’re missing.

Jamie Thomas 9:58
Well, we just got started of course, so we have a little bit more to go today, of course, in terms of the actual kickoff of OpenSSF Day. But I think what I see is real commitment, particularly from the presenters I’ve seen so far, a commitment that they’ve all personally made and outside of their day jobs, frankly, to make a difference in security for open source software. And that’s really the key here.

Jamie Thomas 10:23
Are we turning the corner on a new level of commitment around security, there’s always been a commitment in open source around innovation, around feature function. I mean, that’s what’s, loved it, you know, that’s what’s driven open source and allowed it to be so successful. And for others, other corporations like IBM, we take an enormous advantage out of that, right, we’ve all gotten a huge advantage in productivity out of that. But now, it’s really about turning the focus a little bit more, getting that focus on security, so that we can use open source and continue to have that productivity, but with confidence as we go forward. And I’ve really been, I’ve been impressed with all the speakers today and their personal commitment to this topic. And, and that’s really impressive. And I think we’ll see that for the rest of the day as well.

Alan Shimel 11:12
I’m gonna come back to it that to you in one second. I want to touch on something else, though. And that and that is this look, I’ve been in security for 25 going on 30 years. Well, security 25, IT 30 plus years. And, I, you know, if I had a nickel for every survey I read that said security is one of the top three priorities of IT, or the CIO, or an organization, I’d be a rich, rich person right now. But as like I always said their arms were too short to reach their pockets oftentimes. And it wasn’t until something bad happened like a log4j. You know, some incident. Yeah, Code Red. And I could go through a whole history of the things that people trying to get religion, right.

Alan Shimel 12:00
Excuse me, sometimes it takes that for them to get religion. I don’t, I don’t know why. I hope I always hope that it changes that people finally do start taking it seriously. I think for the OpenSSF though, the important thing to remember, especially in our audience, this is a fact we give them all the time, today’s applications, they are 75% 80% open source components added kind of stitched together with maybe 20- 25% of you know, sort of original code, if you will.

Alan Shimel 12:35
And so if someone’s not watching the store on those open source components, whether they’re artifacts or scripts or whatever. Your it’s only a matter of time. It’s not if, it’s when right. And so that’s why I think this is such a vital, it’s such a vital function, this Foundation. Something needed to happen. Yeah. And this is a perfect place for it. And we I step off the soapbox, you mentioned a couple of speakers anything stand out to you or that you can kind of clue our audience and tell

Jamie Thomas 13:11
I think other than the commitment of there’s a keen focus on making it easy for the developers, right? How do we make it easy for the maintainers of these open source projects? How do we make it easy for the contributors, because without doing that, it will not have the consumption by developers at large, right. And I know this, even inside a corporation, we have the same challenge, really, it’s all about codifying the best practices in an automation framework. And, you know, whatever that is for your organization, that’s going to be critical. And that’s why it’s so critical for these open source projects.

Jamie Thomas 13:45
You know, I think that with the right approach, we will make a difference. But it also, as you said, require stakeholders involved to continue to educate their organizations about why is it important, because all of us actually have the ability to increase the number of contributors we have on these projects, to contribute our expertise. And that’s going to be very important. I think that we as the governing body and other organizations really create a sustaining promise around open source. So it’s not just what the OpenSSF is doing itself. But how we enable that to be successful in the long run. Because we’re all getting the advantage from open source, and, like IBM we of course, it’s IBM plus our company, Red Hat, it has a little bit to do with open source. But those kinds of efforts and keeping that keen focus are going to be very, very important as we go forward.

Alan Shimel 14:38
There’s no doubt about it. It also goes back to what we said before is, look there’s a new lock log4j kind of horizon out there every day where there is so you’re not going to prevent them. You’ve you’ve got to put in your response. You’ve got to have your protocols in place.

Jamie Thomas 14:59
I will tell you that It, you know, I have a window into cyber operations, which is my job every day at IBM. And we’re getting over 100 billion events a day. So that gives you kind of the context for what you got to deal with and landscape. And product security, of course, is one of those triggers. If it’s not, if you’ve got malware, if you got issues, they’re going to be one of your events, right? So it’s a little bit of a reflection on our responsibility to enable effective cyber operations for organizations. I mean, we have a huge responsibility. But we have a huge opportunity here. And I think I want to make heroes out of developers for really worrying about security. That’s kind of one of the goals.

Alan Shimel 15:41
You know, look, you’re preaching to the choir here, because, you know, I started devops.com in 2013, 2014. And I did it because, as a security person, I thought it was the best thing that happened in security. If we can get developers security, aware, security conscious, that’s half the battle. And, you know, for a long time it was it was an uphill battle. Let me say that. But this whole notion of what we call DevSecOps and making security for developers, it’s really gone mainstream. Right. And I think part of that is realizing is developers, security is everyone’s responsibility is a very overused thing.

Alan Shimel 16:24
Developers are not security people, but I’ve never met a developer in my life who says, I’d like to develop insecure software, right. I want to use an old version of an open source, you know, component that has some known vulnerabilities. None of them want to, we don’t have pride in our work. It’s just we need to make it easier for them to do, and I think that’s something OpenSSF can really help with.

Alan Shimel 16:55
Anyway, I know you’re busy as heck, I want to thank you for coming down and hanging out with us a little bit. To you, Brian, the whole OpenSSF team, keep up the great work well, we’re expecting big things. No pressure. We’re expecting big things from you guys. You really make make a difference.

Jamie Thomas 17:11
Thank you, Alan. I’m really pleased to be here today and immerse myself in this topic and get to know many of the players that are here today. And thanks. Thanks for the opportunity to chat. No problem.

Alan Shimel 17:20
Just before we leave real quickly, the OpenSSF website. I think it’s openssf.org. So go check it out. If you’re not here in person, I believe it is virtual, as well. We love to see you as part of it in support the OpenSSF. We’re gonna take a break here in Austin. We’ll be back in a bit.

Stephen Hendrick and Matt Jarvis on TechStrong TV

While open source software is ubiquitous and generally regarded as being secure, software development practices vary widely across projects regarding application development practices, protocols to respond to defects, or lack of standardized selection criteria to determine which software components are more likely to be secure. Consequently, software supply chains are vulnerable to attack, with implications and challenges for open source project communities. 

To help improve the state of software supply chain security, the Linux Foundation, the Open Source Security Foundation (OpenSSF), Snyk, the Eclipse Foundation, CNCF, and CI/CD Foundation conducted research and released the findings in the report, Addressing Cybersecurity Challenges in Open Source Software, during the 2022 Open Source Summit North America. 

At the Summit, Stephen Hendrick, LF’s Vice President of Research, and Matt Jarvis, Director of Developer Relations at Snyk, sat down with Alan Shimel of TechStrong TV to discuss the findings and next steps. Here are some key takeaways:

Alan: “ I think we’re always disappointed when we do the surveys that we find out, you know, beyond the lip service that gets paid to security, what actually is going on under the covers, and we’re always wishing for and hoping for more. That being said, I don’t want to be pessimistic. I am of the glass half full opinion that we are doing better and more security now than we probably ever have done.”

Stephen: “On the issue of, do organizations have an open source security policy. What we found was 49% said they had one, that’s good. 34% did not. And 17% said they don’t know.”

Matt: “In larger enterprises… you’ve got that kind of ingrained culture over a long time in terms of security and about how you consume software. . . the hardest problem in security isn’t really about technology at all. It’s always about people and culture. . . We’ve got two kinds of things happening in almost a perfect storm. At the same time, we’ve got this massive rise in supply chain attacks on open source, because, you know, it’s a victim of its own success. And attackers have realized it’s a lot easier to get into the supply chain than it is to find zero days in end user applications. So you’ve got that going on, where all of a sudden, folks are going, well, everything we do is based on open source, like, what do I do about security? And then, as Steve pointed out, you’ve got this, this ongoing, massive transformation of how we develop software, you know, this superfast high velocity.”

Stephen: “We asked. . . how do you intend to improve on the situation?. . . Top of the list was organizations are looking for more intelligent tools. . .  That was at 59%. . . Right behind that at 52% was a strong desire to understand and essentially codify best practices for how to do secure software development”

Matt: “Culture change is such a big part of how you make that transition from your kind of old school, security as gatekeeper kind of function, to this thing, where we put it to the developers, because the developers are the ones who, you know, you fix it at the developer eyeball before it’s got anywhere near production. That is the cheapest.”

Stephen: “You know, I did a report last year on SBOMs. And I gotta tell you that factors right into this. . . we did some stats in this survey on dependencies, you know, both direct and transitive, and found, really, sort of low levels of strong, strong security around organizations understanding the security posture of all these different dependencies and dependencies of dependencies. Really low numbers there. SBOMs would go so far in helping sort all that out.

“They’re going to give you knowledge about the metadata, it’s gonna give you usability, so you know that you’re licensed to use the stuff, and it’s going to know if it was good, if you trust that not only what you’re looking at for metadata is not falsified, but also understanding quite clearly, you know, what’s been fixed, what hasn’t been fixed from a vulnerability standpoint.”

Matt: “I think when people think about policies, they think, Oh, this needs to be like a 100 page document of some kind, you know, then it becomes overwhelming, but really a policy can be a one liner.”

Watch the full interview and read the transcript below.

Alan Shimel 0:00
This is Alan Shimel witih Tech Strong TV. We’re back here live in Austin streaming out at you from the Open Source Summit. We’re having a great time. This is our third day of coverage here, though technically, it’s only day two of the event. It’s a long story, but we’ll talk about it later. Let me introduce you to our next two guests. This is a conversation I was really looking forward to. To my left here is a gentleman who’s been on fixture on TV a few times with us and talked in person, great person. He’s the VP of Research for the Linux Foundation, Steven Hendrick. Steven, welcome.

Stephen Hendrick 0:42
Thanks, Alan.

Alan Shimel 0:43
And joining Stephen and I from our friends at Snyk. Matt, Javis. And Matt, if I’m not mistaken, your director of developer relations. Welcome.

Matt Jarvis 0:55
Thank you.

Alan Shimel 0:55
So Steven, and Matt presented, was it yesterday. Yeah, yesterday, on a new survey that you guys recently announced and revealed and report. Why don’t you if you don’t mind share with everybody

Stephen Hendrick 1:10
Sure, I’d love to. OpenSSF is a very large project inside of the Linux Foundation. Brian Behlendorf, yeah. And so at his request, we went out and did a survey into sort of what’s happening in the open source space as far as secure software development. So we put together a survey in March, we fielded that it in April, we wrote it up in May, and had it produced in June. And so it’s being released here at the event. I think that happened yesterday morning. We did it in partnership with Snyk. So that’s why we’ve been working together with the messaging on all this.

Stephen Hendrick 1:55
And it’s it was not a surprise from the standpoint of what the results were. But it wasn’t, I was a little disappointed in kind of where we are at this point, from the standpoint of the uptake of attention to security when it comes to open source. So anyway, so we’ve got information that talk a little bit about, you know, where we are, you know, help, sort of, to understand the context of the problem. And then they have information about what people are doing about it. And it’s, that’s more exciting in many respects, because good things are happening.

Alan Shimel 2:33
I agree. So first of all, look, I think we’re always disappointed when we do the surveys that we find out, you know, beyond the lip service that gets paid to security, what actually is going on under the covers, and we’re always wishing for and hoping for more. That being said, I don’t want to be pessimistic I am of the glass half full opinion that we are doing better and more security now than we probably ever have done. Yeah. Yeah, I agree. That being said, Before we dive into it, I just wanted to really just quickly, so OpenSSF is the website? Yeah. And I’m going to assume that the report is there for anyone who wants to download it. That’s right. Let’s take let’s say that up front for people at home following along, or whether it’s live while you’re watching this.

Stephen Hendrick 3:30
It’s on the Snyk site. It’s so on the Linux Foundation site and it’s on OpenSSF. So yeah, it’s everywhere.

Alan Shimel 3:37
I think we might have covered it via Snyk over on Security Boulevard.

Matt Jarvis 3:41
I think I did some I did some press interviews. Before flying out here. So, yeah, we may have.

Alan Shimel 3:46
It may very well be on our Securityf Boulevard site. But nevertheless, it’s out there for people. Yeah. Let’s dive in now, though, what was some of the findings? Steven?

Stephen Hendrick 3:55
Sure. Well, let’s see what we’ll start with this, this whole issue of do organizations have an open source security policy? And what we found was 49% said they had one, that’s good. That’s good. 34% did not. And 17% said they don’t know. Everybody uses open source – 98% of organizations use it so. So they don’t even know they don’t know if they have won or not. So if you take put aside the don’t knows at this point, you’ve got about a 60/40 split between use that don’t have have a policy and don’t have a policy.

Stephen Hendrick 4:37
I mean, if you look at little more deeply into that, what you find is that small companies are more likely to not have a policy and that’s not surprising, they have resource constrained so it’s harder for them to have CISOs and OSPOs and policies be it for either just software development or open source software development so I can understand challenges there. So, but the idea of when you, even if you look at company size, we still ended up with about 30% of large in very large organizations that don’t have a policy for open source software development.

Alan Shimel 5:16
So a couple of thoughts. First of all, I empathize with small SMB businesses. We are an SMB business, but in today’s day and age, and maybe it’s when you’re a hammer, everything looks like a nail, but in today’s day and age, how do you not have security policies?

Matt Jarvis 5:40
Yeah, I mean, I think that there’s, there’s a couple of different things at play there. I mean, you know, addressing open source security, you know, is, is more complex than than it seems, because it’s not just about the code itself, you’ve kind of got to understand how open sources is created, how projects are governed, because governance can have a big play into, you know, whether we’re looking at some of those recent things around the sort of protestsware movement, where we’ve seen maintainers kind of go rogue, you know, and this comes down to single maintainer governance projects. And you need to take those things like governance into account if you’re going to base your business on something.

Alan Shimel 6:24
Right, but you just said, and that’s a completely loaded question. I would bet, if I was a betting man, right, that at the large enterprise level, you’re 100%, correct. At the SMB level, if you ask most of these people a threshold question of where is your open source software, it’s 10 o’clock, where’s your open source software? And a lot of them don’t know, because they’re SAS-ops companies, right? They don’t they don’t have a server closet, their cloud installation – they run it on SAS. And so the beautiful part about SAS is, one of the nice things about it, is you don’t know what’s behind the curtain, you just know, you log in on the website, and you’ve got all your information there that you need. Are they using an open source database? Are they using, you know, what are they using behind the curtain? A lot of smaller companies don’t know. And as part of their due diligence, they don’t dig that deep. So I could, again, I can empathize with the larger ones, the larger enterprises, though, that’s a problem that is that I think,

Matt Jarvis 7:39
You know, in a lot of those larger enterprises, you’re you’ve got that kind of ingrained culture over a long time in terms of security and about how you consume software. And you know, the hardest problem in security isn’t really about technology at all right? It’s always about people and culture. And I think, you know, probably in a lot of larger organizations, you’ve got kind of, you know, that sort of friction of, well, we’ve always done it like that.

Stephen Hendrick 8:07
Well, you also have a lot of change going on from standpoint of how software is being developed. And I think that’s part of the problem as well, which is that, you know, change is always hard for people. And especially given the rapid evolution of tools and standards, in essence around how we should do security for software. Everything’s changing so quickly, it’s I think, it’s probably hard for people to keep up.

Matt Jarvis 8:33
Because we’ve got these two kind of things happen in almost a perfect storm. At the same time, we’ve got this massive rise in supply chain attacks on open source, because, you know, it’s a victim of its own success. And attackers have realized it’s a lot easier to get into the supply chain than it is to find zero days in end user applications. So you’ve got that going on, where all of a sudden, folks are going, well, everything we do is based on open source, like, what do I do about security? And then, as Steve pointed out, you’ve got this, this ongoing, massive transformation of how we develop software, you know, this superfast high velocity,

Alan Shimel 9:10
I blame DevOps?

Matt Jarvis 9:15
Unless you do, and unless you can transform, you know, someone’s going to eat your lunch, right? Because there’s some hungry competitor behind you who’s disruptive and who does have a superfast software delivery pipeline. They can deliver new features, they know how to analyze the data. And so for a lot of big organizations, you’ve got these two big problems happening right at the same time, because that change in software development requires a completely different approach to security. You know, this space that it’s the thing that sneaks talk about all the time about developers?

Alan Shimel 9:45
I mean, you look at let’s say, the Phoenix Project by Gene Kim, right. And that’s based on a book called The Goal. Yeah, right. And the thing about, so The Goal is about manufacturing, but really the principle behind The Goal, and I think Gene tried to capture that in the Phoenix Project, is that, look, as soon as we kind of erase one bottleneck, we see that next bottleneck right behind it. Don’t think that once you get rid of that bottleneck its smooth sailing, it’s not. We have massively, revolutionarily speeded up the pace of software development. We did it in large part by creating this software factory pipeline, CI/CD, DevOps kind of things. The enabler of that was having this massive library of open source that we can assemble into a very high quality software.

Alan Shimel 10:43
Man, we blew through that roadblock at 150 miles an hour. The wall we hit right after is, wait a second, now, that’s become a huge security problem. Right? So for companies that are developing their own code, this is a major thing? Knowing that though, and still telling me that 30% of the companies don’t have a policy around it, scary. Yeah, it is.

Stephen Hendrick 11:09
Well, let’s we should we should talk about what people are doing about trying to deal with this. Right.

Alan Shimel 11:14
Here’s the good news.

Stephen Hendrick 11:14
So we asked a question, which was, okay, so how do you intend to improve on the situation? What do you what are you doing, and we had quite a long list of responses. Top of the list was, organizations were looking for more intelligent tools from a threat where repetitive security focus. So we’re talking SCA, SAS DAST, IAC, you know, all the usual suspects, and looking really, to those tools to be able to help them improve their security posture. So that was top of the list. That was 59%.

Stephen Hendrick 11:52
And then right behind that at 52% was a strong desire to understand and essentially codified best practices for how to do secure software development. That was really encouraging, because we know all about best practices. Yep. Know exactly what they all are. In fact, David Wheeler, at LF.

Stephen Hendrick 12:14
We had David A. Wheeler. We interviewed David yesterday and we have a follow-up as well.

Stephen Hendrick 12:32
He and I had lunch yesterday, and we were talking about this, because I said, you know, how many best practices do you have? So we know counted them all up – he’s got like, 150, 160. So that’s kind of daunting. And he said, like the last 25 to get to the highest level can take, in some cases, years to master. So this is, this is, despite understanding what these best practices are, it’s still very challenging to wrap your head around what is necessary to be successful there.

Matt Jarvis 13:00
It’ll be good. And partly because, you know, as we were just talking about that, that culture change is such a big part of how you make that transition from, you know, you kind of old school, security as gatekeeper kind of function, to this thing, where we put it to the developers, because the developers are the ones who, you know, you fix it at the developer eyeball before it’s got anywhere near, you know, production. That is the cheapest.

Alan Shimel 13:28
Say 10 to 100x cheaper to do there.

Matt Jarvis 13:30
I mean, we look at the other interesting thing here that’s slightly tangential to this, but is like how many developers there are in the world, right, and how many we anticipate there being you know, there’s something like, I think the anticipation is something like 30 million developers in the world, and there’s only, like, a tiny proportion of security folk.

Alan Shimel 13:51
So I go by GitHub accounts. Right? There’s about 70 plus million GitHub accounts right now. So let’s assume it’s not one-to-one. But I think it’s safe to say this 40 to 45 million developers, probably growing at somewhere in the area of 10% a year.

Matt Jarvis 14:08
And security professionals aren’t aren’t growing at that rate.

Alan Shimel 14:13
So security professionals are growing because we’re starting to see, look, when I came up, you didn’t have a cybersecurity major in college. We’re seeing schools churn out cybersecurity majors. Are they security professionals? I’ll leave it to you. But there are people coming out here who want to work in security, but not anywhere near I mean, you’re talking here in here.

Alan Shimel 14:39
Here’s an interesting thing, though. And I think it’s what’s turning up the heat on all of this, is that this is getting major focus from the White House, from the federal government. The whole world is saying hey, this is a problem. This is a big problem.

Stephen Hendrick 14:58
Well, you know, you got to do something you You know, I did a report last year survey on SBOMs. Yep. And I gotta tell you that factors right into this No, of course, because, you know, we did some stats in this survey on dependencies, you know, both direct and transitive, and found, really, sort of low levels of strong, strong security around, you know, organizations understanding the security posture of all these different dependencies and dependencies of dependencies. Really low numbers there.

Stephen Hendrick 15:34
SBOMs would go so far in helping sort all that out, because, you know, SBOMs, they’re going to give you knowledge about the metadata, it’s gonna give you usability, so you know that you’re licensed to use the stuff, and it’s going to know if it was good, if you trust that not only what you’re looking at for metadata is not falsified, but also understanding quite clearly, you know, what’s been fixed, what hasn’t been fixed from a vulnerability standpoint.

Alan Shimel 15:59
So I’ll tell you over the last two days, we’ve, we’ve done a lot of interviews, but no shortage of people talking about SBOMs and SBOM solutions. I think we’re going to see, just like everything else in technology, we’re gonna see sort of a camp Cambrian explosion of SBOM solutions out there, and then the market will figure out which ones make sense, which ones don’t. My fear is that we, we think SBOMs are a magic bullet for supply chain security, because we have a tendency of doing that in security.,

Matt Jarvis 16:33
Ultimately I think the real challenge here is going to be the chain of trust part of that, right? Because what’s an SBOM at the end of the day, it’s a text file with some stuff.

Alan Shimel 16:43
Oh, no, but they’re, you know, they’re, they’re building some elaborate text files in there.

Stephen Hendrick 16:49
It’s a lot of good metadata. Yeah, but one more point I want to touch on, though, is that the number three issue from the standpoint of doing improvements to your software security posture was more automation. So IAC tools ended up ranking very highly from the standpoint of helping you address that particular need, and just for our audience, IAC (infrastructure as code), right. Okay. So that one actually surprised me, because this whole idea of, you know, developers/manual activities, not only that is a great way to invite in problems. And so more automation, ultimately, is better.

Matt Jarvis 17:33
We did some some work last year as part of our cloud native application security report. And what was really interesting there was, you know, we kind of use high levels of development automation, i.e.automated CI/CD pipelines and all that stuff as a as a proxy for how far along your cloud native journey you are. I think it’s a pretty reasonable proxy to take. And in organizations with those high levels of of deployment automation, for a start, we see much higher levels of adoption of security tooling, because automation gives you lots of places where you can hook in other automation. But, most importantly, we see massive reduction in the time to fix the vulnerabilities. Because through directly through a direct correlation to okay,

Alan Shimel 18:25
I’ve been in security a long time. We had a vulnerability solution company I founded back in 2005. Back then, there was a company called Hercules – Citadel was the company Hercules was the product, right? They were doing, you know, they were pushing automated remediation. There’s several companies today that have automated remediation. For whatever reason, up until now, organizations have been hesitant to adopt automated remediation, because they’re afraid it’s going to break something else if it’s in a totally automated situation. Now, doing this further left in the in the development pipeline, if it’s broken, supposedly, that should come up in testing.

Matt Jarvis 19:16
I mean, this is again what we see when when companies adopt Snky, is like, you know, the the automated remediation part of in terms of automated fix PRs, you know, is, is probably not where people start, but very quickly, they go out.

Alan Shimel 19:33
Yeah, look, this is a no brainer. Yeah, absolutely, because it goes back to what I said before blame DevOps, right. If we are going to automate the CI/CD pipeline, we’re going to automate building software, the answer cannot be that we’re going to manually do security. It just doesn’t work. It’s a disconnect.

Matt Jarvis 19:54
Yeah, I mean, and it’s an anti-pattern in terms of velocity, right. I mean, velocity is the key differentiator for whether, sort of, visitors in the cloud era are going to survive. Absolutely, because if you don’t have velocity, you are probably don.e.

Alan Shimel 20:08
No, but a lesson we learned in security, or we should have learned over the last 25 years, is if we are going to drag our heels and dig our heels in and say no, no, no, no. You know what? The train leaves the station without you. Yeah. So easy to get on board and figure out yes, we can. And here’s how we’ll get out of the way. Right? Lead, follow, get out of the way. Security cannot be the drag on this because velocity is too important.

Matt Jarvis 20:36
And where we see folks who’ve successfully made this transition to developer-first, you see this sort of this change in security teams from kind of being gatekeepers to being enablers.

Alan Shimel 20:41
DevSecOps right there. Yeah, you just the just the the heart of it. That’s it. Steven, anything else?

Stephen Hendrick 21:02
So what’s the answer to this issue of not having a security policy? I mean, is it, Do you need to start with the CISO? Did you start with an OSPO? Do you need, or at least part time roles and people in organizations, you know, in those functions if you are small? I’m not sure what the answer is, but I mean, we need one.

Matt Jarvis 21:27
I think when people think about policies, they think, Oh, this needs to be like 100 page document of some kind, you know, this is it becomes overwhelming, but really a policy can be a one liner.. I mean, we, we have this conversation a lot when people start to adopt security scanning, right, they’ve done no security scanning before, and they scan this software, and they go, Oh, my God, I’ve got like 500 vulnerabilities. What do I do? But you’ve got to just pick a starting point. Right. And I mean, usually, you know, a sensible place would be no critical vulnerabilities that have got a fix in production. Well, there’s a policy right there, right. And it’s three lines. I, and it’s better than having these zeros.,

Alan Shimel 22:06
100% right. I run into this firsthand. People, they hear we need a security policy, they think I need the employee handbook, right, that comes from you know, this thick. You know, it could be one page or five bullet points. Anything that’s critical is worthy to stop production. Anything not critical does not stop production but has to get fixed within 30 days. That’s a policy.

Matt Jarvis 22:32
I mean, there’s plenty of great, templated stuff of usage of open source. You know, this stuff by the way.

Alan Shimel 22:39
I’d love to the OpenSSF have a library of that kind of thing. Yeah.

Stephen Hendrick 22:47
And actually, the good news is once you have policy then automation can follow up pretty quickly. That’s that’s, that’s the right path. You’re right.

Alan Shimel 22:54
Guys, we’ve got our next guest here in the wings. We could talk about this all day, I’m sure. I’d love to, but it wouldn’t be fair to them. Again, you can get this survey over on the Snky site, which is snky.io. Or on the OpenSSF site.

Alan Shimel 23:17
Steven, good work again. I love you surveys. Very good. We are going to take a quick break. We’re going to make up our next guest and we’ll be right back here with live in Austin.

introduction to DevSecOps for managers

In recent years, DevOps, which aligns incentives and the flow of work across the organization, has become the standard way of building software. By focusing on improving the flow of value, the software development lifecycle has become much more efficient and effective, leading to positive outcomes for everyone involved. However software development and IT operations aren’t the only teams involved in the software delivery process. With increasing cybersecurity threats, it has never been more important to unify cybersecurity and other stakeholders into an effective and united value stream aligned towards continuous delivery.

At the most basic level, there is nothing separating DevSecOps from the DevOps model. However, security, and a culture designed to put security at the forefront has often been an afterthought for many organizations. But in a modern world, as costs and concerns mount from increased security attacks, it must become more prominent. It is possible to provide continuous delivery, in a secure fashion. In fact, CD enhances the security profile. Getting there takes a dedication to people, culture, process, and lastly technology, breaking down silos and unifying multi-disciplinary skill sets. Organizations can optimize and align their value streams towards continuous improvement across the entire organization. 

To help educate and inform program managers and software leaders on secure and continuous software delivery, the Linux Foundation is releasing a new, free online training course, Introduction to DevSecOps for Managers (LFS180x) on the edX platform. Pre-enrollment is now open, though the course material will not be available to learners until July 20. The course focuses on providing managers and leaders with an introduction to the foundational knowledge required to lead digital organizations through their DevSecOps journey and transformation.

LFS180x starts off by discussing what DevSecOps is and why it is important. It then provides an overview of DevSecOps technologies and principles using a simple-to-follow “Tech like I’m 10” approach. Next, the course covers topics such as value stream management, platform as product, and engineering organization improvement, all driving towards defining Continuous Delivery and explaining why it is so foundational for any organization. The course also focuses on culture, metrics, cybersecurity, and agile contracting. Upon completion, participants will understand the fundamentals required in order to successfully transform any software development organization into a digital leader.

The course was developed by Dr. Rob Slaughter and Bryan Finster. Rob is an Air Force veteran and the CEO of Defense Unicorns, a company focused on secure air gap software delivery, he is the  former co-founder and Director of the Department of Defense’s DevSecOps platform team, Platform One, co-founder of the United States Space Force Space CAMP software factory, and current member of the Navy software factory Project Blue. Bryan is a software engineer and value stream architect with over 25 years experience as a software engineer  and leading development teams delivering highly available systems for large enterprises. He founded and led the Walmart DevOps Dojo which focused on a hands-on, immersive learning approach to helping teams solve the problem of “why can’t we safely deliver today’s changes to production today?” He is the co-author of “Modern Cybersecurity: Tales from the Near-Distant Future”, the author of the “5 Minute DevOps” blog, and one of the maintainers of MinimumCD.org. He is currently a value stream architect at Defense Unicorns at Platform One. 

Enroll today to start your journey to mastering DevSecOps practices on July 20!

securing your software supply chain with sigstore

Many software projects are not prepared to build securely by default, which is why the Linux Foundation and Open Source Security Foundation (OpenSSF) partnered with technology industry leaders to create Sigstore, a set of tools and a standard for signing, verifying and protecting software. Sigstore is one of several innovative technologies that have emerged to improve the integrity of the software supply chain, reducing the friction developers face in implementing security within their daily work.

To make it easier to use Sigstore’s toolkit to its full potential, OpenSSF and Linux Foundation Training & Certification are releasing a free online training course, Securing Your Software Supply Chain with Sigstore (LFS182x). This course is designed with end users of Sigstore tooling in mind: software developers, DevOps engineers, security engineers, software maintainers, and related roles. To make the best use of this course, you will need to be familiar with Linux terminals and using command line tools. You will also need to have intermediate knowledge of cloud computing and DevOps concepts, such as using and building containers and CI/CD systems like GitHub Actions, many of which can be learned through other free Linux Foundation Training & Certification courses.

Upon completing this course, participants will be able to inform their organization’s security strategy and build software more securely by default. The hope is this will help you address attacks and vulnerabilities that can emerge at any step of the software supply chain, from writing to packaging and distributing software to end users.

Enroll today and improve your organization’s software development cybersecurity best practices.

state of open source security report

The State of Open Source Security Highlights Many Organizations Lacking Strategies to Address Application Vulnerabilities Arising from Code Reuse

BOSTON — June 21, 2022 — Snyk, the leader in developer security, and The Linux Foundation, a global nonprofit organization enabling innovation through open source, today announced the results of their first joint research report, The State of Open Source Security.

The results detail the significant security risks resulting from the widespread use of open source software within modern application development as well as how many organizations are currently ill-prepared to effectively manage these risks. Specifically, the report found:

  • Over four out of every ten (41%) organizations don’t have high confidence in their open source software security;
  • The average application development project has 49 vulnerabilities and 80 direct dependencies (open source code called by a project); and,
  • The time it takes to fix vulnerabilities in open source projects has steadily increased, more than doubling from 49 days in 2018 to 110 days in 2021.

“Software developers today have their own supply chains – instead of assembling car parts,  they are assembling code by patching together existing open source components with their unique code. While this leads to increased productivity and innovation, it has also created significant security concerns,” said Matt Jarvis, Director, Developer Relations, Snyk. “This first-of-its-kind report found widespread evidence suggesting industry naivete about the state of open source security today. Together with The Linux Foundation, we plan to leverage these findings to further educate and equip the world’s developers, empowering them to continue building fast, while also staying secure.”

“While open source software undoubtedly makes developers more efficient and accelerates innovation, the way modern applications are assembled also makes them more challenging to secure,” said Brian Behlendorf, General Manager, Open Source Security Foundation (OpenSSF). “This research clearly shows the risk is real, and the industry must work even more closely together in order to move away from poor open source or software supply chain security practices.” (You can read the OpenSSF’s blog post about the report here)

Snyk and The Linux Foundation will be discussing the report’s full findings as well as recommended actions to improve the security of open source software development during a number of upcoming events:

41% of Organizations Don’t Have High Confidence in Open Source Software Security

Modern application development teams are leveraging code from all sorts of places. They reuse code from other applications they’ve built and search code repositories to find open source components that provide the functionality they need. The use of open source requires a new way of thinking about developer security that many organizations have not yet adopted.

Further consider:

  • Less than half (49%) of organizations have a security policy for OSS development or usage (and this number is a mere 27% for medium-to-large companies); and,
  • Three in ten (30%) organizations without an open source security policy openly recognize that no one on their team is currently directly addressing open source security.

Average Application Development Project: 49 Vulnerabilities Spanning 80 Direct Dependencies

When developers incorporate an open source component in their applications, they immediately become dependent on that component and are at risk if that component contains vulnerabilities. The report shows how real this risk is, with dozens of vulnerabilities discovered across many direct dependencies in each application evaluated.

This risk is also compounded by indirect, or transitive, dependencies, which are the dependencies of your dependencies. Many developers do not even know about these dependencies, making them even more challenging to track and secure.

That said, to some degree, survey respondents are aware of the security complexities created by open source in the software supply chain today:

  • Over one-quarter of survey respondents noted they are concerned about the security impact of their direct dependencies;
  • Only 18% of respondents said they are confident of the controls they have in place for their transitive dependencies; and,
  • Forty percent of all vulnerabilities were found in transitive dependencies.

Time to Fix: More Than Doubled from 49 Days in 2018 to 110 Days in 2021

As application development has increased in complexity, the security challenges faced by development teams have also become increasingly complex. While this makes development more efficient, the use of open source software adds to the remediation burden. The report found that fixing vulnerabilities in open source projects takes almost 20% longer (18.75%) than in proprietary projects.

About The Report

The State of Open Source Security is a partnership between Snyk and The Linux Foundation, with support from OpenSSF, the Cloud Native Security Foundation, the Continuous Delivery Foundation and the Eclipse Foundation. The report is based on a survey of over 550 respondents in the first quarter of 2022 as well as data from Snyk Open Source, which has scanned more than 1.3B open source projects.

About Snyk

Snyk is the leader in developer security. We empower the world’s developers to build secure applications and equip security teams to meet the demands of the digital world. Our developer-first approach ensures organizations can secure all of the critical components of their applications from code to cloud, leading to increased developer productivity, revenue growth, customer satisfaction, cost savings and an overall improved security posture. Snyk’s Developer Security Platform automatically integrates with a developer’s workflow and is purpose-built for security teams to collaborate with their development teams. Snyk is used by 1,500+ customers worldwide today, including industry leaders such as Asurion, Google, Intuit, MongoDB, New Relic, Revolut, and Salesforce.

About The Linux Foundation

The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and commercial adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

SBOMs and food labels

Software Bill of Materials (SBOMs) are like ingredient labels on food. They are critical to keep consumers safe and healthy, they are somewhat standardized, but it is a lot more exciting to grow or make the food rather than the label. 

What is an SBOM?

What is an SBOM? In short, it is a way to tell another party all of the software that is used in the stack that makes up an application. One benefit of having a SBOM is you know what is in there when a vulnerability comes up. You can easily determine if you are vulnerable and where. 

As modern software is built utilizing a base of software already written (no sense in recreating the wheel), it is important that all of the components don’t get lost in the shuffle. It isn’t readily apparent what a particular piece of software utilizes. So, if a vulnerability for Software A arises, you need to know, do I have that piece of software somewhere in my ecosystem, and, if so, where. Then you can remediate if you need to.

I can’t take credit for the food label analogy used in my introduction. I heard it from Allan Friedman, a Senior Advisor and Strategist at the U.S. Cybersecurity and Infrastructure Security Agency (CISA) and a key SBOM advocate, when he presented about SBOMs at the RSA Conference 2022 with Kate Stewart, the VP of Dependable Embedded Systems here at the Linux Foundation. Allan made the point that food labels only provide information. The consumer needs to read and understand them and take appropriate action. For instance, if they are allergic to peanuts, they can look at an ingredient label and determine if they can safely eat the food. 

SBOMs are similar – they tell a person what software is used as an “ingredient” so someone can determine if they need to take action if a vulnerability arises. It isn’t a silver bullet, but it is a vital tool. Without SBOMs no one can track what component “ingredients” are in their software applications.

SBOMs and the Software Supply Chain

Supply chains are impacting our lives more than just restricting availability of consumer goods. Software supply chains are immensely more complicated now as software is built with pre-existing components. This makes software better, more effective, more powerful, etc. But it also introduces risk as more and more parties touch a particular piece of software. Much like our world has become so interdependent, so has our software. 

Understanding what is in the supply chain for our software helps us effectively secure it. When a new risk emerges, we know what we need to do. 

SBOMs and Software Security

SBOMs are increasingly being recognized as an important pillar in any comprehensive software security plan. A global survey conducted in 2021 Q3 by the Linux Foundation found that 78% of organizations responding plan to use SBOMs in 2022. Additionally, the recently published Open Source Software Security Mobilization Plan recommends SBOMs be universal and the U.S. Executive Order on Improving the Nation’s Cybersecurity requires SBOMs be provided for software purchased by the U.S. government. And, as Allan points out in his talk, “We buy everything.” The E.O. actually lays out a nice summary of SBOMs and their benefits: 

The term “Software Bill of Materials” or “SBOM” means a formal record containing the details and supply chain relationships of various components used in building software.  Software developers and vendors often create products by assembling existing open source and commercial software components.  The SBOM enumerates these components in a product.  It is analogous to a list of ingredients on food packaging.  An SBOM is useful to those who develop or manufacture software, those who select or purchase software, and those who operate software.  Developers often use available open source and third-party software components to create a product; an SBOM allows the builder to make sure those components are up to date and to respond quickly to new vulnerabilities.  Buyers can use an SBOM to perform vulnerability or license analysis, both of which can be used to evaluate risk in a product.  Those who operate software can use SBOMs to quickly and easily determine whether they are at potential risk of a newly discovered vulnerability.   A widely used, machine-readable SBOM format allows for greater benefits through automation and tool integration.  The SBOMs gain greater value when collectively stored in a repository that can be easily queried by other applications and systems.  Understanding the supply chain of software, obtaining an SBOM, and using it to analyze known vulnerabilities are crucial in managing risk.

Allan and Kate spent time in their talk going into the current state of SBOMs, challenges, benefits, tools available for creating and sharing SBOMs, what is a minimum SBOM, standards being developed, making them fully automated, and more. Look for some future LF Blog posts digging into these. 

But there are things you can do now. 

What can you and your organization do now?

Allan and Kate laid out several things you and your organization can do, starting now. Starting within your organization: 

Next week: Understand origins of software your organization is using

  • Commercial: can you ask for an SBOM?
  • Open source: do you have an SBOM for the binary or sources you’re importing? 

Three months: Understand what SBOMs your customers will require

  • Expectations: which standards, dependency depth, licensing info?

Six months: Prototype and deploy

  • Implement SBOM through using an OSS tool and/or starting a conversation with vendor

And participate in ongoing discussions to determine best practices for the ecosystem and contribute to open source project any code developed to support SBOMs. 

But there are also steps you can take as an individual:

Next week: Start playing with an open source SBOM tool and apply it to a repo

Three months: Have an SBOM strategy that explicitly identifies tooling needs

Six months

  • Begin SBOM implementation through using an OSS tool or starting a conversation with vendor
  • Participate in a plugfest and try to consume another’s SBOM

And make sure to share any open source and commercial tools you find helpful and work with the tools to help harden them, test and report bugs, and push them to scale.

How can you shape the future of SBOMs?

First, I want to highlight some upcoming opportunities they shared to help shape the future of SBOMs. CISA is running public Tooling & Implementation work stream discussions in July 2022. They are the same, but occur at different times to help accommodate more time zones: 

  • July 13, 2022 – 3:00-4:30 PM ET
  • July 21, 2022 – 9:30-11:00 AM ET 

If you want to participate, please email SBOM@cisa.dhs.gov

Additionally, there will be “plugfests” to be announced soon, and they suggested organizations already adopting SBOMs publish case studies and reference tooling workflows to help others. 

Conclusion

SBOMs are here to stay. If you aren’t already, get on the train now. It is pulling out of the station, but you still have an opportunity to help shape where it is going and how well the journey goes. 

Allan’s and Kate’s slides are available here. If you registered to attend the RSA Conference, you can now watch their full presentation on demand here.



The Software Package Data ExchangeⓇ (SPDXⓇ)

The Linux Foundation hosts SPDX, which is an open standard for communicating software bill of material information, including components, licenses, copyrights, and security references. SPDX reduces redundant work by providing a common format for companies and communities to share important data, thereby streamlining and improving compliance. The SPDX specification is an international open standard (ISO/IEC 5962:2021). Learn more at spdx.dev

SPDX

Software Metadata Standards Wrap Up Bigger Connections

This article originally appeared on Linux.com. The author, Cameron Laird, is vice president of Phaseit, Inc. where he implements software projects and publishes articles about the results. A long-time developer, manager, and author, he’s most recently concentrated on architectural challenges of “continuous everything”: continuous integration, continuous testing, and so on.

You’re in the news. But not with the headline you want.

You’re not getting attention because of your choice of text editor or the number of spaces you use to indent code blocks. However motivating those preferences are for you and me, the non-technical world sees them as private choices. You find your code in the headlines for a different and unpleasant reason: open source dependency management.

We have dependencies, of course, because we know not to “reinvent the wheel”; instead, we software experts re-use the implementations others have created. However, when done poorly, dependency management introduces more risk and degrades the quality of your application. For example, failure to comply with license requirements might be the problem.  Even worse: the absence of a license tied to a component you embedded in your application. In both cases, there are potential legal implications.

Still more traumatic is a media headline announcing that a vulnerability just breached your organization in one of those dependencies. Projects frequently re-use software components to simplify or accelerate development; but sometimes, it can have detrimental results by introducing said vulnerabilities.

That’s not all:  suppose you are experienced and thoughtful enough to recognize this hazard and commit to good dependency management.  It turns out that’s a harder problem than might first appear, and certainly not the kind of thing that can be slipped into a project on its last days, without significant time or other costs.

Building A Standard For Software Bill Of Materials

How, for instance, does an industrial oven manufacturer communicate that one of its products depends on a particular library with a known vulnerability?  How does it say that it does not have such a dependency?  One of the difficulties comes from mixing open and closed information sets. What happens in a scenario where an automotive chip uses an open source sorting algorithm, but the auto manufacturer wants to keep the use of that algorithm proprietary?

Without a better alternative, any discussion about the algorithm has to occur under cover of a non-disclosure agreement (NDA), often one written specifically for the business and technical situation.  Where developers investigating a particular piece of software might be accustomed to connecting to GitHub and inspecting the source in question in a few seconds, even the simplest proprietary questions sometimes take months of legal, security, and compliance negotiation to begin to examine. “Manual” inspection, in any case, is unscalable.  The average application contains 200 OSS components, and each component might manually take three hours to inspect.  Does your project have a better use for 600 hours of effort?  Open source truly begins to pay off when it’s inspected not just by expert engineers but by automatic tools.

Recognize, moreover, that transitive dependencies make dependency management a harder problem than first appears.  Many of the most notorious breaches occurred not because anything was wrong with the source of a product or even the source of the libraries on which it depends; the vulnerability only turned up in a library used by those other libraries.  Over and over again, CEOs who’ve asked, “does $SOME_PROBLEM affect us?” have received the answer, “we don’t know yet: we’re not sure where it shows up in our systems.” We need transparency about dependencies and enough intelligence and standardization around hierarchical relationships to “trace the whole tree.” Organizations must track dependencies through to the operating system run-time and sometimes down to “the silicon,” that is, the microprocessor on which the software runs.

It’s a hard problem but also a solvable one.  Part of any solution is a well-defined software bill of materials (SBOM or sometimes SBoM). That’s where Kate Stewart’s career began to track this story.  Stewart currently serves the Linux Foundation as a vice president of Dependable Embedded Systems.  In previous assignments with such employers as Motorola, Freescale Semiconductor, Canonical, and Linaro, she frequently faced challenges that mixed technical and legal aspects.  As she explained her long-time focus in a recent interview, “if open source components are going to be in safety-critical places … [we need] to be able to trust open source in those spaces …” Good SBOM practices are simply necessary for the level of trust we want to have not just in industrial ovens, but airplanes, medical devices, home security systems, and much more.  An SBOM organizes such metadata about a software artifact as its identity, verification checks it hasn’t been tampered with, copyright, license, where to look up known security vulnerabilities, dependencies to check, and so on. Think of an SBOM as an ingredients list for your software.  It makes those ingredients visible, trackable, and traceable.  It lets you know if you have used the highest quality and least risky open source components to build your software.

Enter SPDX

Stewart and other technologists eventually began to team with specialists in intellectual property, product managers, and others. They developed such concepts in the early years of this millennium as SBOM, the Software Package Data Exchange (SPDX), and the OpenChain Specification.  She co-founded SPDX in 2009 to pursue “[a]n open standard for communicating software bill of materials information ….” Among other features and benefits, these frameworks provide standard and scalable ways to discuss dependencies.

Instead of each vendor having to certify that each of its releases has been verified for security and license compliance of each of eight hundred JavaScript libraries, for example, many of the most time-consuming aspects of compliance can be automated.  When a new vulnerability is identified in an implementation of a networking protocol, automated methods can largely be applied to determine which products embed known vulnerable libraries, even while we developers remain largely unaware of the details of each component and dependency they use.  For Stewart, standards-based transparency and best practices are prerequisites for the security of safety-critical communities she helps serve.  As Stewart observes, “you can’t really be safe unless you know what you’re running.”

Daily Headlines

Does that sound mundane?  The reality’s far different:  SBOM and related technologies actually play roles in events on the world stage.  For example, on the 12th of May, 2021, US President Biden issued Executive Order 14028 on Cybersecurity Improvement; SBOMs play a prominent role there.  The Open Source Initiative just named Stefano Maffulli its first Executive Director precisely because of the need for mature open source licensing practices.  Dr. Gail Murphy argued in a recent interview that it’s time to recognize that open source software is a “triumph of information-hiding [and] modularity …” in enabling the remarkable software supply chains on which we depend.  Emerging information on breaches including SolarWindsRapid7Energetic Bear, and especially the latest on Juniper’s Dual-EC affair shows how disastrous it becomes when we get those supply chains wrong.  The most prominent breaches in computing history have been tied to component vulnerabilities that seemed peripheral until break-ins demonstrated their centrality.

Drone strikes?  Vaccine efficacy?  Voter fraud?  International commerce?  Nuclear proliferation?  Questions about software and data reliability and fidelity are central to all these subjects, not mere technical tangents.

That’s why SPDX’s management of hierarchical relationships is so crucial.

ISO/IEC 5926:2021 Introduces SBOM Standard

SPDX went live as an official international standard at the end of August.  With that milestone, standardization lowers many of the hurdles to the successful completion of an SBOM project.  Implementation becomes more consistent. “Bookkeeping” about external parts becomes largely a responsibility of the standard.  Software engineers focus more on the details specific to an application.  Then, as those external parts–the ingredients of an SBOM recipe–age and security vulnerabilities are discovered in them, developers can reliably track those components to the applications where they were used and update components to newer, hardened versions. What does that mean for you?  In your own work, the faster you identify and update vulnerable components, the less likely the chance you will have of becoming the next breach headline following an attack.

SPDX’s standardization fits in the frameworks of the International Organization for Standardization (ISO) the International Electrotechnical Commission (IEC).  ISO is a post-war transnational creation that originally focused on bolt sizes, temperature measurements, and medical supplies.  ISO tracks human affairs, of course, and its attention in recent years has shifted from materials to business processes and, in this millennium, to software.  IEC is a prior generation’s initiative to pursue the same kinds of standardization and cooperation, specifically in the realm of electrical machinery; the IEC and ISO often collaborate.

In bald terms, ISO and IEC matter to you as a programmer because governments trust them.  The new standard is sure to make its way rapidly into procurement specifications, especially for government purchases.  Suppliers become accustomed to compliance with such standards and apply them in their practices more generally.  The earlier ISO 9000 collection of standards has already greatly influenced software development.

Important Though Abstract

The impact and scope of ISO:IEC 5926:2021 is a challenge to understand, let alone explain.  On the one hand, millions of working programmers worldwide go about their daily chores with little thought of SPDX or even SBOMs.  While we all know we depend on packages, we largely leave it to Maven or npm, or RubyGems, etc., to handle the details for us.  Standardization of SPDX looks like a couple of layers of abstraction, even more remote from the priorities of the current sprint or customer emergency on our desks right now.

And it’s true:  SPDX is abstract, and its technical details look dry to some programmers, the opposite of the “sexy” story many start-ups aspire to.

Without this infrastructure, though, the development of many large, complex, or mission-critical projects would grind to a halt from the friction of communication about proprietary dependencies on open source artifacts.  Think of it on a weight basis:  as the Linux Foundation’s own press release underlines, “… between eighty and ninety percent of a modern application is assembled from open source software components.” SPDX is immensely important at the same time as it’s uninteresting to all but the most specialized practitioners.

Look to history for examples of how momentous this kind of standardization is.  The US’s Progressive movement at the beginning of the twentieth century is instructive.  While often taught in ideological terms, many of its greatest achievements had to do with mundane, household matters:  does a milk bottle actually contain milk?  Can standard doses of medicines be trusted?  Is a “pound” in a butcher’s shop a full sixteen ounces?  Standards in these areas resulted in more convenience and transformed commerce to enable new market arrangements and achievements. That’s the prospect for SPDX:  more transparent and effective management of software dependencies and interactions will have far larger consequences than are first apparent.  Notice, for instance, that while the standard examples of its use have to do with open-source software, the standard itself and the tools that support it can also be applied to proprietary software and other intellectual property.  SPDX doesn’t solve all problems of communicating about dependencies; it goes a long way, though, to clarify the boundaries between technical and legal aspects.

Long Lead Time

The significance and need for secure software supply chains haven’t made SPDX’s adoption easy, though.  Stewart reports that individual companies drag their feet: “why should we do something before we have to?” these profit-oriented companies reasonably wonder.  Even in the best of circumstances, when an industry has largely achieved a technical consensus, “From first proposal to final publication, developing a standard usually takes about 3 years.

Stewart herself cites this year’s Executive Order as crucial: “the one thing that made a difference” in pushing forward adoption of SPDX in 2021 was the emerging SBOM requirements that followed EO14028.  Much of her own emphasis and achievement of late has been to get decision-makers to face the reality of how crucial their dependence on open source is. No longer can they restrict focus to the 10% of a proprietary product because supply chain attacks have taught us that the 90% they re-use from the software community at large needs to be exposed and managed.

Publication of a standard mirrors application development in having so many dependencies “under the covers.” It’s not just Stewart who worked on this for more than a decade, but, as I’ll sketch in follow-ups through the next month, a whole team of organizations and individuals who each supplied a crucial requirement for completion of ISO/IEC 5926:2021.  When you or I think of great software achievements, our memories probably go to particular winning prototypes turned out over a weekend. Standards work isn’t like that.  The milestones don’t come at the rapid pace we relish. Successful standards hold out the promise, though, of impacting tens of thousands of applications at a time. That’s a multiplier and scalability that deserves more attention and understanding.

SBOMs For Everything

And that’s why ISO/IEC 5926:2021 is good news for us.  We still have licensing and security issues to track down. We still need to attend meetings on governance policies. Management of proprietary details remains delicate.  Every project and product needs its own SBOM, and vulnerabilities will continue to crop up inconveniently. With the acceptance of ISO/IEC 5926:2021, though, there’s enough standardization to implement continuous integration/continuous deployment (CI/CD) pipelines usefully. We can exchange dependency information with third parties reliably. SPDX provides a language for describing dependency management chores. SPDX gives answers that are good enough to focus most of our attention on delivering great new functionality.

The best practices of application development applied by developers as a learned methodology can be something more than an exercise in walking a tightrope of intellectual property restrictions. Enterprise-class proposal requests become more engineering than lawyering.  You have a better shot at being in the news for your positive achievements rather than the security calamities into which you’ve stumbled.

Check in over the next several weeks to learn more about what SPDX means to your own programming, how SPDX is a model for other large-scale collaborations the Linux Foundation enables, and how teamwork is possible across profit-making boundaries.  In the meantime, celebrate ISO/IEC 5926:2021 as one more problem that each project does not have to solve for itself.