Hendrick and Jarvis Talk Software Security - Linux Foundation

Alan Shimel 0:00
This is Alan Shimel witih Tech Strong TV. We’re back here live in Austin streaming out at you from the Open Source Summit. We’re having a great time. This is our third day of coverage here, though technically, it’s only day two of the event. It’s a long story, but we’ll talk about it later. Let me introduce you to our next two guests. This is a conversation I was really looking forward to. To my left here is a gentleman who’s been on fixture on TV a few times with us and talked in person, great person. He’s the VP of Research for the Linux Foundation, Steven Hendrick. Steven, welcome.

Stephen Hendrick 0:42
Thanks, Alan.

Alan Shimel 0:43
And joining Stephen and I from our friends at Snyk. Matt, Javis. And Matt, if I’m not mistaken, your director of developer relations. Welcome.

Matt Jarvis 0:55
Thank you.

Alan Shimel 0:55
So Steven, and Matt presented, was it yesterday. Yeah, yesterday, on a new survey that you guys recently announced and revealed and report. Why don’t you if you don’t mind share with everybody

Stephen Hendrick 1:10
Sure, I’d love to. OpenSSF is a very large project inside of the Linux Foundation. Brian Behlendorf, yeah. And so at his request, we went out and did a survey into sort of what’s happening in the open source space as far as secure software development. So we put together a survey in March, we fielded that it in April, we wrote it up in May, and had it produced in June. And so it’s being released here at the event. I think that happened yesterday morning. We did it in partnership with Snyk. So that’s why we’ve been working together with the messaging on all this.

Stephen Hendrick 1:55
And it’s it was not a surprise from the standpoint of what the results were. But it wasn’t, I was a little disappointed in kind of where we are at this point, from the standpoint of the uptake of attention to security when it comes to open source. So anyway, so we’ve got information that talk a little bit about, you know, where we are, you know, help, sort of, to understand the context of the problem. And then they have information about what people are doing about it. And it’s, that’s more exciting in many respects, because good things are happening.

Alan Shimel 2:33
I agree. So first of all, look, I think we’re always disappointed when we do the surveys that we find out, you know, beyond the lip service that gets paid to security, what actually is going on under the covers, and we’re always wishing for and hoping for more. That being said, I don’t want to be pessimistic I am of the glass half full opinion that we are doing better and more security now than we probably ever have done. Yeah. Yeah, I agree. That being said, Before we dive into it, I just wanted to really just quickly, so OpenSSF is the website? Yeah. And I’m going to assume that the report is there for anyone who wants to download it. That’s right. Let’s take let’s say that up front for people at home following along, or whether it’s live while you’re watching this.

Stephen Hendrick 3:30
It’s on the Snyk site. It’s so on the Linux Foundation site and it’s on OpenSSF. So yeah, it’s everywhere.

Alan Shimel 3:37
I think we might have covered it via Snyk over on Security Boulevard.

Matt Jarvis 3:41
I think I did some I did some press interviews. Before flying out here. So, yeah, we may have.

Alan Shimel 3:46
It may very well be on our Securityf Boulevard site. But nevertheless, it’s out there for people. Yeah. Let’s dive in now, though, what was some of the findings? Steven?

Stephen Hendrick 3:55
Sure. Well, let’s see what we’ll start with this, this whole issue of do organizations have an open source security policy? And what we found was 49% said they had one, that’s good. That’s good. 34% did not. And 17% said they don’t know. Everybody uses open source – 98% of organizations use it so. So they don’t even know they don’t know if they have won or not. So if you take put aside the don’t knows at this point, you’ve got about a 60/40 split between use that don’t have have a policy and don’t have a policy.

Stephen Hendrick 4:37
I mean, if you look at little more deeply into that, what you find is that small companies are more likely to not have a policy and that’s not surprising, they have resource constrained so it’s harder for them to have CISOs and OSPOs and policies be it for either just software development or open source software development so I can understand challenges there. So, but the idea of when you, even if you look at company size, we still ended up with about 30% of large in very large organizations that don’t have a policy for open source software development.

Alan Shimel 5:16
So a couple of thoughts. First of all, I empathize with small SMB businesses. We are an SMB business, but in today’s day and age, and maybe it’s when you’re a hammer, everything looks like a nail, but in today’s day and age, how do you not have security policies?

Matt Jarvis 5:40
Yeah, I mean, I think that there’s, there’s a couple of different things at play there. I mean, you know, addressing open source security, you know, is, is more complex than than it seems, because it’s not just about the code itself, you’ve kind of got to understand how open sources is created, how projects are governed, because governance can have a big play into, you know, whether we’re looking at some of those recent things around the sort of protestsware movement, where we’ve seen maintainers kind of go rogue, you know, and this comes down to single maintainer governance projects. And you need to take those things like governance into account if you’re going to base your business on something.

Alan Shimel 6:24
Right, but you just said, and that’s a completely loaded question. I would bet, if I was a betting man, right, that at the large enterprise level, you’re 100%, correct. At the SMB level, if you ask most of these people a threshold question of where is your open source software, it’s 10 o’clock, where’s your open source software? And a lot of them don’t know, because they’re SAS-ops companies, right? They don’t they don’t have a server closet, their cloud installation – they run it on SAS. And so the beautiful part about SAS is, one of the nice things about it, is you don’t know what’s behind the curtain, you just know, you log in on the website, and you’ve got all your information there that you need. Are they using an open source database? Are they using, you know, what are they using behind the curtain? A lot of smaller companies don’t know. And as part of their due diligence, they don’t dig that deep. So I could, again, I can empathize with the larger ones, the larger enterprises, though, that’s a problem that is that I think,

Matt Jarvis 7:39
You know, in a lot of those larger enterprises, you’re you’ve got that kind of ingrained culture over a long time in terms of security and about how you consume software. And you know, the hardest problem in security isn’t really about technology at all right? It’s always about people and culture. And I think, you know, probably in a lot of larger organizations, you’ve got kind of, you know, that sort of friction of, well, we’ve always done it like that.

Stephen Hendrick 8:07
Well, you also have a lot of change going on from standpoint of how software is being developed. And I think that’s part of the problem as well, which is that, you know, change is always hard for people. And especially given the rapid evolution of tools and standards, in essence around how we should do security for software. Everything’s changing so quickly, it’s I think, it’s probably hard for people to keep up.

Matt Jarvis 8:33
Because we’ve got these two kind of things happen in almost a perfect storm. At the same time, we’ve got this massive rise in supply chain attacks on open source, because, you know, it’s a victim of its own success. And attackers have realized it’s a lot easier to get into the supply chain than it is to find zero days in end user applications. So you’ve got that going on, where all of a sudden, folks are going, well, everything we do is based on open source, like, what do I do about security? And then, as Steve pointed out, you’ve got this, this ongoing, massive transformation of how we develop software, you know, this superfast high velocity,

Alan Shimel 9:10
I blame DevOps?

Matt Jarvis 9:15
Unless you do, and unless you can transform, you know, someone’s going to eat your lunch, right? Because there’s some hungry competitor behind you who’s disruptive and who does have a superfast software delivery pipeline. They can deliver new features, they know how to analyze the data. And so for a lot of big organizations, you’ve got these two big problems happening right at the same time, because that change in software development requires a completely different approach to security. You know, this space that it’s the thing that sneaks talk about all the time about developers?

Alan Shimel 9:45
I mean, you look at let’s say, the Phoenix Project by Gene Kim, right. And that’s based on a book called The Goal. Yeah, right. And the thing about, so The Goal is about manufacturing, but really the principle behind The Goal, and I think Gene tried to capture that in the Phoenix Project, is that, look, as soon as we kind of erase one bottleneck, we see that next bottleneck right behind it. Don’t think that once you get rid of that bottleneck its smooth sailing, it’s not. We have massively, revolutionarily speeded up the pace of software development. We did it in large part by creating this software factory pipeline, CI/CD, DevOps kind of things. The enabler of that was having this massive library of open source that we can assemble into a very high quality software.

Alan Shimel 10:43
Man, we blew through that roadblock at 150 miles an hour. The wall we hit right after is, wait a second, now, that’s become a huge security problem. Right? So for companies that are developing their own code, this is a major thing? Knowing that though, and still telling me that 30% of the companies don’t have a policy around it, scary. Yeah, it is.

Stephen Hendrick 11:09
Well, let’s we should we should talk about what people are doing about trying to deal with this. Right.

Alan Shimel 11:14
Here’s the good news.

Stephen Hendrick 11:14
So we asked a question, which was, okay, so how do you intend to improve on the situation? What do you what are you doing, and we had quite a long list of responses. Top of the list was, organizations were looking for more intelligent tools from a threat where repetitive security focus. So we’re talking SCA, SAS DAST, IAC, you know, all the usual suspects, and looking really, to those tools to be able to help them improve their security posture. So that was top of the list. That was 59%.

Stephen Hendrick 11:52
And then right behind that at 52% was a strong desire to understand and essentially codified best practices for how to do secure software development. That was really encouraging, because we know all about best practices. Yep. Know exactly what they all are. In fact, David Wheeler, at LF.

Stephen Hendrick 12:14
We had David A. Wheeler. We interviewed David yesterday and we have a follow-up as well.

Stephen Hendrick 12:32
He and I had lunch yesterday, and we were talking about this, because I said, you know, how many best practices do you have? So we know counted them all up – he’s got like, 150, 160. So that’s kind of daunting. And he said, like the last 25 to get to the highest level can take, in some cases, years to master. So this is, this is, despite understanding what these best practices are, it’s still very challenging to wrap your head around what is necessary to be successful there.

Matt Jarvis 13:00
It’ll be good. And partly because, you know, as we were just talking about that, that culture change is such a big part of how you make that transition from, you know, you kind of old school, security as gatekeeper kind of function, to this thing, where we put it to the developers, because the developers are the ones who, you know, you fix it at the developer eyeball before it’s got anywhere near, you know, production. That is the cheapest.

Alan Shimel 13:28
Say 10 to 100x cheaper to do there.

Matt Jarvis 13:30
I mean, we look at the other interesting thing here that’s slightly tangential to this, but is like how many developers there are in the world, right, and how many we anticipate there being you know, there’s something like, I think the anticipation is something like 30 million developers in the world, and there’s only, like, a tiny proportion of security folk.

Alan Shimel 13:51
So I go by GitHub accounts. Right? There’s about 70 plus million GitHub accounts right now. So let’s assume it’s not one-to-one. But I think it’s safe to say this 40 to 45 million developers, probably growing at somewhere in the area of 10% a year.

Matt Jarvis 14:08
And security professionals aren’t aren’t growing at that rate.

Alan Shimel 14:13
So security professionals are growing because we’re starting to see, look, when I came up, you didn’t have a cybersecurity major in college. We’re seeing schools churn out cybersecurity majors. Are they security professionals? I’ll leave it to you. But there are people coming out here who want to work in security, but not anywhere near I mean, you’re talking here in here.

Alan Shimel 14:39
Here’s an interesting thing, though. And I think it’s what’s turning up the heat on all of this, is that this is getting major focus from the White House, from the federal government. The whole world is saying hey, this is a problem. This is a big problem.

Stephen Hendrick 14:58
Well, you know, you got to do something you You know, I did a report last year survey on SBOMs. Yep. And I gotta tell you that factors right into this No, of course, because, you know, we did some stats in this survey on dependencies, you know, both direct and transitive, and found, really, sort of low levels of strong, strong security around, you know, organizations understanding the security posture of all these different dependencies and dependencies of dependencies. Really low numbers there.

Stephen Hendrick 15:34
SBOMs would go so far in helping sort all that out, because, you know, SBOMs, they’re going to give you knowledge about the metadata, it’s gonna give you usability, so you know that you’re licensed to use the stuff, and it’s going to know if it was good, if you trust that not only what you’re looking at for metadata is not falsified, but also understanding quite clearly, you know, what’s been fixed, what hasn’t been fixed from a vulnerability standpoint.

Alan Shimel 15:59
So I’ll tell you over the last two days, we’ve, we’ve done a lot of interviews, but no shortage of people talking about SBOMs and SBOM solutions. I think we’re going to see, just like everything else in technology, we’re gonna see sort of a camp Cambrian explosion of SBOM solutions out there, and then the market will figure out which ones make sense, which ones don’t. My fear is that we, we think SBOMs are a magic bullet for supply chain security, because we have a tendency of doing that in security.,

Matt Jarvis 16:33
Ultimately I think the real challenge here is going to be the chain of trust part of that, right? Because what’s an SBOM at the end of the day, it’s a text file with some stuff.

Alan Shimel 16:43
Oh, no, but they’re, you know, they’re, they’re building some elaborate text files in there.

Stephen Hendrick 16:49
It’s a lot of good metadata. Yeah, but one more point I want to touch on, though, is that the number three issue from the standpoint of doing improvements to your software security posture was more automation. So IAC tools ended up ranking very highly from the standpoint of helping you address that particular need, and just for our audience, IAC (infrastructure as code), right. Okay. So that one actually surprised me, because this whole idea of, you know, developers/manual activities, not only that is a great way to invite in problems. And so more automation, ultimately, is better.

Matt Jarvis 17:33
We did some some work last year as part of our cloud native application security report. And what was really interesting there was, you know, we kind of use high levels of development automation, i.e.automated CI/CD pipelines and all that stuff as a as a proxy for how far along your cloud native journey you are. I think it’s a pretty reasonable proxy to take. And in organizations with those high levels of of deployment automation, for a start, we see much higher levels of adoption of security tooling, because automation gives you lots of places where you can hook in other automation. But, most importantly, we see massive reduction in the time to fix the vulnerabilities. Because through directly through a direct correlation to okay,

Alan Shimel 18:25
I’ve been in security a long time. We had a vulnerability solution company I founded back in 2005. Back then, there was a company called Hercules – Citadel was the company Hercules was the product, right? They were doing, you know, they were pushing automated remediation. There’s several companies today that have automated remediation. For whatever reason, up until now, organizations have been hesitant to adopt automated remediation, because they’re afraid it’s going to break something else if it’s in a totally automated situation. Now, doing this further left in the in the development pipeline, if it’s broken, supposedly, that should come up in testing.

Matt Jarvis 19:16
I mean, this is again what we see when when companies adopt Snky, is like, you know, the the automated remediation part of in terms of automated fix PRs, you know, is, is probably not where people start, but very quickly, they go out.

Alan Shimel 19:33
Yeah, look, this is a no brainer. Yeah, absolutely, because it goes back to what I said before blame DevOps, right. If we are going to automate the CI/CD pipeline, we’re going to automate building software, the answer cannot be that we’re going to manually do security. It just doesn’t work. It’s a disconnect.

Matt Jarvis 19:54
Yeah, I mean, and it’s an anti-pattern in terms of velocity, right. I mean, velocity is the key differentiator for whether, sort of, visitors in the cloud era are going to survive. Absolutely, because if you don’t have velocity, you are probably don.e.

Alan Shimel 20:08
No, but a lesson we learned in security, or we should have learned over the last 25 years, is if we are going to drag our heels and dig our heels in and say no, no, no, no. You know what? The train leaves the station without you. Yeah. So easy to get on board and figure out yes, we can. And here’s how we’ll get out of the way. Right? Lead, follow, get out of the way. Security cannot be the drag on this because velocity is too important.

Matt Jarvis 20:36
And where we see folks who’ve successfully made this transition to developer-first, you see this sort of this change in security teams from kind of being gatekeepers to being enablers.

Alan Shimel 20:41
DevSecOps right there. Yeah, you just the just the the heart of it. That’s it. Steven, anything else?

Stephen Hendrick 21:02
So what’s the answer to this issue of not having a security policy? I mean, is it, Do you need to start with the CISO? Did you start with an OSPO? Do you need, or at least part time roles and people in organizations, you know, in those functions if you are small? I’m not sure what the answer is, but I mean, we need one.

Matt Jarvis 21:27
I think when people think about policies, they think, Oh, this needs to be like 100 page document of some kind, you know, this is it becomes overwhelming, but really a policy can be a one liner.. I mean, we, we have this conversation a lot when people start to adopt security scanning, right, they’ve done no security scanning before, and they scan this software, and they go, Oh, my God, I’ve got like 500 vulnerabilities. What do I do? But you’ve got to just pick a starting point. Right. And I mean, usually, you know, a sensible place would be no critical vulnerabilities that have got a fix in production. Well, there’s a policy right there, right. And it’s three lines. I, and it’s better than having these zeros.,

Alan Shimel 22:06
100% right. I run into this firsthand. People, they hear we need a security policy, they think I need the employee handbook, right, that comes from you know, this thick. You know, it could be one page or five bullet points. Anything that’s critical is worthy to stop production. Anything not critical does not stop production but has to get fixed within 30 days. That’s a policy.

Matt Jarvis 22:32
I mean, there’s plenty of great, templated stuff of usage of open source. You know, this stuff by the way.

Alan Shimel 22:39
I’d love to the OpenSSF have a library of that kind of thing. Yeah.

Stephen Hendrick 22:47
And actually, the good news is once you have policy then automation can follow up pretty quickly. That’s that’s, that’s the right path. You’re right.

Alan Shimel 22:54
Guys, we’ve got our next guest here in the wings. We could talk about this all day, I’m sure. I’d love to, but it wouldn’t be fair to them. Again, you can get this survey over on the Snky site, which is snky.io. Or on the OpenSSF site.

Alan Shimel 23:17
Steven, good work again. I love you surveys. Very good. We are going to take a quick break. We’re going to make up our next guest and we’ll be right back here with live in Austin.