Measuring OSPO Value

Open Source Program Offices (OSPOs) play important roles within organizations, but that role isn’t always appreciated or understood within the executive team or by other stakeholders. In the current financial climate, some OSPOs have been the targets of cutbacks and layoffs, so this is a particularly important time for OSPOs to be clear about the value that they provide to their organization. Leaders within every organization are responsible for making sure that their organization focuses on the activities that have the biggest impact on helping that organization achieve their goals. As a result, OSPOs need to be able to demonstrate that the value of their work can have a larger impact on the organization than the other initiatives that are also competing for resources. As the CHAOSS OSPO Metrics Working Group co-chair and CHAOSS board member, the topic of measuring OSPO value is one that I have cared deeply about for years, and I recently gave a talk on this topic at the Open Source Summit North America in May. This is why I was so excited to read Ibrahim Haddad’s latest report, Measuring OSPO Value: A Framework for ROI, Resilience, Risk Foresight, and Strategic Influence, and share a few highlights from the report in this blog post.

It’s easy to say that OSPOs should be better at measuring value, but it’s not quite that straightforward. OSPO value has always been difficult to measure because much of the work is preventative, the effects are distributed throughout the organization, the impact is spread across multiple time horizons, and the work is cross-functional. However, it’s become increasingly urgent with the ubiquity of open source software impacting revenue-critical systems, security and supply chain expectations increasing, regulations having a bigger impact on open source, and AI-generated code adding complexity.

There is no one way to measure OSPO value, so the report uses a framework with 4 interrelated dimensions that can help OSPOs reason about value from multiple perspectives that can be applied through the lens of their unique organizational goals.

ROI and cost avoidance. In my experience, when executive leadership and finance are questioning an OSPO’s value, they usually start by asking questions about ROI, but this narrow framing doesn’t tend to be particularly useful in my opinion. OSPOs don’t typically generate direct revenue, but they can have an impact on cost avoidance, including reduced duplication, improved efficiency, and lower maintenance costs that can be used for financial justification.

Resilience. Engineering and security leadership on the other hand want to better understand how the OSPO is helping the organization avoid disruption through preparation related to visibility and management of dependencies, SBOM coverage, licensing or provenance concerns, and readiness around engineering decisions. When this is done well, everything proceeds smoothly and crises are avoided, but this is why measuring resilience proactively is an important part of measuring OSPO value.

Risk foresight. While resilience is about preparedness, risk foresight is about detecting potential issues early enough to mitigate the impact and avoid incidents. This includes detecting potential license issues, governance problems, supply chain concerns, security vulnerabilities, and regulatory / policy changes. This value can be measured and communicated by documenting near-misses and creating a narrative around how the OSPO took action to prevent the issues. The CHAOSS Assessing Viability Practitioner Guide provides additional insight into this dimension.

Strategic influence. This dimension measures an OSPO’s long-term value, including how the OSPO strategically invests in the open source ecosystem with presence, engagement and influence in technologies, standards, and organizations that are critical for the organization now and in the future. We also covered some of this in the CHAOSS Demonstrating Organizational Value Practitioner Guide.

Taken from Haddad's report, "Figure 1: OSPO value framework"

The report also highlighted a few principles for building a measurement system across these 4 dimensions, including measuring outcomes (not activities), focusing on a smaller number of indicators, using both quantitative and narrative approaches, explicitly documenting assumptions, distinguishing between enabled value and owned value, avoiding metrics that punish disclosure, designing for maturity, stating framework limits, and having metric continuity. Ultimately, all of this work to measure and demonstrate value needs to be communicated to executives and other stakeholders in a way that they can understand the importance of the OSPO. Ibrahim’s report has more details on tailoring communications to specific audiences, using scorecards, evolving your approach over time, and a practical roadmap for implementation.

If you work in an OSPO or do open source work within an organization, now is the perfect time to rethink how you measure and demonstrate the value of this work, and this report is a great way to get started or get you thinking about how you can improve your existing approach to measuring OSPO value.

Dr. Dawn Foster is an OSS strategy consultant. She is also on the board of CHAOSS, OpenUK, and the Software Stewardship Lab. She was previously a co-chair of the CNCF Contributor Strategy TAG. She has 20+ years of experience at companies like VMware and Intel with expertise in strategy, governance, and metrics. She has spoken at over 100 industry events and has a BS in computer science, an MBA, and a PhD. In her spare time she enjoys reading science fiction, running, 3D printing, and traveling.