When it comes to measuring your open source program’s success, it’s tempting to focus on the quantitative metrics for your projects: total number of contributors, lines of code, number of projects, etc. We’ll discuss what to measure to assess your project health in the next section. But first, there are many other important ways to measure your program’s success than strictly by these numbers.
Kubernetes is one of the highest velocity open source projects on GitHub, attracting more than 80,000 commits from 2,760 developers at 1,181 companies over the last three years. But from the start, the project has managed its success in terms of whether its users were excited about the technology and using it, not by “some list of open source metrics,” Beda said.
Below are some of the common goals behind an open source program office, and the top ways that program managers measure against these goals to track the overall progress of the program. Some of these goals can’t be measured, per se, but are about improving processes, efficiencies, and quality. Others can be measured by conducting surveys or other methods of assessment such as regularly, and actively soliciting verbal or written feedback. (Talk to your team!)
Goal #1 Ensure the efficient and legally compliant use of open source code.
This is where organizations typically start when they get involved in open source. They realize engineering is consuming a lot of open source software either in their infrastructure, or in their products and services, or both. A program office helps centralize policies and decision-making around open source consumption, track its use, and ensure the organization doesn’t run afoul of its legal obligations under the various open source licenses. Programs can also keep track of how well they help developers resolve any legal issues they may encounter.
Some of the most common ways to measure against this first goal are aimed at ensuring that policies and processes are working the way they were intended and that the organization remains in legal compliance:
- How much open source code do you consume?
- How well is that consumption tracked?
- The policy for using open source code is clear and developers are aware of it.
- The processes and tools for bringing in code are clear and developers are following it.
- Which products and services are third-party code being used in?
- How many compliance issues are you having and how quickly are they resolved? (You do you have a compliance program, right? See our legal resources from the Open Compliance Program for more on this topic.)
Goal #2 Increase developer productivity.
Once you’re tracking and managing your open source use, you’ll want to make it easier for developers to contribute to open source projects. If your engineers have to go through layers of red tape to submit a bug fix or new feature to a project that your business depends on, you’re wasting precious time and resources. Developers also save considerable time over the long run by contributing upstream, rather than maintaining a separate fork of the project which accrues technical debt over time.
Metrics related to this goal are aimed at greasing the wheels for developers to contribute back to open source projects, as well as increasing the overall amount of code your organization contributes back upstream. Once you remove barriers to contribution and make the approval process clear and quick, you can expect more contributions and efficiencies. Things to track include:
- Number of commits made to external projects identified as strategic to the organization
- Number of developers contributing. Also, who are they and which projects do they contribute to?
- Number of project maintainers employed by the organization (hired and grown)
- Project health for the projects you contribute to
- Sentiment analysis: your organization’s reputation in open source communities
- Are developers aware of the policy for contributing? (You have one, right?)
- Do they follow the process for contributing? (ie must they sign a CLA, etc.)
- Do they ask you for help and are you prompt in providing it?
- Amount of time between software releases – is it increasing or decreasing?
- What are the engineering costs associated with contributing upstream vs. maintaining forked code?
Goal #3 Create and grow open source projects.
This is the primary goal of many open source programs at large, engineering-focused organizations such as Facebook, Google, Microsoft, Twitter, and many others. They’re creating hundreds (or even thousands) of open source projects that aim to solve hard technology problems. The goal is to attract outside users and contributors who bring in new ideas and help advance the technology at a faster pace — a concept University of California, Berkeley, professor Henry Chesbrough calls open innovation.
The many data points available to measure project health are key to tracking against this goal (see the top 5 in the next section). But there are other considerations as well:
- Is there a clear policy for creating new open source projects and are developers aware of it?
- Is there a clear and easy process for creating new projects and are developers following it?
- How easy is it for outsiders to contribute to your organization’s projects
- Project maintainers are welcoming and helpful
- Projects are well-maintained and supported
- Code is well documented
- How to contribute is well-defined
- Other quantitative metrics such as number of new contributors, number of issues created, amount of time it takes to close issues, etc. (see the next section
- Number and diversity of external contributions your projects receive
- Popularity of your organization’s projects: GitHub stars, social media followers, etc.
- Number of users in deployment and/or production
- Number, breadth, and quality of projects your organization launches. For example, mobile or data, infrastructure-related projects, etc.
- Performance increases in your project and related product
- Time between releases
Goal #4 Recruit and retain developers.
Participating in and creating open source projects as an organization is a great way to attract developers — and onboard them quickly, with fewer resources devoted to training. Developers who use or contribute to your projects will already be familiar with your processes, tools, and technologies when they join the organization. (See our guide on Recruiting Open Source Developers.)
But chances are that you as a program manager will not have a direct role in recruiting developers, and it may not be clear what immediate effect your organization’s open source participation has on hiring. To help make a more direct connection between program efforts and recruiting, Facebook conducts a biannual survey which asks new hires three basic questions:
- Are they aware of the company’s open source program?
- How did that awareness influence their decision to join the company?
- Does their experience with open source apply to the work that they are doing now?