At the OpenStack summit taking place this month in Barcelona, Ildikó Váncsa will be speaking on metrics in her talk Metrics: Friends or Enemies? She will discuss OpenStack metrics and how they can be used in software development processes, both for the individual developer and manager.

I caught up with Ildikó before her talk to learn more about how metrics in OpenStack help guide developers and companies, and how they also drive evolution of the OpenStack community itself.

What is your thought process when deciding which metrics are most meaningful to track? Which aspects of a given metric are you most interested in?

From an OpenStack developer community perspective, I think it is very important to provide a basic set of metrics. It is also crucial that these counters are easy to interpret and can be collected in an automated manner. In an open source community like OpenStack, the workflows and processes are different from a regular corporate environment, therefore it depends on the consumer of these numbers and how they are using it. In this sense, the simplicity and constant availability of the data are key to the users and participants of OpenStack, such that they have a stable source where they can pick what is most meaningful for them to track.

When we are looking into OpenStack as an open source software package, the numbers to look at are the ones that are the most accurate. It is very important to see how this software package as a product performs in production, to get the latest user feedback, and to follow the trends of evolving new technologies that users and operators are looking at.

What are some of the challenges companies face when transitioning processes to open source?

It is very challenging in itself to start upstream work and open up parts of the development process. This way, a formerly closed ecosystem becomes less predictable by having more moving parts on which the given company or individual does not have full control. The same metrics, like LoC (Lines of Code) are still meaningful, but can paint a false picture about efficiency and open source adaption if read in the same way as before.

From a corporate perspective, the ultimate goal in contributing to open source is usually related to the business. What kinds of metrics do you think companies look for when trying to determine the cost vs. benefit of contributing to open source?

On one hand it is important to follow the market adoption of a given software package and be up to date on how the different services perform that are important for the different industries.

On the other hand we have the development process to follow to ensure the time-to-market values are feasible. It is very easy to track progress—like commits—in modules of an open source project. Although the data is misleading if one uses it only as a value to increase. In order to be successful, you need to be sure to have influence in those modules that are important for your product and business.

Does that mean that for companies to see real benefit from their open source participation, they must build their reputation in those modules that are important to them?

In order to be successful and to have your feature and changes land, you need to become part of the community and be active in those projects that are in your area of interest and where you want to make changes. Moving these projects in the right direction is a team effort. If a company only checks let’s say the number of commits overall and not look any deeper than that, they may still misplace their investments as their developers might not be part of those teams that are important for that company’s business. When developers start to contribute code and add new features, they become part of that team that’s responsible for a project’s direction, which ensures they are visible and their voice is heard.

The emphasis isn’t necessarily on influence, but more on those projects important for the business. So in other words they must do the community engagement activities, where they are really willing to participate to support both the community and their business. I wouldn’t say that they need to steer the module—I would say they need to participate in it.

On the developer front, do you think individual contributors benefit from seeing the published project metrics? Does OpenStack use metrics as part of the onboarding process for new developers?

When I started my journey with OpenStack, it was motivating to see how I progressed. It was true for both the development activities I was involved in and the publicly available metrics that I found. It is also a good source to the active members and their areas of expertise during the release cycle.

From an onboarding perspective, we are teaching the newcomers best practices on how to use the available data in their advantage. We also started to experiment with how to use that data during the OpenStack development process—for example, improving the upstream trainings we offer before our bi-annual summits.

In parallel to this, there are plans to analyze how we as a community and our project teams perform with onboarding new members to improve the process and provide more entry points and better ways to engage.

Are there any examples of OpenStack projects making changes—either to the code or process—as a result of metrics analysis?

There are initiatives to reduce code complexity and remove copy-paste code to ensure the long-term maintainability of our code bases. The analysis and action is not always initiated by the project teams in OpenStack, but by new contributors who have data analysis expertise. It is a really nice example of non-coding open collaboration in the community.

Beyond this we are restructuring our upstream trainings to fit the needs of students better. We are covering more areas of participation to be able to reach out to more people than just the developers, while still keeping the code deep-dive modules in focus.

What are your favorite tools for analyzing and reporting project metrics?

Personally I use Stackalytics the most, but I’m still at the beginning of my journey with metrics. In this sense my focus area at this point is to help contributors and ecosystem companies to leverage all the metrics and statistics that are available for them. It is important to point developers to data that can help them be more efficient and successful and also to help managers to set the right expectations towards their teams by using the numbers in the right way.

Any other talks in Barcelona you’re interested in?

I haven’t finalized my schedule yet for the Summit, but my main areas of interest are success stories of organizational cultural change and sessions related to the Working Groups in OpenStack, and presentations on the Telecom/NFV track.