How do you measure the health of your community, identify problems, and track progress towards your goals? What should you be measuring?

Last month we discussed vanity metrics, those metrics that might sound impressive on the surface, but ultimately give you little insight or guidance to improve the health and well-being of your community. This naturally begs the question: What should you be measuring? And as I mentioned last month, the obvious but annoying answer: It depends. The first and foremost dependency relates to the nature of your community and where you and your members want it to go.

Diversity of communities is a natural result of the diversity of the people that make them up, but this also makes it harder to measure and spot trends for the purpose of improvement. Each community has its own mission, goals, and central thoughts around which people collaborate, and the measurements you take must take those into account, without wasting yours or anyone else’s time. In his book The Art of Community, Jono Bacon states:

 

“The goal here is not to construct an enormous vacuum cleaner to suck every tiny detail of your community into a graph. The goal is instead to identify what we don’t know about our community and to use measurements as a means to understand those things better.”

 

Bacon goes on to explain that you also must be willing and able to make changes based on those measurements. Measuring for the sake of measuring is pointless. So with that in mind, we arrive at the first step in your quest for effective community metrics: setting goals.

Metrics goals

Goals are important at every level of community, because they give you constant and concrete targets to aim at as your community evolves. Open source communities usually start with an implicit goal or mission defined by its founders: to create effective and open software that solves some problem for its users. And that’s usually where it stays for quite a while. If you’re reading this, chances are good that you are involved in an existing open source community building software that solves a problem, with more than a handful of members, and you want an ongoing measurement of its health with an eye toward improvement. If you are going to spend time and money collecting and analyzing metrics, you first need to know what you want out of the effort.

The goals for your metrics program are not specific to each measurement, but to the metrics program as a whole. They should also relate as closely as possible to the overarching mission of the project. This ensures that if you are making progress toward your metrics goals, you are also making progress toward the project’s goals. In my former life as community manager at Liferay, we thought a lot about what we wanted out of the community, how it related to the business, and used that to develop the following goals for our metrics program:

Goal #1: Increase the value of participation and contribution

According to the Community Roundtable’s 2016 State of Community Management, best-in-class communities (the top 20% in terms of maturity) were twice as likely to measure community value. We too believed strongly that the more value a community member received from our community, the more they would be willing to join and remain, advancing their own agenda as well as benefiting the community, the company, and our corner of the industry as a whole. Later on, we used this to define specific measurements we would track, but we had to first understand what our members valued. Instead of guessing, we surveyed our community: What motivates you to be in—and contribute to—our community?

The results reaffirmed our belief that those in the Liferay community were primarily interested in sharing, learning, and giving back. They didn’t want money, which should come as a relief to many of you in volunteer communities with little or no financial support.

Goal #2: Measure the effect of business decisions

We wanted to understand whether we were getting enough bang for our buck. When we made business decisions, what impact did that have on our community? A common scenario for open source projects is whether to spend money to attend or sponsor a conference. We wanted to know whether the spend was worth it. Having this goal in mind helped us define and collect metrics related to conference participation. Our metrics program also had an unintended but welcome side effect: Knowing that we were being measured, we made an extra effort. Instead of showing up and handing out t-shirts, we set up an engaging in-booth developer activity that gave visitors a small yet compelling experience of being a member of our community, and made it even easier to measure success. Yeah, we should have made that extra effort regardless, but putting a little extra rigor into our community brought out the best in our people (myself included).

Goal #3: Understand the relationship between community metrics and business performance

This one is somewhat related to the previous goal, but in the opposite direction. It helped us see relationships between community performance and business performance. It doesn’t tell us if we made the right decisions in the past; it tells us what we should concentrate on in the future. For example, if we see a strong correlation between growth of community champions and business revenue, that can help us fine-tune future spend in this area. However, there are dragons here: correlation does not imply causation. Take a look at this study of lemon imports and highway fatalities, and let me know what you think would happen if the USA imported more lemons.

^{Data from: NHTSA, DOT HS 810 780 and ​U.S. Department of Agriculture}

When using the results of your metrics, you must consider additional factors, such as their accuracy, whether they are the possibility of a common cause, or they are simply coincident and not related at all.

In summary, before embarking on your metrics program, you must define goals for the program itself, and those goals should related to the mission of the project. The goals you develop will help you decide exactly what you are going to measure, the topic of the next article. See you next month!