Metrics and ROI. Tell Your Story. Where Do I Start? Matt Hutchinson

Transcription

1 Metrics and ROI Tell Your Story Matt Hutchinson One of the first steps I took on taking charge of the Admiralty and becoming a member of the War Cabinet was to form a statistical department of my own They examined and analyzed with relentless pertinacity all the departmental papers which were circulated to the War Cabinet and also pursued all the inquiries which I wished to make myself. At this time there was no general governmental statistical organization. Each department presented its tale on its own figures and data. The Air Ministry counted one way, the War Office another, the Ministry of Supply and the Board of Trade, though meaning the same thing, talked different dialects. This led sometimes to misunderstandings and waste of time when some point or other came to a crunch in the Cabinet. Sir Winston Churchill wrote this in 1948, but it could have been written last week. It is still difficult for an organization to tell its story in spite of the wider availability of data and often because of it. The challenge is acute in the public sector, where leaders do not have standardized financial statements to lean on. In that context, how does one define and demonstrate a solid return on investment? When funding is available, how can one leverage metrics to capture it? When cuts are being made, how does one erect defenses? Appropriate metrics enable leaders to make the best possible case for their organization. One can even see this battle being waged in Churchill s comments above; who won the argument the Air Ministry or the War Office? Odds are it was the one who knew how to tell its story the best. Where Do I Start? We refer to anyone who provides funding to your organization as an investor. Loosely speaking, they want to measure their return on investment. Your investors have many options for their investment; they will likely rely heavily on performance metrics that you provide. Further, they are often in a position where they have to compare your apples to someone else s oranges. They do not need to necessarily understand the apple business, but they do need to understand how their investment will interact with it. The metrics one tracks should correspond to the manner of their investment. This leads us to principle number one: Principle Number One: Any metric one formally reports should tie directly to the context of your investors investment. When their investment goes up, your metric should go up. When their investment goes down, your metric should go down. One may need to estimate a lag between the investment increasing and the metric increasing, but at some lag the investment and the metric should be very highly correlated. Even if it does not perfectly track your performance, it will provide investors with a return on their investment. Remember, they are likely making a wide range of investments and trying to compare them your competitors

2 may not have any appropriate measures, and in that case you will prevail even with a reasonable, if imperfect, metric. Your business will change over time, and some metrics that were perfect five years ago may have outlived their relevance. Investors understand this, provided it is communicated properly. Principle Number Two: Institute an open season, say once per year, for you to ask for and recommend changes in your currently reported suite of metrics. This gives you the opportunity to communicate with your investors about the changes in your business and ensures that your performance is not rated on antiquated measures. One of the easiest things to put off, particularly when it comes to metrics, is documentation. Without it, employee turnover can leave you flat footed with key metrics yet to be collected. The third principle is one of the toughest to do, but will someday save you: Principle Number Three: Carefully document the methods for gathering your metrics and create a data steward that is responsible not only for producing the metrics, but also for this documentation. Make that person sign off on the metrics when they are produced. To test the level of documentation, periodically ask someone else to gather the metrics based purely on the available documentation. Now, it is likely that you are also an investor yourself. Once you have defined your key metrics, you can use the above principles to create sub-metrics that support your own metrics. This leads us to the final principle: Principle Number Four: Think of yourself as an investor, and require your investees to follow principles one, two, and three on your behalf. People Drive Data Now that we have some overarching principles to work from, what do we do? To answer that question one really needs to understand the various players in the metrics ecosystem and, more importantly, their biases. The era of big data is built on the foundation of technology that allows easy storage, retrieval, and processing of extremely large data sets. This is good news in general, but it naturally leads to specialists each playing their particular roles. These could be broken down further, but there are basically four main roles in an organization of any size (with some overlap one person could play several of these roles): Database Experts: They are trained to store and retrieve data in an efficient manner and have no real need to know what they are storing, past its size and other related requirements. They do not really require an intuitive feel for the context of the data, and documentation is typically a low priority.

3 Statisticians/Data Scientists: They are trained to take a given data set and create a model that best explains it. Here again their training does not demand a detailed knowledge of the data s context. They typically enjoy creating models much more than tracking down data integrity issues. This can lead to incorrect (and material) assumptions. Businesspeople/Analysts: This could be a cashier in a grocery store or a trader on a stock exchange. In any case, they know what is happening on the ground. They own the context of the data and often populate databases. Because they are not focused on the integrity of the data, what they enter can be inconsistent and ultimately confusing to the statistician when he retrieves it from the database expert. We refer to this class as analysts below. Leadership: They know the story they are trying to tell, but do not necessarily know all the data that is available to help tell that story. Many leaders naturally treat metrics as fact and are largely unaware of the many assumptions that the other players are making assumptions that contribute materially to the final metrics product. Thus one needs to understand the entire system s dynamics into account: leadership knows what investors are looking for, but does not necessarily know what is being measured and what resulting metrics could be gathered. The statistician does his best to approximate what he thinks the leader is looking for hence a dilution of what the investors are looking for but he relies on analysts to enter data correctly and the database expert to pull the correct data. In this case, the correct data is based on how the database expert interprets the request from the statistician. Note that this entire process could be completed without the statistician or database expert talking to any analysts about the context of the data they are presenting. Critically: One must take the entire metrics system into account; even though every member of the system might be doing something reasonable and rational when considered alone, that member s interaction with the other members can lead to material mistakes. Metrics are rarely as perfect as one takes them to be once they are produced. What we see is: 5,325 What we are really getting is:

4 Final Notes The following steps, properly executed, can provide useful ammunition in funding battles: 1. Suppose you have a list of formal goals given to you by your investors. If not (even better), create them. In any case, make sure you have a set of goals that both you and your investors have agreed to. 2. At the beginning of the year/period (not the end!), find out what data is being collected already and document it. Ask yourself: if I had to tell my story today, what data would I have to do it with? If your goal is to increase the number of people working for you, say, how are you going to count it? How will you count part-time people? Even something this simple can surprise you. 3. As soon as you have collected enough data (say one month), do a dry run. This serves two purposes: a. It helps you work out the data gathering process and the inevitable nuances that arise. You will find out what data you want but do not have, and as long as it is early on, you may be able to start collecting it now to report at the end of the year. b. The results may surprise you, and you may want to make some managerial changes already if your numbers are not where you want them. 4. Repeat step 3 for another few months (say). Obviously, the more you can compress this process the better. 5. Now that you have the process under your belt and you know what metrics are useful, start mapping your high level metrics to lower-level metrics for groups that report to you. 6. When the final reporting period comes around or, actually, every reporting period after the first few you will have a handy story that you can tell to anyone. In many cases simply having some numbers will win the battle. One must infuse the four principles listed above into this process, though. Make sure your metrics correlate to the funding you receive. Make sure you build in flexibility from year to year with an open season. Ensure that data stewards really understand what they are reporting, especially all the subtleties that come with collection and reporting (as discussed in People Drive Data ). Now that we have covered what to do, we might ask the question: What happens if I do not use these principles and follow these steps? The graduate school admissions figures for the fall of 1973 at UC Berkeley include: Applicants Admitted Men % Women % Berkeley was sued for gender bias; any statistician will tell you that these statistics were very unlikely to occur randomly. An open and shut case, right? Maybe

5 Department Men Women Applicants Admitted Applicants Admitted A % % B % 25 68% C % % D % % E % % F 272 6% 341 7% As you can see, many women were applying to the departments that were difficult for everyone to get into, resulting in a lower overall acceptance rate. In fact, based on this information there is actually a small bias in favor of women. Is your business a little more complicated than counting applications?