Digital transformation: Are you using outdated IT metrics?

It’s practically a commandment in IT: Thou shalt measure in order to manage. Performance metrics are critical to the success of IT leaders, their teams, and the enterprise as a whole.

Unfortunately, many traditional IT metrics – uptime numbers, service availability, project management benchmarks, budget adherence – fail to tell the full story of IT performance in the digital transformation age. As the business relies almost completely on IT to compete today, IT leaders must develop some new metrics, or tweak existing ones, to paint an accurate picture of IT’s contribution to the business.

[ Read our new report from HBR Analytic Services: Transformation Masters: The New Rules of CIO Leadership. ]

A 2017 Gartner study found that CIOs think the ideal balance should be 56 percent of metrics related to business outcomes, such as those related to revenue growth, business margins and influencing business strategy, and 44 percent related to IT delivery. But even those who believe they have an ideal ratio may find it hard to hone in on the best individual metrics.

Land O’Lakes CIO Mike Macrie recently wrote for us about how his team found that cost-minded IT metrics actually conflicted with stated digital transformation goals. So he separated IT operations from the rest of IT, with separate funding mechanisms and metrics. Then he designated a group within IT that focused on innovation and didn’t have to worry about cost or efficiency as a primary measure of success. (Read the full article: Land O'Lakes CIO: Old IT metrics hamper new transformation goals.)

Finding the right metrics for any particular IT organization today will require continued experimentation and tweaking over time. But here are seven that show promise and may spark ideas for other leaders looking to better manage the technology organization and calculate its value to the business.

Customer satisfaction

“The most important thing IT organizations can do when determining which metrics to track is to take a step back and evaluate what is important for them and what is going to drive value back to the business,” says Justin Stefano, director in West Monroe Partners’ Performance Services practice. “There are a slew of operational metrics that can be tracked across a wide range of operational teams. The challenge is ensuring that the operational metrics can feed the management metrics, in turn creating a value picture for leadership.”

Every firm values IT in different ways, but for Stefano the most important metric CIOs should track is one that can put a number of the value IT provides to its internal customers. In other words, some form of an internal Net Promoter score. “From there, I would design a handful of metrics that cascade down to each team that supports improving that value score,” he says.

Customer ticket volume

Red Hat technology evangelist Gordon Haff notes that teams using the DevOps/Agile way of working as they chase digital transformation goals must be especially careful to choose metrics that are tied to business outcomes in some manner. “You probably don’t really care about how many lines of code your developers write, whether a server had a hardware failure overnight, or how comprehensive your test coverage is. In fact, you may not even directly care about the responsiveness of your website or the rapidity of your updates. But you do care to the degree such metrics can be correlated with customers abandoning shopping carts or leaving for a competitor,” he notes.

One example: “Customer ticket volume is a reasonable proxy for overall customer satisfaction, which, in turn, strongly affects higher-level (and highly valued) measures such as Net Promoter Score, the willingness of customers to recommend a company’s products or services to others. At the same time, tickets tend to be filed for specific perceived shortcomings related to applications and infrastructure,” he says. (Read the full article: DevOps metrics: Are you measuring what matters?)

First time right (FTR)

Wendy Pfieffer, who led technology teams at GoPro, Yahoo!, and Cisco before taking on the CIO role at cloud computing software company Nutanix, has been focused on using automation and machine learning to help IT scale more effectively and improve the user experience. Using the First Time Right (FTR) metric, with roots in Lean Six Sigma processes, she is able to measure both the quantitative and qualitative impact of automation on IT and its constituents.

“We measure all of our interactions with our users and we use the scoring from those measurements to prioritize processes and interactions that need to be worked on,” she explains.

When IT began measuring its FTR score with regard to software download requests, they found they were getting them right the first time in only 8 percent of instances. The lackluster numbers led IT to implement a machine-learning tool to automate what had been a series of cumbersome interactions and interfaces on the back-end of the download request process. Now if a user requests a software download via Slack, for example, the AI tool handles all of the processes involved, including creating the tickets and managing signoffs, in less than a minute. And the FTR score for the process has jumped from 8 to 100 percent.

Downtime costs

Systems availability will always be an important metric for the IT organization. However, in isolation, it fails to measure the true impact of downtime or system slowdowns. “The most important thing organizations should know, and most don’t, is the cost per minute of downtime. Degraded performance can be considered downtime as the result is often the same,” says Trent Fitz, chief strategist with Zenoss, which makes cloud-based IT management software. “Understanding the cost of degraded performance and downtime is required to make intelligent decisions on investing in the tools and processes to detect and mitigate these scenarios.”

There are some industry standard benchmarks on downtime costs for companies, but they are unlikely to give more than an average across all kinds of systems in all kinds of companies. Instead, Fitz suggests CIOs should determine the cost of downtime per minute for a particular service. “In some cases, customers may spend more money to ensure services are up than it would cost them if there were outages,” he says. In other cases, the reverse is true.

Innovation investment

The common wisdom is that most IT organizations spend 80 percent of their budgets keeping the lights on and 20 percent on innovation. In the digital era, the stretch goal for most CIOs is to invert that. In a recent article for the CIO Association of Canada, business and IT strategy advisor Peter D. Moore suggests CIOs set a target percentage of the budget each year for IT to shift to changing the business and report results quarterly. To do that, IT leaders can identify and quantify the savings they achieve through automation and other efforts.

Shadow IT spending

Ian Pitt, CIO at LogMeIn, has incorporated some of the metrics noted above, including user NPS scores. One other important thing he keeps an eye on is the level of independent spending on IT solutions, which serves as an indicator of how well IT is meeting business needs.

“With it being so easy for end users to introduce technology with nothing more than a wish and a credit card, we monitor the mix of technology we’ve introduced versus that brought in by the users,” Pitt says. “If it’s in our favor, we know that the IT teams are staying ahead of business needs and being proactive. If not? Well, we need to up our engagement levels.”

Top-line impact

The holy grail for IT metrics are those that relate to actual business outcomes – revenue growth, profitability, and new products and services. Intel’s technology organization has gone as far as delivering an IT annual report each May documenting not only IT’s operational performance over the previous year but also its top-line impact. Its 2016-17 report highlights, for example, a 39-week improvement in bringing Intel products to market via a machine-learning platform, and an enterprise predictive analytics platform that created $656 million in business value.

[ How can automation free up more staff time for innovation? Get the free e-book: Managing IT with Automation. ]