Last week, I had the pleasure of speaking to a small group of Women in Data Science Puget Sound volunteers about my book Driving Data Projects. They specifically wanted to talk about the topic of measuring ROI.
In the world of data-driven decision making, we often fall into a trap that I call the “measurement paradox.” It’s a sneaky cognitive bias that leads us to focus on what’s easily measurable, rather than what’s truly important. Let me illustrate with a simple example.
The Measurement Mirage
Imagine I ask you what makes a restaurant “efficient.” Most people immediately think of speed of service. Similarly, when asked about an “efficient” house, the common response is heating costs. But here’s the rub: do you choose a restaurant primarily for its quick service? Sometimes, but not always. Do you buy a house for its low heating bill? Generally, people buy a home for location.
The most more interesting things are hard to measure. Consider the concept of “efficiency.”
However, we zero in on the quantifiable aspects, missing the forest for the trees. As management scholar Henry Mintzberg aptly noted, “The interesting things don’t lend themselves to measurement”" [1].
This paradox isn’t just a quirk of casual conversation. It’s a pervasive issue in how organizations approach data initiatives and measure their ROI. Let’s explore two common storytelling mistakes that stem from this paradox.
Mistake 1: Overdependence on Progress Metrics
Many data teams fall into the trap of overemphasizing technical progress metrics. They meticulously track story points completed, models deployed, or databases optimized. While these metrics are important for project management, they often fail to resonate with business stakeholders.
The problem? This approach focuses too much on the “how” rather than the “why” of a project. It can create a false sense of accomplishment where completing tasks is seen as success, even if those tasks don't translate to business value.
Mistake 2: Overreliance on Financial ROI
On the flip side, some organizations swing too far in the other direction, reducing complex data initiatives to a single financial figure. While ROI is crucial, an overreliance on these numbers can be misleading and often fails to capture the full value of data projects.
This approach may create unrealistic expectations or promises that are difficult to meet. More importantly, it often fails to capture intangible benefits like improved decision-making quality or increased data literacy in the organization.
Real-World Lessons: Netflix and Amazon
Let's look at how two tech giants grappled with these storytelling challenges:
Netflix: Beyond Binge-Watching
Netflix initially focused on completion rates and total viewing hours as key metrics for their recommendation system. However, they realized these metrics didn't truly capture user satisfaction. A user who watches many hours of mediocre content isn't necessarily a happy user.
In response, Netflix shifted to a more nuanced “joy per minute” metric [2]. This change required more sophisticated measurement techniques but ultimately provided a more accurate picture of the recommendation system's value.
Amazon: The Hidden Value of AWS
When Amazon first developed AWS, they primarily justified it based on projected cost savings for their own operations. However, they soon realized this narrow focus missed the bigger picture.
As AWS evolved, Amazon broadened their value assessment to include metrics on speed of innovation and new revenue streams [3]. This shift in storytelling helped transform AWS from an internal cost-saving initiative to a major profit center and industry-changing product.
Crafting a Better Data Story
To avoid these common pitfalls, we need a more balanced approach to measuring and communicating the value of data initiatives:
Translate progress metrics into business outcomes
Use financial projections judiciously, alongside qualitative benefits
Tell a holistic story that combines data, narrative, and business context
Focus on how the project solves real business problems or creates new opportunities
For instance, instead of just saying “our model is 95% accurate,” you might say “our 95% accurate model is helping us reduce time to launch Marketing Campaigns by 2 Weeks, which will increase Average Order Size by 12% for “Tech Enthusiasts” and increase customer lifetime value.”
By adopting this more nuanced approach, we can bridge the gap between data teams and business stakeholders, ensuring that our data initiatives not only deliver value but are recognized for doing so.
Remember, in the world of data, the most valuable insights often lie beyond what’s easily measured. Our challenge is to illuminate these insights, not just count what’s under the streetlight.
“I only see examples like that in resumes.”
This candid response from a junior participant struck a chord. It mirrored a question a seasoned colleague asked the week before: “Is this (what you write about) even possible?” These sentiments highlight a pervasive issue in our field: the gap between idealized ROI measurements of data projects and the messy reality most of us face daily.
On one side, we have polished case studies and resume bullet points showcasing perfect alignment between data initiatives and business outcomes. On the other, we have the complex, often chaotic world of real-world data projects where measuring true impact can feel like catching smoke.
This disconnect isn't just an academic concern. It impacts how we allocate resources, demonstrate value to stakeholders, and, ultimately, drive data-led transformation in our organizations.
So, why does this disconnect exist? Let's unpack the key reasons:
Time and resource constraints: Comprehensive evaluation requires significant investment. Teams often prioritize immediate tasks over long-term assessment.
Driving Data Projects provides efficient frameworks for quick yet comprehensive business and data project collaboration and execution, offering templates and tools to streamline the approach and the process. It teaches project management and change management techniques specific to data project evaluation.
Skill gap: Translating technical metrics to business value isn't easy. It requires a rare blend of technical, business, and communication skills.
Christine provides workshops on translating technical metrics to business value. Driving Data Projects covers effective communication between technical and business teams and shares case studies and best practices to bridge the technical-business divide.
Organizational silos: Data teams and business units often operate separately. Lack of collaboration hinders holistic evaluation.
Driving Data Projects introduces collaborative methodologies for bringing data teams and business units together. It provides a framework for teams to consider a strategy for creating cross-functional data teams and offers guidance on implementing data initiatives that encourages collaboration.
Short-term focus: Pressure for quick wins overshadows long-term value assessment. Quarterly targets can discourage in-depth, time-consuming analysis.
Driving Data Projects demonstrates methods to balance short-term wins with long-term value assessment and provides frameworks for ongoing project evaluation beyond initial implementation. It offers strategies to communicate long-term value to stakeholders effectively.
Measurement complexity: Intangible benefits are inherently difficult to quantify. Long-term impacts may not be immediately visible.
Driving Data Projects introduces advanced techniques for quantifying intangible benefits and provides tools and methodologies for tracking and projecting long-term impacts. It offers guidance on creating comprehensive dashboards that capture both tangible and intangible value.
The answer to their questions? Yes, it’s possible. I’ve lived it and helped build it. Bridging these gaps is crucial for data initiatives’ success and recognition. Schedule time with me to discuss the next steps for your organization.
Data leaders: How are you addressing this challenge in your organization?
Resources
[1] Mintzberg, H. (2017). Managing the Myths of Health Care: Bridging the Separations between Care, Cure, Control, and Community. Berrett-Koehler Publishers.
[2] Gomez-Uribe, C. A., & Hunt, N. (2015). The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Transactions on Management Information Systems, 6(4), 1-19.
[3] Barr, J. (2021). 15 Years of Amazon EC2. AWS News Blog. Retrieved from https://aws.amazon.com/blogs/aws/15-years-of-amazon-ec2/