As we entered the 2010s, corporate America underwent a seismic shift. The relentless pursuit of efficiency that characterized the 1990s and early 2000s gave way to a new paradigm—one that prioritized purpose and profit. While addressing crucial issues of employee burnout and societal expectations, this transformation inadvertently set off a chain of events that would profoundly impact data culture and leadership across organizations.

The Data-Driven Years & the Purpose Revolution

In the late 1990s and early 2000s, CIOs stood at the forefront of the data revolution. They oversaw the implementation of massive data warehouses, complex business intelligence tools, and enterprise-wide systems that promised to turn information into a competitive advantage.

The rise of purpose-driven organizations was swift and widespread. By 2015, over 80% of Fortune 500 companies had adopted explicit purpose statements, a dramatic increase from less than 30% a decade earlier. This shift represented more than just a change in corporate messaging; it signaled a fundamental reimagining of business's role in society and its relationship with employees.

Human Resources departments tasked with implementing this new vision began restructuring incentive systems and training programs to align with purpose-driven goals. The focus shifted toward developing soft skills, emotional intelligence, and cultural competence. Methodologies like Six Sigma and PMP fell out of fashion and were regarded as “less human-centric.” Lenses like IDEO’s human-centered design became popular again. While these initiatives addressed important aspects of employee well-being and engagement, they had an unintended consequence: the de-emphasis of certain critical technical and analytical skills.

The Project Management Institute's 2023 Pulse of the Profession report highlights a shift in the skills organizations prioritize. While technical skills remain important, there's an increasing emphasis on power skills such as communication, problem-solving, and collaborative leadership. This trend aligns with broader shifts in workplace learning priorities. The 2023 LinkedIn Workplace Learning Report notes a growing focus on programs that develop human skills and support employee wellbeing, with leadership and management training being the top priority for L&D professionals globally.

The Emerging Skills Gap

As organizations pivoted towards purpose, a subtle but critical skills gap emerged. The hard, data-driven competencies that had been the hallmark of effective management in the efficiency-focused era began to fade from prominence in many corporate training programs and organizational cultures. The World Economic Forum's Future of Jobs Report 2023 underscores this shift, projecting that 44% of workers' skills will be disrupted in the next five years, with analytical and creative thinking among the top skills expected to be in high demand.

Inadequate training opportunities further exacerbate this skills mismatch. The same WEF report found that only 35% of workers currently have access to adequate training opportunities in their organizations, highlighting a growing gap between the skills needed in the modern workplace and the training provided by employers.

The de-emphasis on certain technical skills coincides with a trend towards outsourcing developmental support. A recent study by the International Coach Federation reported a 33% increase in executive coaching from 2015 to 2022, suggesting that while organizations recognize the need for ongoing skill development, they may be essentially outsourcing or relying more on external resources rather than integrating these skills into their core training programs.

This shift hasn't gone unnoticed by employees. More than half (52%) of workers say their job requires specialized training, yet almost one-third (29%) of workers don’t feel optimistic about the opportunities they have for training to learn new skills or upskill. This disconnect between required skills and provided training represents a significant challenge for organizations striving to balance purpose-driven initiatives with data-driven decision-making.

The impact of this skills gap is particularly pronounced in data literacy. A 2023 Gartner survey found that poor data literacy is cited as a key factor in 60% of organizations' failure to realize value from data and analytics investments. This underscores the critical need for organizations to address the data skills gap, even as they pursue purpose-driven initiatives.

As we'll explore in the next section, this evolving skills landscape has had profound implications for the roles of data leaders in organizations. With HR's focus shifting toward purpose-centric (now “employee-centric” or “human-centric”) competencies, Chief Data Officers and Chief Information Officers increasingly find themselves at the forefront of addressing these critical skill gaps.

The Evolving Landscape of Data Leadership

As these changes unfolded in HR and broader organizational culture, the roles of Chief Information Officers (CIOs) and Chief Data Officers (CDOs) were undergoing their evolution. Initially focused primarily on infrastructure and data management, these roles began to expand to fill the accountability gap left by the shift in HR priorities (from efficiency to purpose).

This makes sense. Organizations with early data legacies (e.g., Yahoo!, Microsoft, Amazon, General Electric, etc.) were helping the business become more informed. The 2023 State of the CIO report by IDG Communications found that 84% of CIOs report expanding their role to include more strategic business initiatives. Similarly, the 2021 NewVantage Partners Big Data and AI Executive Survey revealed that 80% of CDOs now list "cultivating a data culture" as a top priority, up from just 45% in 2018.

This expansion of responsibilities meant that skills once considered general competencies fostered by HR through performance management mechanisms–critical thinking, effective feedback, data analysis–increasingly fell under the purview of data leadership. CDOs, in particular, found themselves tasked not just with managing data assets, but with building organization-wide data literacy and fostering a culture of data-driven decision-making.

The Challenge of Cultural Transformation

However, this shift has created new challenges. Unlike HR departments, which traditionally have organization-wide reach and influence, CDOs often lack the structural position, budget, and cultural clout necessary to drive such broad-based change.

The same NewVantage Partners survey that highlighted the increased focus on data culture also revealed a stark reality: despite its acknowledged importance, only 23% of organizations report successfully cultivating a data culture. This gap between aspiration and reality represents one of the key challenges facing data leaders today.

The adoption of the CDO role has grown significantly over the past decade, with 65% of firms reporting having a Chief Data Officer, up from 12% in 2012. However, the mere presence of a CDO doesn't necessarily indicate organizational data maturity. A 2022 Gartner study found that only 21% of CDOs report high levels of effectiveness and success in their role. Recognizing the need for a role and translating that into an effective data strategy remains challenging. Today, CDAOs must reinvent themselves or risk being assimilated back into the CIO’s organization. Three-quarters of CDAOs who fail to make companywide influence and measurable business impact their top priorities by 2026 will be assimilated IT functions.

The next frontier for data leadership may be reconciling the often-competing demands of data-driven efficiency, ethical considerations, and purpose-driven initiatives. Success in this arena will require a new breed of data leader who can speak the language of technology, business strategy, and societal impact with equal fluency. The CDO of the future will not just be a data expert but also a business strategist, an ethical guardian—in a single change agent.

It is an open debate whether the CDO role should remain a stand-alone role or return to the fold of the CIO. Data strategy often underpins digital strategy. As noted by Gartner in their 2022 CIO Agenda, “Data and analytics are the foundation of digital business... CIOs must ensure that data and analytics are an integral part of their digital business strategy, not an afterthought.” This interconnection highlights why CIOs remain deeply involved in data strategy even as they focus on digital transformation. The two are not mutually exclusive but rather complementary aspects of modern IT leadership.

The Paradox of Integration: A New Challenge for Data Leadership

As we look to the future, the path forward for data leadership is far from clear-cut. The pursuit of organizational ambidexterity - balancing purpose-driven exploration with data-driven exploitation - presents a paradox that challenges traditional notions of corporate structure and leadership.

The rise of the CDO role, initially seen as a solution to the data literacy gap, may have inadvertently contributed to the very problem it sought to solve. By centralizing data expertise, organizations risk creating new silos, potentially widening the gulf between data-savvy leaders and purpose-driven employees. The collaborative C-suite model, while promising, faces significant hurdles in implementation, not least of which is the entrenched culture of CEO-centric leadership in many organizations.

Moreover, the push for data literacy across all levels of the organization may be at odds with the trend towards specialization in other areas. As the World Economic Forum's Future of Jobs Report 2023 indicates, the skills landscape is becoming increasingly complex and fragmented. This raises a provocative question: In our quest for data-driven decision making, are we inadvertently creating a new class divide between the data-literate and the data-illiterate?

The solution may lie not in further specialization or in creating more C-suite roles, but in a fundamental rethinking of how organizations approach learning and development. Rather than treating data skills as a separate domain, forward-thinking companies might consider embedding data literacy into every aspect of their training programs, from onboarding to leadership development.

However, this approach requires a level of organizational agility that many companies lack. It demands a willingness to continuously evolve job roles, performance metrics, and organizational structures in response to rapidly changing skill requirements. Such flexibility runs counter to the stability and predictability many large organizations traditionally value.

As we move forward, the true test of data leadership may not be mastering the latest analytics tools or crafting the perfect data strategy. Instead, it may lie in the ability to navigate the complex human dynamics of organizational change, bridge the gap between purpose and profit, and foster a culture where data literacy is not just a skill but a mindset shared by all.

The future of data leadership, then, is not just about data. It's about reimagining the very fabric of organizational life in an era where purpose and precision must coexist and synergize. Those who can master this delicate balance may well define the next era of business leadership.

Dr. Christine Haskell is a collaborative advisor, educator, and author with nearly thirty years of experience in Information Management and Social Science. She specializes in data strategy, governance, and innovation. While at Microsoft in the early 2000s, Christine led data-driven innovation initiatives, including the company's initial move to Big Data and Cloud Computing. Her work on predictive data solutions in 2010 helped set the stage for Microsoft's early AI strategy.

In Driving Data Projects, she advises leaders on data transformations, helping them bridge the divide between human and data skills. Dr. Haskell teaches graduate courses in information management, innovation, and leadership at prominent institutions, focusing her research on values-based leadership, ethical governance, and the human advantage of data skills in organizational success.