Executive Summary
Hakkōda State of Data Report
2024
01.
Executive Summary
Organizations across industries are poised to modernize their data stack in 2024 in order to harness the power of GenAI, tap into data monetization, and streamline operations with a centralized primary cloud platform.
Which organizations will emerge as the victors in this pivotal moment of data modernization, however, largely depends on their ability to address crucial challenges like insufficient organizational data literacy, internal talent and resource gaps, and dependence on costly, ineffective legacy systems.
A survey of 500 data leaders from organizations with 1,000 or more employees across eight major industries categorized participants into four unique stages of data maturity, identified key focuses and themes for the years ahead, and revealed their biggest opportunities for investment.
This report differentiates between organizations based on their alignment with data best practices in order to better understand how companies are currently utilizing their data stack and how they are preparing for the future of data utilization in their respective industries.
Organizations in chaos often have no idea that they are falling behind.
Industry-leading organizations and their executives know they need to modernize their data stack in 2024. They also know they will need to enlist outside help to do so.
Nearly every organization surveyed (94%) said they will need to modernize their data stack in 2024. Among the most data-mature organizations surveyed, two-thirds said they will need to modernize a “great” deal. The majority (79%) of companies looking to modernize their data stack in 2024 said they will need a “moderate” or“large” amount of external help.
The odds are that your competition will be using GenAI in 2024, if they are not already.
While 42% of organizations said they used GenAI tools in 2023, that number is expected to jump to 85% in 2024 as more organizations plan to integrate AI tools and technologies. A majority of the most data-mature organizations are already using GenAI tools for data cleaning (69%) and automation (63%) purposes, unlike the least data-mature organizations, among whom only 28% are using GenAI tools for data cleaning and 37% currently use GenAI tools for automation.
Data literacy rates indicate need for improvement across all industries and stages of data maturity.
Based on current best data practices, only 27% of organizations surveyed demonstrated a high level of data literacy across their respective organizations. That percentage was significantly higher for the most data-mature organizations (49%) as compared to the least data-mature organizations (17%), but the numbers still show a definitive lack of high-level data literacy, even among top performers.
Organizations that have found ways to monetize their data are still ahead of the curve, but monetization is no longer a cutting-edge tactic—it’s a necessity for success.
While just 24% of the organizations we surveyed monetized their data in 2023, that figure jumps dramatically (44%) among organizations who are most aligned with data best practices. By contrast, 64% of organizations intend to monetize their data in 2024 (a 167% increase from 2023) indicating that this year, data monetization will become the norm, not the exception.
Across industries, organizations are seeing a robust ROI from their investment in data tools and technology—but the most data-mature organizations are lapping the competition.
Organizations reported an average ROI of 126% for their investments in data technologies and tools. However, the most data-mature organizations reported an average ROI of 164%, as compared to an average of 73% for the least data-mature organizations. This gap indicates that data-mature organizations may be extracting twice as much value from their data tools and technologies on average.
02.
Tracking the Data Journey From Chaos to Innovation
Chaos, Order, Insight, and Innovation.
A Chaos Organization is still at the beginning of its data journey and has yet to identify and harness the strategies, tools, and services that will optimize its data stack; meanwhile, an Innovation Organization is well into its data journey and has moved past centralizing and standardizing its data to leverage advanced capabilities like AI automation and data monetization.
The Data Innovation Journey
03.
Organizations in Chaos Don’t Know They’re In Chaos
In questions that asked respondents to self-assess, there was a pronounced break between how data leaders at Chaos organizations viewed their internal ability to execute and the need for additional investment or support to achieve their goals. While many leaders from Chaos organizations reported little need to invest in modernization, seek external help, or make significant improvements to the efficacy of their data strategy, these same leaders reported low ROI, limited goal attainment, and poor overall performance.
In contrast, Innovation organizations were much more likely than their peers to recognize areas for improvement in their processes, technologies, and resources. These orgs were more aware of the need to modernize their data stack and more willing to embrace external support in order to do so.
Consider the discrepancies between Innovation and Chaos organizations when it comes to modernizing their data stacks.
While 66% of Innovation organizations believed their data stack needed a “great deal” of modernization, only 24% of Chaos organizations answered similarly. Only 3% of Innovation organizations believed their data stack did not need to be modernized, compared to 12% of Chaos organizations.
04.
Chaos Organizations Feel Successful, But Their Outcomes Tell A Different Story
Discrepancies between self conception and actual achievement extended deep into the outcomes of organizations’ data strategies.
While 83% of organizations believed their data strategy was “moderately” to “extremely” effective in 2023, those same organizations achieved just 56% of their strategy goals in 2023—a figure that dropped to 43% among Chaos organizations.
There is a clear gap between perceived and actual effectiveness of organizations’ data strategies. Despite having confidence in their data strategies and a high perceived level of achievement, most organizations aren’t actually achieving their goals at an acceptable rate.
Data literacy is yet another arena where organizations tend to overreport and underperform.
97% of organizations believe they maintain a high level of data literacy across their organization. But what does data literacy really mean, in terms of best practices, and how many of these organizations are actually living up to their own self-assessments?
While Innovation organizations outpace their peers by a significant margin, the data show that even these organizations overestimate their org’s true level of data literacy.
The struggle to keep up with an evolving industry has created a gap between organizations’ perceived data literacy and their actual data literacy.
With limited modernization goals despite having antiquated data stacks, perhaps it’s no surprise that Chaos organizations were more likely to believe they can do it all in-house.
“For leaders at organizations with legacy data environments, this research should be a call to action. Companies that don’t prioritize modernization will risk falling even farther behind in the market in 2024, drifting toward obsolescence as they struggle to compete against their highly efficient, data-driven peers.”
Erik Duffield, CEO of Hakkoda
Chaos Organizations Feel Successful, But Their Outcomes Tell A Different Story
Just 6% of Chaos organizations indicated they would need a “large” amount of outside help from managed service providers, systems integrators, or IT consultants to successfully modernize their data stack while maintaining operations. For comparison, more than 42% of Innovation organizations reported the same.
Interestingly, 45% of C-Level executives reported their organization will need a “Large” amount of outside help in 2024 as compared to just 18% of Director-level executives, revealing a level of dissonance even amongst the executive leadership’s vision on this point.
Organizations in Chaos believe they can do it all, but data reveals that they have neither the blueprint for success nor a clear alignment on how to get there.
05.
For Most Companies, Modernization Isn’t
Over–It’s Happening Right Now
Industry Trend: Healthcare organizations will take on massive data migrations in 2024.
The era of data modernization hasn’t come and gone. In fact, the evolution from legacy technology to a centralized cloud platform is still very much in progress, with 2024 set to be a transformative year across data-driven industries.
In 2023, only 29% of organizations had centralized their data on a primary cloud platform. That number is set to increase dramatically by 2024, with 74% of organizations expecting to have their data centralized on a primary cloud platform by the end of the year.
Organizations’ plans to centralize varied significantly by industry, with healthcare lagging behind in this crucial milestone for modernization. However, healthcare organizations also report the most ambitious goals for centralizing in 2024.
Organizations Will Take on Modernization as a Major Project
While centralizing on the cloud is just one piece of modernization, organizations showed a strong investment in tackling modernization holistically, with 94% of organizations indicating that they need to modernize their data stack “somewhat” to “a great deal” in 2024.
Though modernization is a priority for most, it may not look like a total reimagining of companies’ tooling. Instead, modernization in 2024 will focus on optimizing and centralizing critical technologies that are already in place, but not yet being utilized to their full potential.
The Time to Modernize is Now: Your Bottom Line Will Thank You For It
The journey to modernization and data maturity is ultimately about achieving more organizational data goals and seeing a more robust ROI on data tech investments, as Innovation organizations can attest.
The most data-mature organizations are already achieving more of their data goals, with Innovation companies reporting that they achieved 68% of their data goals in 2023, as compared to just 43% of Chaos organizations.
By achieving more of their data goals, data-mature organizations are also getting more value out of their data, with Innovation organizations reporting an average ROI of 164% for their data tech investments, as compared to an average ROI of just 73% among Chaos organizations.
In short, staying ahead of the curve of modernization translates to greater success, both in terms of realizing data goals and generating a strong ROI on data tech investments. In 2024, more and more organizations will look to emulate this sort of success, but no matter what stage of their data journey they are in, they’ll need help to achieve it.
A Majority of Organizations Are Poised to Begin Monetizing Their Data in 2024
While organizations leading the push for modernization are enjoying a higher ROI from their data strategies, they are also unlocking new revenue streams—particularly through monetizing their data.
Nearly two-thirds of organizations reported that they expect to monetize their data in 2024, but while the hope for a monetization boom is common, the infrastructure to support it is not. Only 24% of orgs reported that they monetized their data in 2023, and even industries perfectly poised to generate good revenue from their data struggled to deliver.
$1B+ Organizations Led the Data Monetization Charge in 2023
Perhaps unsurprisingly, the big players are moving faster to tap into monetization. 29% of $1B+ organizations monetized their data in 2023, as compared to 17% of companies with less than $500M in revenue.
Organizations hoping to make the jump to the Fortune 2000 list in the near future will need to treat data monetization as critical to their growth strategy in the years ahead, as they face off with competitors already benefiting from this new revenue stream.
Despite clear evidence that modernization and the ability to monetize will be critical over the next few years, leadership alignment may prove to be the chief obstacle to success.
06.
All Roads Lead to GenAI, but They’ll Have to Pass Through Data Quality & Governance
Leadership Alignment Will Be the Achilles’ Heel for Organizations Hoping to Modernize and Start Monetizing
While 45% of Executives reported that their organization currently monetizes data, only 15% of their VP level counterparts agree. Numbers remained similarly low among Directors. These disparities indicate a rupture between strategy setters and executors around what data monetization means and how their organization is pulling it off.
The Modernization Gap Distinguishes Innovation Orgs from Their Competition
While the majority of organizations are still transitioning towards a more modernized, optimized, and centralized data stack, there’s still plenty of work to be done and pitfalls to avoid along the way. As data leaders set their own roadmap for modernization, here’s a look into the capabilities of the modernization leaders vs. the laggards.
• 47% of Innovation organizations have already centralized their data on a primary cloud platform, as compared to just 5% of Chaos Organizations.
• 44% of Innovation organizations were already monetizing their data in 2023, as compared to just 5% of Chaos Organizations.
• 69% of Innovation organizations used GenAI tools for data cleaning purposes in 2023, as compared to just 28% of Chaos Organizations.
• 63% of Innovation organizations used GenAI tools for automation purposes in 2023, as compared to just 37% of Chaos Organizations.
Though GenAI Use is On the Rise, Ensuring Data Quality & Governance Remains a Major Challenge for Most Organizations
It comes as no surprise that the use of GenAI tools is growing and will continue to grow across data-driven industries in 2024, with these tools expected to be of critical importance to organizational success in the near-future for most orgs.
42% of organizations used some form of GenAI tools in 2023, and that number is expected to rise to 85% in 2024. Meanwhile, two out of three organizations reported that GenAI will be “very” or “critically” important to their success by 2027—a number that rises to 93% among Innovation orgs .
Yet, despite increasing usage of GenAI tools, organizations still face considerable challenges when it comes to fully integrating these technologies in an efficient and effective manner. Ensuring data quality and governance was a major challenge for 46% of organizations, while applying AI to automate decision making posed a major challenge for 45% of orgs. Meanwhile, only 1% of organizations reported having at least 4 data quality measures automated, and only 24% reported having at least 3.
The Modernization Gap Distinguishes Innovation Orgs from Their Competition
Though most organizations are clear-eyed about the important role GenAI will play in their success, many orgs are in denial about what they will need to do in order to make effective use of GenAI technologies in 2024.
Only 4 in 10 organizations strongly agreed that they had the skills and expertise to support the use of GenAI tools.
In contrast, 86% of organizations are very or extremely
confident that their data team can build the
GenAI tools
capabilities needed
While most organizations remain hopeful that their internal data teams will be sufficient in making effective use of GenAI technologies, they simultaneously expressed a profound lack of confidence in the skills and expertise of those same data teams.
Organizations that are proactively looking to resolve their internal talent gaps and begin harnessing the power of GenAI immediately will naturally look to outside support, with 84% of Innovation organizations reporting that they will need a “moderate” to “large” amount of outside help in modernizing their data stack this year.
Varied Confidence in Internal Data Resources Reveals Dissonance Across Leadership Levels & Organizations
Confidence in internal data capabilities varied considerably across leadership levels, with 45% of C-Level Executives reporting that they were “extremely confident” in their organizational data team’s ability to build the GenAI capabilities they need, as compared to just 29% of Senior Directors and 28% of VPs and General Managers. This disconnect is contextualized by a broader lack of implementable use cases for most organizations.
Only 31% of organizations strongly agreed that they had defined GenAI use cases that they were ready to implement—a figure that varied dramatically between Innovation Organizations (54%) and Chaos Organizations (11%)
Most Organizations Will Need External Support to Make Effective Use of GenAI Tools in 2024
Making effective use of GenAI tools is a sign of a modernized data stack, but most organizations were not able to truly harness GenAI technology in 2023. The lack of implementable use cases and inconsistent confidence levels in the skills and expertise of internal data teams underscore what we already know to be true: the vast majority of organizations will still be pushing hard to modernize their data stack in 2024, with many looking to enlist external support in order to begin maximizing the utility of their GenAI technology.
To expedite their path to GenAI, including achieving the data quality, governance, and security measures that will make large-scale use cases possible, organizations are likely to turn to GenAI consultants, whose expertise can bridge the gap between in-house knowledge and a broad understanding of the “art of the possible” in this emerging industry.
07.
Innovators Outsource to Beat the Talent Gap, Their ROI Speaks for Itself
It’s Not a Question of Whether to Outsource Data, but How to Best Outsource Your Data
Outsourcing data management is already a key element of success for organizations who are making the most out of their data stack despite in-house talent gaps. In fact, the question for most orgs isn’t whether they are outsourcing their data but how much of their data they are currently sharing with external support, with 55% of organizations reporting that they outsource at least half of their data, and only 2% reporting that they manage all of their data in-house.
Innovation organizations reported that they were outsourcing 63% of their data on average, in contrast with Chaos organizations, who were reportedly outsourcing just 39% of their data. $1Billion+ organizations were also significantly more likely to outsource more of their data than lower-revenue orgs.
Organizations Who Outsource More of Their Data Report Higher Returns on Investments
Innovation organizations, who reportedly outsource 63% of their data, reported an average ROI of 164% when it came to data technology investments. Comparatively, Chaos organizations outsourced just 39% of their data and reported an ROI of 73% on their data tech investments.
$1Billion to $9.9Billion organizations reportedly outsourced 57% of their data and reported 153% ROI on data tech investments, in contrast with >$500M organizations, who outsourced 45% of their data and reported an ROI of 84% on their data tech investments.
The highest reported returns on investments were reported by Financial Services & Insurance (136%), Distribution & Supply Chain (130%), and Education & Government (129%). Meanwhile, Manufacturing reported both the lowest rate of data outsourcing across industries and the lowestROI on data tech investments.
08.
Conclusion: 2024 is the Year of Data Modernization—for Orgs that Know How to Get There
Outsourcing Data Pays Off When In-HouseData Teams Can’t Keep Up
Outsourcing has produced undeniable results for organizations who have chosen to make external data support a part of their playbook, as our data reveals a consistent correlation between data outsourcing and ROI in data tech and tools.
With a lack of internal expertise and inconsistent confidence levels within-house capabilities, organizations know that they need outside help to make the most of their data stack in 2024. Of the 94% of organizations reportedly seeking to modernize their data stack in 2024, 79% of those orgs believed they would need a moderate to large amount of external support.
42% of Innovation Organizations (who reported an average ROI of 164% on their data tech investments in 2023) believed their org would need a large amount of external support in 2024, as compared to just 6% of Chaos Organizations.This trend suggests that the most data-mature organizations expect to enlist more outside help with their data management in 2024, as they continue to seek out ways to maximize the returns on their investments in data tools and technologies.
The data shows that outsourcing data management is an important part of the playbook for the most successful organizations and a great way to makeup for the inevitable limitations of internal data teams who are caught up in the throes of data modernization.
The state of data has pivoted dramatically even in the last year, and there will be further shifts in the months to come, with an overwhelming majority of organizations poised to make the leap to a modern data stack.
In 2023, only 29% of organizations had centralized their data on a primary cloud platform, just 24% of organizations were able to monetize their data, and 42% of organizations made use of GenAI tools tools.
But those numbers are set to increase dramatically in 2024, as organizations prepare to make the push to data-maturity in this transformative moment. 74% of orgs expect to deploy a centralized cloud platform this year, 64% of orgs expect to monetize their data, and 85% of orgs expect to deploy GenAI tools.
To achieve these lofty goals, organizations in every industry and at every stage of data maturity will need to shore up a consensus among internal stakeholders, aligning from the C-Suite down on how their data strategies are actually performing compared their data goals and capturing a more accurate picture of both their capabilities and their org-wide data literacy.
The high volume of data quality and governance challenges reported by organizations, coupled with a low level of data literacy structured around data best practices, should serve as a wakeup call that data strategies are not as effective as perceived and that effective monetization and AI capabilities are not yet in reach.
From there, organizations will also need to reevaluate gaps between their org’s data maturity goals and their in-house talent and resources, embracing the likely realization that they may need to modernize a great deal more than previously anticipated and that, like their most innovative peers, they will need to lean on outside assistance and expertise to remain competitive.
Innovation organizations are leading the charge in modernizing, monetizing, and automating their data workloads by understanding the inevitability of internal talent gaps and relying on trusted managed service providers, systems integrators, and IT consultants to close them. And, most importantly, they are seeing more than twice the return on their data investments on average as a result.
With a majority of organizations broadcasting their intention to move to a centralized cloud platform, monetize their data, and automate with AI technologies by the end of 2024, the gulf between innovators and late arrivals is likely to deepen rapidly in the coming months.
The window of opportunity is narrowing quickly for those organizations still trailing behind in a state of data chaos, and their success will largely depend on their ability to identify and address the gaps between their data strategies and their data goals, between their modernization needs and their internal resources, and between their investments and their bottom line.
