While data management and analytics are now firmly in a new era with AI by far the main focal point of users’ interests and vendors’ product development, readiness for AI is key for organizations before they can make use of cutting-edge capabilities.
In another era, the rise of self-service analytics required enterprises to modernize data infrastructures and develop data governance frameworks that balance setting limits on access to data depending on an employees’ role while enabling their confident exploration and analysis.
Now, similarly, the era of AI requires organizations to modernize, according to Fern Halper, vice president of research at research and advisory firm TDWI. As a result, top priorities for organizations are supporting sophisticated analytics and making sure data is prepared and available for AI models and applications, according to TDWI’s research.
“Organizations are trying to get ready for AI because many of them are viewing it as an imperative for something like digital transformation, competitiveness, operational efficiency and other business drivers,” Halper said on July 10 during a virtual conference hosted by TDWI.
Ensuring readiness for developing and deploying AI models and applications is process, she continued. Included in the process are proper data preparation; operational readiness, including the sophisticated data platforms; and appropriate AI governance.
Support for AI
While technology and governance are critical aspects of AI readiness, the process of preparing for AI development and deployment begins with organizational buy-in. Those who want to use AI to surface insights and inform decisions need to get support from the executive suite that trickles down throughout the rest of the organization.
The new era of AI in data management and analytics started in November 2022 when OpenAI released ChatGPT, marking a significant improvement in generative AI capabilities.
Enterprises have long wanted to make analytics use more widespread given that data-driven decision spur growth at a higher rate than decisions made without data. However, due to the complexity of analytics data management platforms, which require coding to carry out most tasks and data literacy training to interpret outputs, analytics use has stagnated for around two decades. Only about a quarter of employees within organizations regularly use data in their workflows.
Generative AI has the potential to change that by enabling the true natural language processing that tools developed by analytics and data management vendors never could. In addition, generative AI tools can be programmed to automate repetitive tasks, which eases burdens placed on data engineers and other data experts.
As a result, many vendors have made generative AI a focus of their product development, building tools such as AI assistants that can be used in concert with an enterprise’s data to enable natural language queries and analysis. Simultaneously, many enterprises have made generative AI a focus of their own development, building models and applications that can be used to generate insights and automate tasks.
Still, getting executives to recognize the importance of generative AI sometimes takes effort, according to Halper.
“None of this works if an organization isn’t committed to it,” she said.
Commitment is an ongoing process barely two years into this new era, Halper continued, noting that a TDWI survey showed that only 10% of respondents have a defined AI strategy in place and another 20% are in the process of implementing an AI strategy. In addition, less thar half of all respondents report that their leadership is committed to investing in the necessary resources, including the people required to work with the requisite tools, such as data operations staff.
To get executive support, it takes demonstrating that existing problems that can be solved with AI capabilities and showing the potential results, such as cost savings or increased growth.
“Your organization is going to need to be made aware of what’s needed for AI,” she said. “It’s really best to understand the business problems you’re trying to solve with AI so that you can frame [the need for AI] in a way the business leaders understand. Then you can show how you’ll measure value from AI. This can take some doing, but it’s necessary to engage the business stakeholders.”
Enterprises might receive these seven benefits when using generative AI.
Models and applications trained with high quality data will deliver high quality outcomes. Models and applications trained with low-quality data will deliver low-quality outcomes. In addition, the more quality data that can be harnessed to train an AI model or application, the more accurate it will be.
As a result, structured data such as financial and transaction records that has historically informed analytics reports and dashboards is required. In addition, unstructured data such as text and images often left unused is important.
Accessing unstructured data in addition to structured data and transforming that unstructured data to make it discoverable and usable takes a modern data platform. So does combining that data with a large language model, such as ChatGPT or Google Gemini, to apply generative AI.
A 20-year-old data warehouse doesn’t have the necessary technology, which includes the compute power, to handle AI workloads. Neither does an on-premises database.
“Organizations are concerned about futureproofing their environment to handle the needs of increased data availability and workload speed and power and scalability for AI,” Halper said.
Cloud data warehouses, data lakes and data lakehouses are able to handle the data volume required to inform AI models and applications. Toward that end, spending on cloud-based deployments is increasing while spending on on-premises deployments is dropping.
But that’s just a start. The trustworthy data required for AI readiness remains a problem with less than half of those surveyed by TDWI reporting they have a trusted data foundation in place.
Automation can help, according to Halper. By using data management and analytics tools that themselves use AI to automate data preparation, organizations can improve data quality and the trustworthiness of insights.
Data ingestion, integration, pipeline development and curation are complex and labor intensive. Tools that automate those processes improve efficiency given that machines are much faster than humans. They also improve accuracy. No person or team of people can examine every data point among potentially millions for accuracy, whereas machines can be programmed to do so.
“Automation can play a key role in data mapping for accuracy, handling jobs and automating workflows,” Halper said. “Where we’re seeing most is automation and augmentation for data classification and data quality.”
For example, AI-powered tools such as data observability platforms are used to scan data pipelines to identify problem areas.
“Using these intelligent tools is important,” Halper said. “Organizations are realizing they need to look for tools that are going to help them with [data readiness for AI]. There are these tool organizations can make use of as they continue to scale their amount of data.”
Governance
Data quality and proper technology — in concert with organizational support — are still not enough on their own to guarantee an enterprise’s readiness for developing and deploying AI models and applications.
To protect organizations from potentially exposing sensitive information, violating regulations or simply taking actions without proper due diligence, guidelines must be in place to limit who can access certain AI models and applications as how those can be used.
Organizations are trying to get ready for AI because many of them are viewing it as an imperative for something like digital transformation, competitiveness, operational efficiency and other business drivers. Fern HalperVice president and senior director for advanced analytics, TDWI Research
When self-service analytics platforms were first developed, enabling business users to work with data in addition to the IT teams that historically oversaw all data management and analysis, organizations needed to develop data governance frameworks.
Those data governance frameworks, when done right, simultaneously enable confident self-service analysis and decision-making while protecting the enterprise from harm. As the use of AI models and applications — specifically generative AI applications that enable more people to engage with data — becomes more widespread within the enterprise, similar governance frameworks need to be in place for their use.
“For AI to succeed, it’s going to require governance,” Halper said.
AI requires new types of data, such as text and images. In addition, AI requires the use of various platforms, including data warehouses and lakes, vector databases that enable unstructured data discovery, and retrieval-augmented generation pipelines to train models and applications with relevant data.
Governance, therefore, encompasses diverse data and multiple environments to address AI readiness, according to Halper. Governance also must include oversight of the various types of AI, including generative AI, to determine whether outputs are toxic or incorrect as well as whether there’s bias in a model or application.
“The future starts now, and there’s a lot to think about,” Halper said. “Data management is going to continue to be a journey in terms of managing new data for AI and beyond. Organizations really need to think forward strategically and not be caught off-guard.”
Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with more than 25 years of experience. He covers analytics and data management.
