Professional Space of Joe Esteves

Image Text

A Brief Evolution of Data Management: From Business Intelligence to Artificial Intelligence

(8 min read)

The first time I learned about organising and processing business data was during an "Analysis and System Design" course. It introduced the concept of designing a system from business requirements to interface, focusing on Transactional Systems. Additionally, the course taught a methodology to create a data structure for further data analysis and visualisation, applicable to Decision Support Systems, which formed the foundation of what was referred to as conventional data analytics and business intelligence before the rise of artificial intelligence in data analytics.

The Challenges of Data Analytics

In the early 2000s, conventional data analytics was groundbreaking and disruptive, but academic publications rarely mentioned terms like Business Intelligence, Business Analytics, or Big Data. By 2010, these terms had become exponentially popular (1). That year, 2.4 trillion dollar was spent on software services in the business sector. Despite this investment, only 32% of technology projects were successful, meaning 68% failed to deliver value to the organisation (2). Successful projects typically achieve this by ensuring alignment between the solutions and the business strategy. Conversely, misalignment between these can also lead to project failures.

Data analytics projects aim to deliver business value, prioritising practical solutions over perfect ones. Data scientists typically spend between 75% and 80% of their time cleaning, organising, and preparing data for analysis. Therefore, ensuring that the data is well-prepared and meets the business's needs greatly improves the chances of achieving successful outcomes from analytics projects. The results of this effort are often presented in dashboards with KPIs for performance, frequently using descriptive and diagnostic analytics (3). This process is time-consuming and challenging, especially when bridging the gap between the objectives of business users and the requirements for data visualisation.

Every company has different levels of data processing and development capabilities, and aligning these capabilities with business requirements is crucial. When there is a mismatch, there is an opportunity to enhance the company's capabilities if aligned with the business strategy. Nonetheless, data analytics is generally a daunting task, and results can become obsolete rapidly. Evolving organisational needs, strategic changes due to competition, mergers, acquisitions, and the emergence of new market players can demand new data analytics capabilities for rapid decision-making. This impacts the feasibility and longevity of data analytics projects, as dynamic organisations find it difficult to manage the cost of changes, undermining their confidence in data analytics as a reliable business solution.

The Advent of Artificial Intelligence

Artificial intelligence is not a new concept. The first neural network was created in 1950 by Marvin Minsky and Dean Edmonds at Harvard University, simulating 40 neurons on a computer (4). Nowadays, over 80% of organisations see AI as a strategic opportunity, with nearly 85% viewing it as a way to gain a competitive advantage (5).

In the early 2010s, conventional data analytics dominated the field. Nevertheless, AI's emergence was expected to make data analytics more powerful, addressing difficulties unexplained by conventional descriptive and diagnostic analytics. AI and machine learning enhanced new aspects of data analytics, such as predictive and prescriptive analytics, automating data preparation, unification, and organisation. Additionally, AI could automate some parts of the code needed for conventional data analytics projects, reducing errors and accelerating development, thus helping organisations remain competitive. By analysing historical data and current business priorities, AI can suggest actions to enhance company performance (6).

Notes
Artificial Intelligence (AI) is transforming various sectors. For example, in education, AI analyses student data, including scores and other relevant information, to suggest improvements in learning methods and automate tasks such as tracking student performance. In retail, AI improves demand forecasting and personalises product recommendations to enhance inventory management. In manufacturing, AI predicts when machines need maintenance and oversees factory operations through image recognition. In healthcare, AI assists in diagnosing conditions and allocates resources for patient care. In telecommunications, AI optimises service quality through maintenance and operational efficiencies. In banking, AI enhances customer service by efficiently routing calls, detects fraud, and customises financial assessments for credit eligibility.

Notes

Artificial Intelligence (AI) is transforming various sectors. For example, in education, AI analyses student data, including scores and other relevant information, to suggest improvements in learning methods and automate tasks such as tracking student performance. In retail, AI improves demand forecasting and personalises product recommendations to enhance inventory management. In manufacturing, AI predicts when machines need maintenance and oversees factory operations through image recognition. In healthcare, AI assists in diagnosing conditions and allocates resources for patient care. In telecommunications, AI optimises service quality through maintenance and operational efficiencies. In banking, AI enhances customer service by efficiently routing calls, detects fraud, and customises financial assessments for credit eligibility.

Barriers to AI Adoption

Despite its potential benefits, AI adoption faces several challenges. As of 2022, only 8% of companies in the EU used at least one AI technology. Early adopters of AI have applied it to stock market analysis, valuing real options, and fraud detection. Smaller companies are less likely to adopt AI. For instance, in Austria, 92% of small companies, 85% of medium-sized companies, and 74% of large companies have not yet considered using AI. Common challenges include high development costs, lack of skilled staff, management and legal risks, and ineffective data management practices (7).

These data management challenges, a barrier to AI adoption, are connected with issues in implementing conventional data analytics projects. If an organisation cannot effectively implement conventional data analytics projects and foster a data culture, it is unlikely to succeed in implementing advanced data analytics projects, where AI plays a crucial role. A strong indicator of a mature data culture is the industrialisation of conventional data analytics projects, where the company routinely reuses procedures and development components as similar cases arise. This standardises knowledge and increases production, indicating that the company has achieved a level of stability necessary for implementing advanced data analytics projects powered by AI (8).

Generative AI as part of the solution

Organisations that use third-party IT infrastructure often consume cloud solutions from providers like Amazon, Google, and Microsoft. These providers offer three categories of services: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Not surprisingly, they are integrating generative AI into their services. For example, Microsoft's 11 billion dollar investment since 2019 in an exclusive partnership with OpenAI illustrates how cloud providers are enhancing their services with generative AI.

Notes
Generative AI uses advanced techniques in natural language processing (NLP) and machine learning to create different types of content, such as text, images, audio, and synthetic data, based on prompts or questions. It can be used for many purposes, like writing articles, powering chatbots, creating art and designs, composing music, generating speech, and making synthetic data for training and testing AI models.

Notes

Generative AI uses advanced techniques in natural language processing (NLP) and machine learning to create different types of content, such as text, images, audio, and synthetic data, based on prompts or questions. It can be used for many purposes, like writing articles, powering chatbots, creating art and designs, composing music, generating speech, and making synthetic data for training and testing AI models.

Combining these services allows cloud-based enterprises to leverage IT solutions more effectively. Generative AI algorithms can interact with the cloud in a more human-like manner, making data interaction, access, and outputs more intuitive. Additionally, using the cloud infrastructure to create AI models offers benefits such as scaling computing resources up or down depending on the needs, providing flexibility and cost efficiencies. This adaptability is particularly advantageous for clients keen on using AI to improve the industries they operate in. By integrating generative AI with cloud services, organisations can optimise their operations and maintain competitiveness in a rapidly evolving technological landscape (9).

Data Preparation for enhanced Business Intelligence powered by AI

A typical AI business case often includes a clear application description, how the company will use it, the required data, and an AI expert to identify the best-fit algorithm. Pursuing an AI case incompatible with AI techniques can lead to failure, with non-existent, uncertain, or unpredictable results.

Can a company use AI technologies if it has varying data quality? Even if the data is not clean or is difficult to process, which is common in companies with emerging data cultures, AI technologies can still be valuable. Both Generative AI and machine learning models can be combined to assist in cleaning processes, standardising, and labelling unstructured data. While assessing quality after these steps can be complex, the use of AI can significantly enhance the overall data analytics efforts and improve project outcomes. This effort can increase the initial cost of AI-driven projects, especially in cleaning and organising data, which is incomplete or inconsistent (10). Compared to the conventional data analytics era, this is advantageous because AI can handle and process more varied data types. Although more unstructured and incoherent data may require additional resources and costs, and extend the time before AI projects start delivering value, the long-term benefits of leveraging AI technologies make it a worthwhile investment.

Notes
A business case highlighting the importance of data culture in one of Germany's largest utility firms demonstrates that a key factor behind the firm's AI maturity is its data-driven approach. This includes efforts to make the workforce see data as a valuable resource and the development of a digital infrastructure that supports strategic changes. These changes enable employees to participate in the data revolution as ambassadors of AI use cases that add value to the organisation. The spread of data culture within the firm has led to successful AI project implementations, supported by 3,000 rules that define good quality data, making it more visible and objective. The automation of digital and data processes is evident in three key data-driven decision support systems: real-time operations for performance stability, maintenance, replacement and repair of components, and long-term asset management. As more decisions become automated, clear responsibilities for decisions ensure accountability and organisation (10a).

Notes

A business case highlighting the importance of data culture in one of Germany's largest utility firms demonstrates that a key factor behind the firm's AI maturity is its data-driven approach. This includes efforts to make the workforce see data as a valuable resource and the development of a digital infrastructure that supports strategic changes. These changes enable employees to participate in the data revolution as ambassadors of AI use cases that add value to the organisation. The spread of data culture within the firm has led to successful AI project implementations, supported by 3,000 rules that define good quality data, making it more visible and objective. The automation of digital and data processes is evident in three key data-driven decision support systems: real-time operations for performance stability, maintenance, replacement and repair of components, and long-term asset management. As more decisions become automated, clear responsibilities for decisions ensure accountability and organisation (10a).

Conclusion

The transition from traditional data analytics to advanced data analytics powered by artificial intelligence presents significant opportunities for organisations. AI can automate many operational tasks, offering improved value and efficiency. Additionally, the emergence of generative AI models and their integration with cloud services can enable unprecedented scenarios for delivering value from data analytics projects faster and more powerfully than ever before. However, successful AI adoption still requires a robust data culture within companies. While advanced technologies can mitigate challenges associated with an underdeveloped data culture, establishing a strong data culture is crucial, even before developing AI solutions. By fostering this culture, businesses can fully leverage AI to achieve their strategic goals and remain competitive in a rapidly evolving technological environment.

Sources

(1) Tuncay Bayrak (2015), A review of Business Analytics: A Business Enabler or Another Passing Fad Consulted on 22/06/2024 URL: Link

(2) Dennis et al. (2012), System Analysis and Design

(3) Ralph Schroeder (2015), Big data business models: Challenges and opportunities Consulted on 22/06/2024 URL: Link

(4) Chandeepa Dissanayake (2021), Artificial Intelligence, a brief overview of the discipline Consulted on 22/06/2024 URL: Link

(5) Ida Merete Enholm et al. (2021), Artificial Intelligence and Business Value: a Literature Review Consulted on 22/06/2024 URL: Link

(6) Mariya Yao et al. (2018), Applied Artificial Intelligence, a handbook for business leaders

(6a) David Oyekunle et al. (2024): Digital Transformation Potential: The role of Artificial Intelligence in Business Consulted on 22/06/2024 URL: Link

(7) Rudolf Grünbichler et al. (2023), Implementation barriers of artificial intelligence in companies Consulted on 22/06/2024 URL: Link

(8) Mathieu Bérubé et al. (2021), Barriers to the Implementation of AI in Organisations: Findings from a Delphi Study Consulted on 23/06/2024 URL: Link

(9) Christophe Carugati et al. (2023), The competitive relationship between cloud computing and generative AI Consulted on 23/06/2024 URL: Link

(10) Sulaiman Abdallah Alsheibani et al. (2020), Winning AI Strategy: Six-Steps to create value from Artificial Intelligence Consulted on 23/06/2024 URL: Link

(10a) Philipp Staudt et al. (2024), How a Utility Company Established a Corporate Data Culture for Data-Driven Decision Making Consulted on 23/06/2024 URL: Link