My professional journey in data management has been a fascinating process, shaped by the different business situations I have been a part of. It is a story of continuous learning, which began not in an IT department, but in a marketing team.

The Start of My Data Story: The Early Days

Indeed, in late 2006, during the fourth year of my industrial engineering degree, I was doing an internship in a marketing department. The company needed me to learn how to get data from their transactional database of supermarkets to look at what customers were buying. The main aim was to find the best offers in order to help increase the company’s income.

This experience gave me a first look at the data environment and how data could help business teams make better decisions. The data came from two main sources: one was a relational database that stored detailed records of each customer ticket, including products, prices, and categories; the other was a separate database containing information about clients with a loyalty card. This data from the transactional system was moved to the relational database regularly. At that time, there was no automatic way to process and show this data. My role was to get parts of it from the database, put it into Excel, clean it up, use pivot tables to sort it, and finally, make charts in Excel. These charts were a form of report to show managers a clear view of sales performance and customer response. They supported data-driven marketing by helping to optimise pricing through specific, targeted loyalty card discounts, aimed at improving customer retention and increasing revenue. This work was the start of my career in data, as I became good at using Excel and also Visual Basic (VB), a programming tool, to perform all the steps of moving and preparing the data into a report.

At the same time, I was attending a university course called "Systems Analysis and Design". This course increased my interest in this area, as I learned the basics of understanding a company’s information needs and how to design a software system to support them. I also learned how to model a database using two approaches: multidimensional and relational, and how to design systems using flowchart diagrams, which are very useful for showing how data moves from one point to another. I was very excited about this area and later became a teaching assistant for the course. I am sure that my previous marketing internship helped me understand this field better, and I felt a strong connection to how important business data insights are for businesses.

The Same Methods in Different Companies

In my next roles, the technology I used remained quite similar. At an insurance company, once again as a marketing intern, I worked with transactional, claims (also known as "sinister"), and client databases. These allowed us to extract data and build models to estimate the expected income from different insurance products. We were also able to identify products whose revenue was below target, understand the possible causes, and explore whether new agreements with partners could help increase sales.

Not long after, I worked full-time at a telecommunications company during the late 2000s and early 2010s. We used a relational database to identify the services customers had and compared this data against the promotions catalogue to detect any errors proactively. This process helped us verify invoices were accurate and reduced the number of customer complaints. In this job, I got data from the system using a daily extraction job run by the IT department. I created a program using Visual Basic and a Microsoft SQL Server database to transfer and process the data every night, so we had fresh information in the morning. This program automatically checked if the offers matched the services, so we could fix any invoice before sending it to the customer. This work reduced complaints, which was a positive result for the team handling customer issues. A colleague in my team created a website using PHP, which is a programming language for building websites. This website was one of the tools customer service representatives used during calls to quickly check and correct invoice errors with customers. Combined with automatic error checks, we created an innovation that significantly improved the management of customer invoicing.

After gaining this experience, I felt it was time to grow professionally and decided to leave the company to pursue a Master of Business Administration (MBA) in Belgium. My goal was to learn about team management, operations, products, marketing, finance, accounting, and more. This broadened my knowledge of business, although the technical tools I used in my daily tasks remained the same. However, my view of IT expanded through the different business cases we studied and the professionals I worked with during the course.

Moving to Europe and a Big Step in Automation

When I moved to Europe in the mid-2010s, my first full-time job at a publishing company felt quite familiar compared to my previous experience. The company analysed sales using a transactional system, but my role focused on improving supply chain management through data. At the beginning, I used familiar tools such as Excel to process data, extract insights, create dashboards, and identify opportunities to improve the supply chain. I also developed models to predict demand.

However, I personally chose to make the processes more automatic, with the support of my team. I looked for a way to add an automatic system inside the company to get, transform, and show data that would help the different supply chain teams. Using AutoIt, a tool for automating Windows tasks, to create scripts that extracted data from the system, together with MySQL, a database system, I was able to run all the data transformation workflows automatically in the background. My new role became supervising the data jobs instead of doing manual data transformation tasks. This was a very eye-opening experience for me, and the team was pleased with the results. However, all our solutions relied on the company’s existing IT infrastructure, which limited their potential to scale. Still, the new insights we discovered proved very useful and helped the company increase sales and maintain a steady income.

My next job, in the media industry during the late 2010s and early 2020s, was a significant change. I spent the first two years working on understanding the audit media calculations and helping a global department automate these calculations using simple tools such as Excel and Visual Basic. However, the plan was always to move this process to the cloud. When the time came, the calculation methods were transferred to the cloud. This was a major shift, as it allowed me to learn from other colleagues, who were experts in cloud data management. With the company’s support, I took on a data engineering role, extracting data from digital platforms, transforming it to meet global standards, and making it available for users through visual tools or system exports.

My role grew positively. I started with audit calculations and then took responsibility for IT projects across Europe. These projects involved understanding data from various company agencies, following cloud standards and architecture, and helping teams automate media performance analysis. Initially, I developed solutions independently, but later I worked closely with other data engineers, overseeing development and managing both data and project tasks. This role continued to evolve, and eventually I led one of the company’s strategic projects, which involved handling large amounts of data from worldwide providers. The project aimed to carry out marketing research that helped agencies understand audience behaviour and measure the global impact of advertising.

The Future of Data: New Challenges

As I have gained more experience in this field, it is clear that more challenges will appear in the near future, as the IT world has been changing quickly in the last few years. For example, it is always good to reuse different parts of a system. This creates a system of parts in the cloud that works like a framework, usually for business needs that have similar solutions. But when new needs come up that are different from other projects, it is important to decide how those new parts will fit into the standard solutions. However, sometimes cloud tools that were once suitable become costly, so it is necessary to move parts to new tools. This causes migrations and can lead to unexpected work.

The Generative AI has created a new need for companies to adapt to this trend. This requires using new services compatible with Large Language Models (LLMs) and putting them into the system to first test them with Proofs of Concept (POCs) and then create projects that become part of the system. At the beginning, it can be challenging to decide which IT team should build these new systems, whether in data extraction, transformation, or visualisation. Generative AI can be used in many different layers, and the company must decide where the AI models will work and how they will connect with each other. There are also challenges with data security, making sure that access is strong and that processes control the security of the company's information to meet standards like ISO certifications. It is also important to understand the costs of the cloud and third party LLM providers to always be efficient and change tools to keep the business competitive.

A Look to the Future

Without a doubt, a cloud infrastructure has been a huge step forward for me and has many benefits compared to the old ways I used to solve problems in my career. I am very interested to see what the future holds with the growth of the cloud, new ideas from cloud providers, and the use of Generative AI in IT. But I am sure that the best approach to this change in technology is to keep learning, keep moving, keep adapting, and never stop.

Joe Esteves