The energy industry is facing a major challenge: members of the baby boomer generation, who have for so long been responsible for the data on which firms rely, are now retiring and suitable successors are proving hard to find. The situation has highlighted the need for more efficient, effective and sustainable ways of managing data within the industry, and the search is now on for solutions.

Much of the data that energy firms consume is collected and maintained internally by specialists who have worked in the sector all their lives. A lack of new entrants into the industry during the downturn of the 1980s means there are few people with the experience necessary to replace these specialists when they retire. As a result, detailed knowledge of important energy data is at risk of being lost.

Firms realize that in order to avoid losing their competitive edge, they must digitize all remaining manual data sets and make them available for analysis by the new generation of data scientists and digital whizz kids who are emerging. The current emphasis on achieving cost efficiencies has only increased the importance of being able to thoroughly analyse data sets and derive insights from them.

As firms prepare for the changes I have described and re-evaluate their approach to data management, one of the most significant challenges they come up against is a lack of data standardization, which makes it complicated, time-consuming and expensive to work with multiple data sets.

For example, in order to build a complete understanding of a well, a firm may wish to use data from three or four external providers, as well as their own internal records. However, the lack of standardization in the way the data is represented (e.g. the use of different naming conventions for the same attributes) makes it difficult to combine and analyse. The challenge is no less acute when taking multiple sources of data from a single provider.

Of course, these issues are not unique to the energy sector. Energy firms can learn a lot from other industries, such as financial services, where a drive to increase efficiency, eliminate operational risk and unlock new insights has already led to the adoption of automated data management systems. In these industries, data management technology is being used to aggregate, validate, enrich and normalise data from different sources, creating a single, consistent version of the truth that is available to users throughout the organization.

As they lay the foundations for a new approach to data management, one of the most important lessons for energy firms to bear in mind is the importance of working with technology that is data-agnostic. This will ensure they can easily combine disparate data sets regardless of where they come from and the formats used.

To see the greatest benefits, firms should work with technology that allows them to combine traditional energy data with data sets from other sectors. For example, users can gain valuable insights if the data management platform they use allows them to co-mingle energy production data with broader financial and economic data.

In many cases, the data sets that energy firms have built up over decades are proverbial gold mines. However, antiquated data management practices mean these organizations can only scrape the surface of their valuable repositories of data. By equipping themselves with technology that enables them to analyse their complete data sets, firms can unlock the full potential of this data and successfully manage the generational changes that are taking place in the industry.

Spiros Giannaros is global head of Markit EDM, a data management platform that acquires, validates and distributes data of different kinds. He has 20 years' experience of helping firms in different industries to optimize their approach to data management.

Posted on January 12, 2017

This article was published by S&P Global Commodity Insights and not by S&P Global Ratings, which is a separately managed division of S&P Global.