In this essay, which originally appeared in The Wall Street Journal, Daniel Yergin, Vice Chairman, IHS, examines the patterns of energy transitions and the state of energy innovation.

Energy innovation and energy "transition" are today's hot topics. President Barack Obama aims to have 20% of US electricity come from wind and solar by 2030. Presidential candidate Hillary Clinton has gone one better: A few weeks ago, she pledged that within 10 years of her taking office, there would be enough renewable electricity to power every home in America. That would certainly be a sprint, given that wind and solar now account for less than 6% of the country's electricity.

Some are more cautious about such prospects. Bill Gates recently committed $2 billion to "breakthrough" energy innovation because he is convinced that current technologies can reduce carbon dioxide emissions-and the human contribution to climate change-only at costs that he has called "beyond astronomical."

One thing is certain: Over the next few months, with the approach of December's big climate change conference in Paris-more than 190 countries are expected to attend-the discussion will grow more intense over how quickly the planet can move away from coal, oil, and natural gas and toward a low-carbon future.

Such energy transitions are nothing new. They have been going on for more than two centuries. They have been transformative and undoubtedly will be again-but if history teaches anything, it is that they don't happen fast.

The beginning of energy transitions

In 1824, a young French scientist and engineer named Sadi Carnot published a paper on "the motive force of fire." His aim was to explain the workings of an amazing half-century-old invention: James Watt's steam engine. His explanation- the "Carnot cycle"-is still taught to engineers. Carnot was convinced that this new technology was a critical factor in Britain's defeat of France in the Napoleonic wars, and he wanted to ensure that his countrymen could gain the same technological mastery.

We think of the nineteenth century as the era of coal, but as the distinguished Canadian energy economist Vaclav Smil has pointed out, coal reached only 5% of world energy supply in 1840, and it didn't get to 50% until about 1900.

The modern oil industry began in 1859, but it took more than a century for oil to eclipse coal as the world's number one source. "The most important historical lesson," Dr. Smil says, is that "energy resources require extended periods of development."

A no less important lesson is that even as newer sources overtake older ones, they also overlay them; the older hardly go away. Oil may have overtaken coal as the world's top energy source in the 1960s, but since then, global coal consumption has tripled.

Previous transitions have occurred because of new technology and applications, changing costs and prices, and concerns about energy security. Today it is climate change policy that is pushing the transition, seeking to replace lower-cost energy with what is, at least for now, higher-cost energy. The cost gap is currently being closed by a host of subsidies, incentives, and regulations and by advances in technology and manufacturing.

The "rule of energy inertia"

Two big innovations are now playing out across this new energy landscape. One of them is renewable: solar energy. The other is conventional: shale gas and shale oil. Both demonstrate what the physicist Steven Koonin, who served in the Obama administration as the Department of Energy's Under Secretary for Science, calls the "rule of energy inertia."

As he explains, "The energy system evolves much more slowly than other technology-dependent sectors" because of its "sheer scale … and its ubiquity throughout our society." Other factors, he adds, include "the amount of capital that is invested, the fact that infrastructure like power plants lasts so long, and the interconnectedness and interdependence of the whole system."

Both shale and solar provide proof for Dr. Koonin's rule. The shale revolution might seem to have burst on the energy scene almost overnight, but it was actually a long time in the making. It largely began as the conviction of one man, a Houston natural-gas producer named George P. Mitchell. In the early 1980s, Mr. Mitchell became convinced that commercial natural gas could be produced from dense shale rock. Hardly anyone believed him.

It wasn't until the late 1990s and early 2000s that the concept was proved with the successful yoking together of hydraulic fracturing (more famously known as fracking) and horizontal drilling, whose development went back to the 1980s.

Instead of the permanent shortage of natural gas that was assumed a decade ago, it is now thought that the United States holds a supply that will last more than a century. Indeed, the United States is on track to become one of the world's largest exporters of LNG.

The same new techniques have been applied to oil as to shale, with transformative results. In 2008, the fear of running out of oil was pervasive. In 2015, just seven years later, US oil production has almost doubled. This surge, combined with increasing Saudi and Iraqi supply, does much to explain the current oil price collapse.

Capturing the sun

The roots of the solar revolution go back even further, to a paper that Albert Einstein wrote in 1905 on the "photoelectric effect," for which he was awarded the Nobel Prize. It took more than a half-century for Einstein's theoretical insight to be applied. Functioning solar cells were hastily developed in the late 1950s to supply reliable electricity for US satellites in the space race with the Soviet Union, but the cells were fantastically expensive.

If solar energy was to become a practical terrestrial source of electricity, the cells needed to be cheaper-much cheaper.

One of the pioneers in that effort was a chemist named Peter Varadi. In 1973, he and fellow Hungarian refugee Joseph Lindmayer launched a company called Solarex in Rockville, Maryland.

When they started, there was hardly a market for photovoltaic (PV) cells. Then customers began to emerge, mainly for applications in remote locations, off the grid. The US Coast Guard bought solar cells to power its buoys. The oil industry did the same for offshore platforms. Illicit marijuana producers needed a lot of power for their greenhouses but also wanted to avoid getting fingered by the police because of oversize electric bills.

But it seemed as if the solar business would never reach sufficient scale. Solarex was profitable but short of capital, and Dr. Varadi and Dr. Lindmayer ended up selling it in 1983. Exxon, the other early entrant in the field, got out in 1984 because it couldn't see a significant market ahead in any reasonable time frame. By the beginning of the 1990s, The Economist was calling the solar industry "a commercial graveyard for ecologically minded dreamers." For struggling solar (and wind) entrepreneurs, the decade became known as the "valley of death."

But then, at the beginning of this century, solar came back to life. The reason was Germany. In pursuit of a low-carbon future, the country launched its Energiewende (energy transition), which provided rich subsidies for renewable electricity.

The biggest beneficiary of Germany's solar policy turned out to be not German industry, as had been expected, but China. Chinese companies rapidly built up low-cost manufacturing facilities and captured the German market, driving Q-Cells, the leading German company, into bankruptcy.

The resulting overcapacity of Chinese factories pushed down costs, as did the falling price of silicon, the raw material that goes into solar cells. As a result, the cost of a solar cell has fallen by as much as 85% since 2006. Installation costs have also come down, though not to the same extent.

With declining costs and expanding capacity, and with government subsidies and regulations, PV use has taken off.

Global sales of solar modules in 2014 were 70 times greater than in 2003. By the end of last year, the installed PV capacity added up to nearly 200 gigawatts (GW). In terms of actual generation, that matches the output of about 40 one-gigawatt nuclear reactors, since a nuclear plant produces power steadily while the output of solar panels requires daylight and varies from sunny days to cloudy ones.

Solar is compelling in hot, sunny regions. Even there, it needs backup generation for times when it cannot operate. In many other locations, however, solar isn't competitive without subsidies and incentives of one kind or another. But a great deal of effort is going into technological innovation aimed at improving the efficiency of cells and lowering installation costs. And new financing mechanisms are emerging to facilitate adoption of the technology.

"Solar is growing fantastically," says Dr. Varadi, who chronicles solar's rise in his new book, Sun Above the Horizon.

"Something like this requires time. Shale oil and shale gas had a ready market. When we started, we had no market at all, zero. And the industry had to get to mass production to bring down costs."

Over the past six years, the contribution of solar PV to global electricity has increased tenfold. It is now up to 1% of world electricity, which is about 0.2% of total world energy. Scenarios developed by IHS show it getting up to 4% to 9% of total global electricity by 2040.

Solar is growing rapidly in the United States and could account for nearly 1% of total electric generation by the end of the year. The amount of electricity from the other "new" renewable-wind-is currently much larger, almost 5% of total US electricity.

Wind gears up

Like solar, modern wind power got its start during the energy crisis of the 1970s, which led to the "California wind rush" of the early 1980s. The industry was born from the marriage of sturdy Danish wind turbines with California tax credits and energy policies.

Today, wind is on a growth path. In the past few years, manufacturing improvements and new designs have brought down costs, but wind, like solar, still needs incentives and subsidies to be competitive in most places. Wind now generates 3.5% of world electricity. According to scenarios developed by IHS, it could reach 9% to 13% of the global total by 2040.

Lithium-ion batteries-Back to 1976

What could speed up solar and wind? The development of batteries that can store renewable electricity for those times when the sun isn't shining or the wind isn't blowing. A lot of investment and effort is going into meeting the challenge.

Meanwhile, innovation with batteries is already having an impact on transportation with the reemergence of the electric car.

Here, too, there is a very long provenance. More than a century ago, Thomas Edison poured a substantial amount of his own money into trying to launch an electric car. He was absolutely convinced, he said, that "more electricity will be sold for electric vehicles than light." But in that race, he lost out to Henry Ford and his Model T.

The introduction of the Tesla Model S in 2012 was a very impressive engineering and marketing feat. But the rechargeable lithium-ion battery that powers the car was originally invented in an Exxon laboratory during the energy crisis of the 1970s, when it was thought that oil was about to run out.

The purpose, as M. Stanley Whittingham, the lead scientist on the project, wrote in the journal Science in 1976, was to develop batteries "for electric vehicle propulsion and for the storage of off-peak and solar power." But oil prices went down, and the "electrics" never arrived.

Lithium-ion batteries languished commercially until Sony began to use the technology in the 1990s to power video cameras, and then lithium-ion batteries became ubiquitous for a whole host of small portable products, including PCs and smart phones. Tesla has brought the lithium-ion battery full circle, back to its original purpose of "electric vehicle propulsion."

Energy efficiency-Different tempo

Another frontier for energy innovation gets less attention because it is less dramatic and certainly less visual: using energy more efficiently and thus using less of it. But the potential in this realm is very great.

The United States is more than two and a half times more energy efficient today than it was in the 1970s, when oil crises catapulted energy to the forefront of national politics. But there is still a great deal of slack in the system. By combining information technology, the Internet, and sophisticated monitoring and control tools, large buildings (for instance) can reduce their electricity consumption by 30% or more.

Dr. Koonin, who is now at New York University, is working on these applications. His current research is in "urban informatics"-sensing, collecting, and analyzing the enormous amount of "big data" generated by city life.

"Demand technology, whether based on informatics or better light bulbs, doesn't require massive investment or time. And it's shorter to demonstrate," says Dr. Koonin.

"One of the things you want to do in a city is make it more efficient, whether in delivery of services, the flow of traffic, picking up trash, or energy use in buildings. If you want to optimize, you need to know what is happening at a high level of granularity in terms of time and space. What's happening with traffic on 52nd Street right now? Or what is the load on the grid right now?"

What innovation will power the next revolution in human civilization? It may well be something, as Bill Gates suggests, that we can't see clearly now. But when the breakthrough occurs, the chances are that it will have been 20, 30, or even 40 years in the making-or maybe longer.

By Daniel Yergin, Vice Chairman, IHS, and author of The Quest: Energy, Security, and the Remaking of the Modern World

This article was published by S&P Global Commodity Insights and not by S&P Global Ratings, which is a separately managed division of S&P Global.