Thursday, August 2, 2018

Rapid Cyclogenesis of the US Super-Snowstorm, 10-16 March 1993

Introduction


The 1993 Storm of the Century broke low pressure records and brought large amounts of snowfall and rainfall, as well as high winds, tornadoes and thunderstorms, across the East coast of America. Surface winds speeds reached 144mph in Mount Washington and 60 inches of snowfall was recorded over Mount LeConte. The cost of damage from these severe weather conditions is estimated around $6-10 billion and the storm was responsible for more than 300 deaths.

This case study examines some of the dynamical mechanisms that led to the storm’s explosive cyclogenesis, aided by figures 1-4, developed using GEMPAK.


Fig.1. Analysis at 1200 UTC 12 March 1993. (a) Mean sea level pressure is labelled in hPa and contoured every 4hPa. (b) The 300hPa geopotential height (solid orange lines) is labelled in metres and contoured every 120m. The 300hPa wind speeds is shaded at intervals of 15ms-1. The jet streaks are identified as “A” and “B”. The 300hPa divergence (solid blue lines) is labelled in 10-5s-1 at intervals of 2x10-5s-1.  (c) The 300-700hPa thickness (solid pink lines) is labelled in metres and contoured every 60m. The 500hPa absolute vorticity is shaded at intervals of 10x10-5s-1. The 500hPa positive vorticity advection by the thermal wind (solid purple lines) is labelled in 10-8mkg-1 at intervals of 3x10-5mkg-1.  (d) The 500hPa geopotential height (solid orange lines) is labelled in metres and contoured every 60m. Omega is shaded at intervals of 10x10-3cPa s-1, with the shading corresponding to upward vertical motion.

Fig.2. As in Fig.1 but for 0000 UTC 13 March 1993.

Fig.3. As in Fig.1 but for 1200 UTC 13 March 1993.

Fig.4. As in Fig.1 but for 0000 UTC 14 March 1993.

Possible storm deepening mechanisms


a) Effect of upper-level jet streak-induced circulations


Throughout the time period of 1200 UTC 12 March to 0000 UTC 14 March, the location of the low pressure system, in comparison to the location of the upper-level jet streaks A and B (noted on figures 1-4), was favourable for cyclogenesis.

At 1200 UTC 12 March, the low pressure system is located in the left jet exit region of jet streak B, particularly for the north-western side of the low pressure system. Our knowledge of the direction of the ageostrophic wind vector in the exit region of a jet streak, in association with the rapid deceleration of the geostrophic winds, tells us that there will be upper-level divergence of the air in the left jet exit region. The upper-level divergence provides a forcing for upward vertical motion throughout the column, by the mass continuity equation. Negative omega at 500hPa, a variable representative of upward vertical motion throughout the tropospheric column, is plotted in Fig.1d. The location of the large values of negative omega off the southeast coast of Texas relates well with the large upper-level divergence present in Fig.1b. In turn, the low pressure system experiences a reduction in its surface pressure and hence indicates that jet streak B is providing a cyclogenetic forcing at this time. The effect of this forcing can be observed on the plot of MSLP 12 hours later at 0000 UTC 13 March (Fig.2a). The circulation of the low pressure system has strengthened, as seen by the tightening of the isobars. The circulation has strengthened most notably in the northwest quadrant of the system, which corresponds to the location of the strongest forcing for ascent at 1200 UTC 12 March, so it is reasonable that this is where the low pressure system notably deepened.

At 0000 UTC 13 March, the cyclone is now located in the right jet entrance region of jet streak A (Fig.2b). This is another favourable position for upper-level divergence of the ageostrophic winds, due to the rapid acceleration of the geostrophic winds in the entrance region of the jet streak. Upper-level divergence suggests there is low-level convergence and hence upward vertical motion. Comparison of Fig.2d and Fig.2b shows that the strongest ascent at 500hPa is directly in the region of strongest upper-level divergence at 300hPa. This sets up a thermally direct ageostrophic circulation in the entrance region. The reduction of mass in the column that results from the ascent is favourable for continued deepening of the cyclone. The significantly greater strength and stubbiness of jet streak A at 0000 UTC 13 March compared to jet streak B at 1200 UTC 12 March - and hence much more rapid changes in speed in the respective jet entrance and exit regions - provides much larger values of upper-level divergence close to the cyclone at this time compared to the divergence values present 12 hours before. As a result, the upward vertical motion is much stronger at this later time, and so the deepening that occurs in the 12 hours after this is much more explosive. This explosive deepening causes the MSLP to drop from 992 hPa to 976 hPa at 1200 UTC 13 March (Fig.3a). The isobars are now much tighter around the cyclone – the cyclone has a much greater intensity.

Jet streak A remains extremely stubby through 1200 UTC 13 March, with great acceleration of the geostrophic winds in its entrance region, and so a thermally direct ageostrophic circulation remains. The upper-level divergence of the ageostrophic winds in the right entrance region of jet streak A is over a much greater area (Fig.3b) at this time and the surface cyclone centre is located just to the south of the right jet entrance region. However, jet streak B has now combined with jet streak A to a certain extent, and so the effect of jet streak A is not the only upper-level feature to be considered. The surface cyclone is also located in close proximity to the left exit region of jet streak B, another favourable position for upper-level ageostrophic divergence. Furthermore, the surface cyclone is downstream of an upper-level jet stream trough. This is yet another desirable location for upper-level divergence. The positioning of the surface cyclone among these three different factors results in the total upper-level divergence remaining very large over the surface cyclone. Once again, this upper-level divergence corresponds to location of large negative omega (Fig.3d). The removal of mass via upward vertical motion causes a strong deepening of the cyclone, as observed 12 hours later at 0000 UTC 14 March (Fig.4a) by the further tightening of the isobars.

b) Effect of midlevel short-wave troughs


Investigation the vorticity advection by the thermal wind around the location of the low pressure system can give insight into the rapid cyclogenesis. Regions of positive vorticity advection (PVA) by the thermal wind are directly associated with upward vertical motion, as seen using the Trenberth form of the quasigeostrophic omega equation.

At 1200 UTC 12 March, the low pressure system is located at the base of a mid-level short-wave trough, where a maximum in absolute vorticity is positioned (Fig.1c). The PVA downstream of the thermal wind is over the eastern side of the low pressure system, and so the PVA may be providing some degree of cyclogenetic forcing for the low pressure system. By comparing the location of the PVA by the thermal wind with the area of large, negative omega in Fig.1d, it is evident that they are co-located. Therefore, the PVA is likely providing a forcing for ascent at this time. Ascent is favourable for cyclonic deepening, and so this region of PVA by the thermal wind most probably caused some of the storm’s development in the following 12 hours, to 992hPa at 0000 UTC 13 March.

At 0000 UTC 13 March, there is a region of large PVA by the thermal wind over the northern side of the low pressure centre (Fig.2c), over the south coast of New Orleans. 12 hours later the surface cyclone has deepened from 992 hPa to 976 hPa (Fig.3a), as expected for strong ascent. Most notably, the circulation in the northern quadrant of the cyclone has strengthened, as seen by the tightening of the isobars 1200 UTC 13 March. This part of the cyclone is where the greatest forcing for ascent was located at 0000 UTC 13 March, so it is reasonable that this is where the cyclone notably deepened.
At 1200 UTC 13 March, there is now an area of strong PVA by the thermal wind located over the southern side of the cyclone centre and there is a corresponding region of negative omega to the south of the surface cyclone in Fig.3d.

c) Other cyclogenetic effects


Latent heat release may have played an important role in the deepening of the cyclone. It is likely that the warming produced by condensation processes will have provided a large forcing for cyclogenesis. The convectively unstable pre-storm environment would also have had a large role in the development of the storm. These effects are not analysed here.

Conclusions


The amplification and evolution of the 1993 superstorm was partly driven by two midlevel short-wave troughs merging and then interacting with a thermally direct ageostropic circulation in the entrance region of an upper level jet streak.

Research Study: The influence of Synoptic Meteorology on UK Air Quality

The weather often plays an important role during episodes of poor air quality, allowing local surface pollution emissions to accumulate, but UK air quality is also affected by meteorological processes over much greater length scales. Observations and modelling can be used to study these influences, helping to improve air quality forecasts and understanding.

Background

Air pollution has detrimental impacts on human health. Exposure to air pollution increases the risk of disease from a stroke, heart disease, lung cancer and respiratory diseases, including asthma. Poor air quality, particularly from nitrogen dioxide (NO2), ozone (O3) and particulate matter (PM), contributes to an approximate 40,000 premature deaths per year in the UK1, as well as costing UK society £16 billion annually2.

Nitrogen oxide (NO) and nitrogen dioxide (NO2), known together as nitrogen oxides (NOx), are produced from natural and anthropogenic sources, particularly by power generation, motor vehicles, fuel and solvent use, and other industrial activities.

Ozone is not emitted directly, but is formed in the atmosphere through photochemical reactions involving NOx, volatile organic compounds (VOCs) and ultraviolet radiation (sunlight).
Particulate matter (PM), or aerosols, are a suspension of tiny particles in air, each with a diameter of 0.001 – 100 µm. PM2.5 and PM10 are defined as having an aerodynamic diameter of less than 2.5 µm and 10 µm, respectively. The main species of PM are sulphates, nitrates, organics, dust, soot and sea salt.

Wind-driven atmospheric transport can allow these pollutants to move large distances away from its location of formation, which is essentially governed by the position of synoptic-scale (large-scale) pressure systems. Wind also strongly affects the vertical mixing of air. High pressure, anticyclonic
conditions are often associated with calm weather, which reduces pollution dispersion.

Methods

The interactions between meteorology and air quality is relatively well understood theoretically, but few studies have used the UK's air quality measurements network and air quality forecast models to monitor the effects of meteorology on UK air quality.

Understanding the effect of meteorology on pollution accumulation and dispersion/removal is important for improving public health, particularly through the improvement of national air quality forecasts. The AQUM is also used to inform policy decisions on emissions controls (i.e. by DEFRA), and for research into the wider effects of poor air quality.

The Met Office’s AQUM (Air Quality in the Unified Model) is operationally used to produce regional air quality forecasts, allowing air quality warnings to be delivered to the public. The model considers the emissions of pollutants, the transport and dispersion of pollutants by winds, the chemical reactions amongst reactive gases and aerosols, and the removal processes, such as wet and dry deposition.

The skill of AQUM forecasts is tested by comparison against surface observations. The Automatic Urban and Rural Network (AURN) provides measurements of pollutant concentrations on an hourly basis at approximately 200 sites across the UK. Observations from MODIS on NASA’s TERRA and AQUA satellites monitor the ambient aerosol optical depth (AOD) globally.

Data for the period 2006-2010 was used in this study and two poor air quality episodes were analysed: 18th-20th July 2006 and 26th-28th March 2007.

Results

Comparison of observations and modelling showed the ability of the Met Office’s regional air quality model (AQUM) to reproduce surface pollution concentrations, making it a practical tool for investigating the influence of synoptic meteorology on UK air quality. Some bias does exist however, with an NO2 negative bias of around 30 µgm-3 in cities (neutral away from cities) and an ozone positive bias of around 10 µgm-3 (figure 1).


Figure 1. Difference between gridded midday AQUM output and gridded midday AURN data (AQUM – AURN), during July 2006 for (a) NO2 and (b) ozone, and during March 2007 for (c) NO2 and (d) ozone.

The influence of the UK synoptic weather regime on poor UK air quality episodes was indicated by the model to be significant.

Poor air quality episode: 18th-20th July 2006

During 18th-20th July 2006, low winds and recirculating air flow over the UK and northwest Europe allowed pollution to accumulate close to UK emission sources6, particularly close to the city hotspots in southern Scotland, central England, Bristol and London (figure 2).

High pressure was present over central Europe and the UK and synoptic set up caused there to be an easterly flow over the UK and allowed heatwave conditions to build across large parts of Europe and UK during this episode.

High NO2 levels remained close to sources due to its short summer lifetime, reaching 40 µgm-3 in the city hotspots. Being a summertime episode and with high levels of ozone precursors, ozone concentrations were also high (>130 µgm-3).
Figure 2. Gridded midday AQUM output for the 18th-20th July 2006 episode, showing the absolute values for (a) NO2 and (b) ozone, and the difference from the 5-year mean for (c) NO2 and (d) ozone, with surface wind vectors overlaid.

PM2.5 had concentrations of around 25 µgm-3 across the UK. PM2.5 was greatest over the north coast of Northern Ireland (> 50 µgm-3), over the Irish sea (~40 µgm-3), and over the English Channel (>50 µgm-3). PM2.5 was slightly higher than the average over the city hotspots noted with NO2, with PM2.5 concentrations of ~30 µgm-3. PM2.5 concentrations increased over the whole UK, by 15 µgm-3 on average. The greatest difference compared to the mean was on the north coast of Northern Ireland (~45 µgm-3 greater).

Poor air quality episode: 26th-28th March 2007

During 26th-28th March 2007, a well-defined easterly synoptic regime existed across most of Europe. The air mass arriving in the UK, from the ground to 3km up (above the boundary layer), passed long distances over Europe. This allowed pollution to build as air flowed over the continent, especially over highly-industrialised areas such as the Benelux region, and caused the long-range transport of pollution towards the UK (figure 3), including NOx and PM.

Figure 3. Gridded midday AQUM output for the 26th-28th March 2007 episode, showing the absolute values for (a) NO2 and (b) ozone, and the difference from the 5-year mean for (c) NO2 and (d) ozone, with surface wind vectors overlaid.

With high levels of NO present over the UK, the breakdown of ozone dominated over ozone formation, so ozone concentrations were lower than its average for this time of year. This was particularly the case to the west of UK city hotspots, due to the easterly winds causing streaks of ozone negative anomalies to form, collocated with NOx positive anomaly streaks.

There appears to be a sharp north-south gradient in ozone concentrations, cutting through central Northern Ireland, with higher concentrations to its west (>80 µgm-3) and lower concentrations to its east (<80 µgm-3).

The north-south sharp concentration gradient is also present in the PM2.5 output, but does not extend as far north as for ozone; instead cutting across southern Scotland and down the east coast of England. Unlike for ozone, there are lower PM2.5 concentrations to the west and north of the divide (~15 µgm-3) and higher PM2.5 concentrations to the east of the divide (~40 µgm-3). PM2.5 is greater than the mean, by about 35 µgm-3 to the east of the divide and by about 10 µgm-3 to the west of the divide.

The dividing boundary between higher and lower ozone in is likely due to the presence of a front separating two different air masses. Synoptic analysis charts show this north-south front moving eastwards towards the UK. The air mass behind (to the west of) the front progress over the North Atlantic, so has very low pollution levels. Hence, PM2.5 is low behind the front, and less NO2 means less is available to photochemically breakdown to form ozone. However, there is no sharp change in NO2 along this frontal line, but this may be due to NO2 much shorter lifetime.

WHO guideline threshold pollution values

The World Health Organization (WHO) air quality guidelines provide safe limits for the key air pollutants that pose health risk, based on expert evaluation of current scientific evidence. The bias in the AQUM must be considered when comparing the pollution concentrations to the WHO limits. The WHO guideline values are for mean concentrations over a number of hours, depending on the lifetime of the pollutant; this must be taken into account when comparing the WHO thresholds to the three-day episode midday means used in this study.

Although NO2 concentrations were well within the WHO 1-hour NO2 mean threshold value of 200 µgm-3 during 18th-20th July 2006, the elevated NO2 levels are likely to have had some impact human health. Being a summertime episode and with high levels of ozone precursors, ozone concentrations were very high, reaching 130 µgm-3 in the south of England. The WHO ozone guideline value of 100 µgm-3 (8-hour mean) was therefore greatly exceeded, hence having a significant impact on human lung function. The WHO PM2.5 threshold value of 25 µgm-3 (24-hour mean) was also exceeded during this event.

NO2 concentrations were greater than 30 µgm-3 in the city hotspots during 26th-28th March 2007 - below the WHO NO2 threshold value of 200 µgm-3 (1-hour mean). The WHO ozone threshold (100 µgm-3 8-hour mean) was also not exceeded during this episode, reaching 65 µgm-3 for large parts of the UK. However, PM2.5 rose above the WHO PM2.5 threshold value of 25 µgm-3 (24-hour mean), to over 30 µgm-3 in large parts of England and Wales. Thus PM, in particular, is likely to have had a serious impact on human health during this poor air quality episode.

Conclusions

This study concludes that the processes of pollution accumulation and long-range transport have a strong influence on UK air quality under favourable synoptic meteorology. It highlights that background air quality is not just a local scale issue and is affected by European pollution sources under easterly flow regimes.

During 18th-20th July 2006, low winds and recirculating air flow over the UK and northwest Europe allowed pollution to accumulate close to UK emission sources, particularly close to large cities.
During 26th-28th March 2007, a well-defined easterly synoptic regime existed across most of Europe. This allowed pollution to build as air flowed over the continent, especially over highly-industrialised areas such as the Benelux region, and caused the long-range transport of pollution towards the UK, including NOx and PM.

WHO safe limit thresholds of NOx, ozone and PM were greatly exceeded in these two case studies.
The conclusions of this study will aid the interpretations of air quality forecasts and the effects of different weather regimes on UK air quality; for example, helping UK authorities prepare for, and help mitigate, the health impacts of poor air quality.

Limitations and future work

Analysis was limited by systematic bias of the AQUM model, causing an overestimate of the absolute values of ozone and an underestimate of the absolute values of NO2. This affects the assessment of the individual impacts of the pollutants. The multiple factors affecting air quality results in it being extremely difficult to separate effects of synoptic meteorological processes on air quality from other factors, including the effect of local emissions and local meteorology, vertical atmospheric mixing and pollution diurnal cycles.

This study has multiple branches for future work. Most importantly, it would be interesting to determine why the NO2 positive anomaly streaks and ozone negative anomaly streaks seen over the Irish sea (figure 2 & 3), westwards of the city hotspots of Bristol, Liverpool and Glasgow, are not present downstream of other big cities.

Key references

1. Holgate, S.T. 2017. ‘Every breath we take: the lifelong impact of air pollution’ – a call for action. Clinical Medicine. 17(1), pp.8–12.
2. DEFRA. 2010. Valuing the Overall Impacts of Air Pollution. [Online]. [Accessed 02 February 2017]. Available from: archive.defra.gov.uk.
3. Jones, A.M., Harrison, R.M., Baker, J., 2010. The wind speed dependence of the concentrations of airborne particulate matter and NOx. Atmospheric Environment. 44, pp.1682–1690.
4. Pope, R.J., Butt, E.W., Chipperfield, M.P., Doherty, R.M., Fenech, S., Schmidt, A., Arnold, S.R. and Savage, N.H. 2016. The impact of synoptic weather on UK surface ozone and implications for premature mortality. Environmental Research Letters. 11(12), p.124004.
5. Savage, N.H., Agnew, P., Davis, L.S., Ordónez, C., Thorpe, R., Johnson, C.E., O’Connor, F.M. and Dalvi, M. 2013. Air quality modelling using the Met Office unified model (AQUM OS24-26): model description and initial evaluation. Geosci.Model Dev. 6, pp.353–72.
6. Prior, J., and Beswick, M. 2007. The record-breaking heat and sunshine of July 2006. Weather. 62, pp.174–182.
7. World Health Organization, 2006. Air quality guidelines: global update 2005: particulate matter, ozone, nitrogen dioxide, and sulfur dioxide. World Health Organization.
8. Camalier, L., Cox, W. and Dolwick, P. 2007. The effects of meteorology on ozone in urban areas and their use in assessing ozone trends. Atmos. Environ. 41, pp.7127–37.

Can Renewable Energy and Machine Learning Enable an Energy Independent India?


These are high-times for Artificial Intelligence (AI), specifically its Machine Learning (ML) form. Both are increasingly coming into the public consciousness, as they are applied to an array of sectors. Applications in the Power sector, though less visible than say physical robotics, are the most fundamental, especially in an Indian context, as its evolution will also transform many other sectors.

As a tropical country we are endowed with abundant sunshine and wind, but still import much of our energy in the form of oil and coal. The current government is pushing to transition us to a ‘Solar Nation’, in no small part to reduce the national import bill. Achieving this could accelerate the Make in India campaign, because national energy independence will mean the cheapest power possible. The vision is for a 100% renewables-powered grid, giving 100% reliability of supply, enabling prices to be a fraction of what they are today.

Technically India already has ample power, with approximately 300GW of installed capacity, and only 150GW of peak demand. However, this is not a practical reality. Rather, it reflects the challenges of managing supply and demand. Our per capita consumption remains low because most people still have only a limited access, in terms of connectivity and availability. Further, though abundant, Solar and Wind are inherently variable, and will exacerbate grid-volatility as integration increases. Even with the small penetration of renewable generation today, the intermittency has caused a significant rise in balancing costs, which in the future could be huge.

Added into the mix, the future will see increasing Distributed Energy - both localised green generation, and storage. Though in some instances this will take pressure off the central grid, for example by reducing the load on transmission infrastructure, overall it will increase volatility. Today generation and consumption locations are essentially fixed, we at least know where they occur, even if not quite what volumes and when. But we have little idea of the future profile, especially with Electric-Mobility coming into the frame.

It is already difficult for humans to manage this complexity, but these challenges aren't dire portents for the sector. What we need is a modern approach to address key deficiencies, by analysing disparate real-time data for optimal decision-making. This must be cost-effective, and of all available avenues software will always be much cheaper and more efficient than physical buildout.

This is where AI & ML excel, with by far the greatest benefit-to-cost ratio. Crunching the exponentially growing volumes of systems data required to balance gird-volatility, forecast future trends, optimise existing assets and plan future infrastructure. They are already handling these issues in the real-world, from forecasting of solar generation to prices on the power markets, and even detection of power-theft.

In a renewable and distributed energy future, AI & ML can manage key aspects of the energy system, with human-oversight replacing human-intervention for the highest decision levels. It thus has the capability to help us realise the vision of an energy-independent India, with affordable power for all.


Credit to Niladri Roy for co-writing this article.

Human vs Machine Learning

In previous articles I've written quite a bit about Artificial Intelligence and Machine Learning. This article hopes to explain these terms in fine detail and compares them to the workings of the human brain.

Learning Machines - Not So New

Though the terms Artificial Intelligence (AI) and now Machine Learning (ML), have become widely vaunted for an array of areas in industry and life, they conceptually have a very long history, being prevalent in prior decades and even centuries. We can actually go back several millennia, to the ancient world, for the first instances. 

There have been examples of ‘intelligent’ machines peppered throughout ancient mythology, and even an allegedly real one in ancient China. Given there is still limited understanding of what AI and ML actually are, examples from myth can help with differentiating and understanding both concepts, and ultimately explore ML in more detail.
The Automatons of Hellenistic Greek mythology, were metal statues crafted by Hephaestus, the smith-god of the Olympian deities. The best of them could think and feel just as humans do, like the Kourai Khryseai - ‘golden maidens’. This is the most common perception today of AI (just without the moving limbs, or heart and charm).

But more usually, the Automatons were only able to think, feel and ‘learn’ within the boundaries of what they were designed for. Like those created to assist Hephaestus in his weapons forge up on Mount Olympus. Or Talos, the giant who patrolled the shores of Crete to defend against attacking fleets. Or the Keledones, which were also golden maidens, but only able to sing and learn new songs and music. All were excellent at doing and learning about their respective tasks (or datasets as it were), but not much else. So the Automatons are pretty apt early examples of what AI, and particularly ML actually are.

What Are Artificial Intelligence and Machine Learning?

In simple terms, Artificial Intelligence is the broad notion of machines being able to perform intellectual tasks in a manner similar to human cognition. This is usually imagined to be the much conjectured ‘technological singularity’, where human cognitive capabilities are surpassed, possibly resulting in drastic and dire existentialist threats to humanity (so say messers Hawking, Musk, Gates, and co anyway). This concept is also known as ‘Artificial General Intelligence’ (AGI), which though of course a future possibility - whether malevolent, benevolent, or indifferent - is far removed from current reality. One reason being that human intelligence is not singular in nature, nor is the physical brain yet fully understood.

Figure 1. Types of human intelligence.

Machine Learning is a subset of AI, based on giving machines access to datasets, then letting them ‘learn’ from these. Specific problems are solved without the need for explicit programming, by feeding in data and training the system to derive information out of them. Once trained, any new data given to a machine will yield desired outputs - solutions to the initial problem. Starting from this basic and well-defined premise, better insights can be gained as to how a machine learns.

Current (mis)Perceptions

There is very varied understanding of AI and ML. Both are frequently misperceived as interchangeable, adding to the confusion. AI is indeed a rather nebulous term, but ML is a well-defined subset of it with tangible applications. It’s really ML that is making great strides in an ever more diverse range of fields.

However, because of the misperceived interchangeability, customers – in the energy sector and more widely - expect ‘AI’ to deliver everything now. Customers are often unaware of the numerous remaining incremental steps that are necessary to achieve their respective visions. As with all technology, and indeed life, development takes time, an often frustrating universal truth.

Supervised, Unsupervised, and Reinforcement Learning

Machine learning is commonly split into three main types: supervised, unsupervised, and reinforcement learning. A simple example of supervised-learning is having labels for sorting colours, or animals, whereby the machine is supervised (by a human) in how it categorises these. In unsupervised learning, the machine comes up with its own labels and logical groupings. So just like the mythological Automatons, a supervised machine can adapt and learn within the parameters it is given. However, it can only expand its range, by reprogramming through external intervention - from Hephaestus or a human respectively.

Also in unsupervised learning, whereas a human may use groupings such as age for classification, a machine may form entirely different classifications from the same dataset that are otherwise unapparent to a human. In reinforcement learning, the algorithm tries to find the best way to earn the greatest reward. These now lead to to a more dynamic and interesting space, where the lines get blurred a little.

Deep Learning

Deep Learning is a subset of Machine Learning, which is closer to human learning - give a machine massive amounts of data and let it come up with something by itself. This can be both supervised and unsupervised, so some real projects dedicated to deep learning from Google Brain’s DeepMind, can illustrate these.  

Neural networks are designed with the purpose of modelling or mimicking the human brain, better than previous approaches. One such network was given access to unlabelled YouTube images, and nothing else, to see what it came up with. Eventually, the machine learned to identify….cats! Machine image recognition is quite analogous to the human visual cortex and neurological processing, so has been well replicated. Though humans can still pick a face out in a crowd more easily, machine image recognition has made great strides recently with the advent of convolutional neural networks, which are specifically designed for shift-invariant object recognition in images.

Figure 2. Convolutional neural network for image recognition.

Of more interest and complexity is AlphaGo. The game of Go was invented in China over 2,500 years ago, and is believed to be the oldest board game still being played today. It has simple rules but has long been regarded as a difficult challenge within the field of AI and ML. Some of the complexity is attributed to the sheer number of possible game variations, which are on a scale similar to the number of atoms in the universe. Building a system that masters this game requires considerably more components that mimic human thought processes than other games, such as chess.

The AlphaGo neural network was given no rules, just historical games and datasets. The network was able to incrementally learn the rules, or ‘policies’, through self-simulated games, and what the best moves were to defeat a human. Millions of games were simulated, with high scores used to justify, or gradually reinforce its learned behaviour. So in very similar manner to a human, it post-rationalised, learning tactics to get high scores, which reinforced its behaviour.

Previously, it was generally believed that a decade or more would need to pass before a machine could defeat a professional player. Yet in 2016, a 5-match human vs machine contest, between one of the best professional human players and AlphaGo, saw the machine unexpectedly triumph 4-1. This was through finding ways of winning which ran contrary to professional convention, and has actually redefined the way the game will be played in the future by humans.

While the original AlphaGo needed the expertise of humans to get to such a level, the latest evolution of the program, AlphaGo Zero, has been able to consistently demonstrate superhuman performances well exceeding its predecessor. All without any human inputs or historical datasets, by knowing only the rules of the game and by playing games with itself. This is definitely a significant achievement in both ML, and wider AI.

Explicability, Predictability, and Volatility 

Despite these strides, there is still no way to understand why machines make any of these decisions. Nor can we really get any insight either, given the inherent black box nature of these algorithms. This lack of explicability is significant from a human perspective. Humans still always look for explicability and predictability - formulas for ‘what’s going on behind the scenes’. Classical ML techniques which were derived in the 80’s, are all statistical in nature – support-vector-machines (SVM), tree-based, or knowledge-based algorithms. In other words, they can be grasped by (capable) human minds. It is very challenging for humans to accept anything otherwise. Even with all the volatility and uncertainty around us in daily life, humans are neurological deeply hardwired to search for patterns and predictability.

Yet almost dichotomously, the human mind also functions in a very similar ‘inexplicable’ way to these black-box machines. What is called ‘gut’ feeling or instinct, is simply the brain making superspeed assessments based on a lifetime of data points and bypassing conscious consideration. We can’t explain it, but we do trust it without hesitation, even when we can’t fathom the pattern. So why not the same for machines? Machines can and have been proven to see patterns that humans can’t – patterns that sit outside of classical information theories and conventional explicability.

As the share of renewable generation on the grid increases, so will grid-volatility. So ‘inexplicable’ ML that can process patterns at super-speeds far beyond what humans are capable of, is exactly what will be needed. ML is an essential tool to manage extreme grid volatility, through forecasting of weather and demand based on past behaviour and data points. ML doesn’t need to be explicable to be useful, and its applications in forecasting within the context of energy specifically, will be delved into more deeply in the follow up article. This will include the 5-to-10-year incremental journey from reality today to where customers think capability should be.
 Figure 3. The Turing Test

The Big Question

With ‘creativity’ and learning in mind, can machines mimic that so cherished of human traits, the one held up as the highest of virtues and only possible with a ‘soul’? Well let’s flip the question around - how unique is human creativity really? When analysed, much of what humans produce is surprisingly repetitive. Take Art, even with its seemingly endless variety, it all still falls within set bounds - for a machine with enough processing power. There are already examples of computers painting and drawing pictures, which are indistinguishable from those by a human hand and heart. Then there’s music, which has wide and ubiquitous commonalities across genres and cultures.

Human creativity then maybe isn’t that special, and of course “all art is quite useless”, as Oscar Wilde famously said. However, there is another dimension that should be separated out. The preceding examples are all about production, rather than interpretation, and understanding ‘why’ humans react to art is still beyond machines (and it should be noted, many humans too). Another aspect of human creativity is the ability to do research, design and innovation - the highest level of insight, generated from a combination of multiple cognitive faculties, or modules. This is the ultimate ability of the human brain, which is what AGI aims to achieve. Currently, there is narrow AI within bounded modules, but no sense yet of how to plug them together. Though raw computational power has already far surpassed human cognitive capacity, it still doesn’t enable multidimensional integration of these modules. Development is certainly far from anything that could pass the Turing Test. This experiment was hypothesised by arguably the Godfather of ML, Alan Turing, to assess a machine's ability to exhibit intelligent behaviour equivalent to, or indistinguishable from a human.

Conclusion

Hence true machine creativity is still far from possible yet - a neural network will not learn to paint then suddenly decide to become a sculptor unprompted, nor write a new company energy policy. So, is human learning better than machine learning? The ideal scenario is to combine both - ML’s ability to crunch enormous volumes of data, with the still uniquely human ability of creativity, to achieve the highest levels of decision making.


Credit to my colleagues Niladri Roy, Rohan Nongpiur, Sho Akama and Aseem Khanduja for co-writing this article.

Harnessing the Wind and Sun to Achieve an Energy Surplus Future

Imagine a future where a 100% renewables-powered electricity grid harnesses the abundance of energy beamed down on us daily, to give 100% reliability of supply. Enabling energy prices to be a fraction of what they are today, or perhaps even universally free. Let’s consider two paths of thought, one into the far future, the other more nearer-term for India, and rather imperative.

Looking to history to inform our future-gazing, the indications are that we’ll find ever newer ways to consume energy. So we may take indeterminate time to reach ‘surplus’ in the strictest sense, though that does not preclude universal free access. Even if we developed the ability to absorb all the energy emitted by a star - perhaps through the hypothesised Dyson Sphere - and become a ‘Type 1 civilisation’. Tech advancement will likely hog whatever we produce. More likely, we’ll develop ships that are electric instead of solid-fuel run, and hyperloops to replace trains. Energy is like money, we will always find ways to consume it - our way of life will just change to match.

Today however, India already has an energy surplus, with approximately 300GW of installed capacity, and only 150GW of peak demand. But this is a technical rather than practical reality. There is still load shedding in most parts of the country after all, and our per capita consumption remains low because most people still have only limited access, in term of both connectivity and availability of power. Further, India is a tropical country blessed with ample sunshine and wind, even in the monsoon months. Yet at present, India still imports much of its energy from abroad, notably in the form of oil and coal. The government is convincingly pushing for renewables, and wants to transition us to a ‘Solar Nation’, in no small part to reduce the national import bill. Achieving this could accelerate the Make in India campaign, because national energy-independence will mean the cheapest energy possible.

Once India reaches national energy independence, another equally fundamental development and national health challenge that can be addressed, is the stark water distress. Cheap energy will allow unlimited pumping of water across immense distances, enabling this demon to finally be dispelled. Nascent water desalination technologies - which already exist, but are hamstrung by huge scaling costs - will then take off along the coast. This will enable pumping of clean water to the northern inlands, and onwards to other countries in similar need.

Technologies such as distributed energy generation and storage will take some pressure off the future central electricity grid - for example by reducing the load on transmission infrastructure - there can be no such ‘distributed desalination’ equivalent for water. The oceans are fixed, so plants must remain coastal, or possibly at some suitable inland locations, such as Sambhar lake in Rajasthan. This will mean building significant new pumping infrastructure, with massive energy demands to fully service the country,

Thus, a prerequisite for clean water is vast amounts of cheap energy, and this can only be possible from solar and wind, which are inherently variable, so will increase grid-volatility. Today, even with just a small penetration of renewable generation, the intermittency has caused a significant rise in balancing costs, and in the future this could be huge.

Given this, we must identify solutions with the highest benefit-to-cost ratio across the energy-ecosystem, and software will always be cheaper and less cumbersome than physical buildout. This is where artificial intelligence and machine learning (AI & ML) come into frame and excel - crunching the exponentially growing volumes of data required to balance gird-volatility, forecast future trends, optimise existing assets, and plan future infrastructure. We are already seeing AI & ML starting to handle these issues in the real-world, from forecasting of generation and demand, to scheduling of supply. This bodes well for the future water-energy nexus.

So, there are a multitude of ways we can imagine using an envisaged future energy surplus, but solving clean water access for all, with the help of AI & ML, seems to be the one that will most benefit the country and its people.


Credit to Niladri Roy for co-writing this article.

Thursday, June 28, 2018

Aggregated Weather Forecasts for Large Load Areas


Introduction

When checking the weather forecast in the mornings to see whether you’ll need an umbrella that day, you’ll probably be looking at the forecast for your current point-location, giving data specific to your neighbourhood. If you’re planning a weekend away, the forecast display can easily be changed to another specific point-location. But what if your business need to know what is happening across an area much greater than what can be represented by a single point location? For large areas, obtaining weather forecasts from point-locations is unlikely to suffice and may lead to strategic errors for the business, and ultimately, financial losses.

One example of where aggregated weather forecasts are required is in the energy industry. Electricity demand (load) is strongly dependent on the weather conditions, and so weather data is a vital component of electricity load forecasting. Aggregate-level load forecasts must consider the weather conditions across the entire area, as opposed to using just one location to represent the weather conditions everywhere. This article aims to address the challenges of weather location selection and proposes the best methodologies for determining how many, and which, weather locations should be used when forecasting load over large areas.
The Indian state of Uttar Pradesh, India’s most populous state, is taken as an example case study. Uttar Pradesh covers 243,290 km2, equalling the size of the United Kingdom. Weather therefore varies significantly across the state – there are higher altitude areas in the north, and more rainfall occurs in the north and eastern areas. As a result, the impact of weather on the state’s load will also vary significantly across the area.
Uttar Pradesh - India’s most populous stare.

In particular, localised extreme weather events can be hugely problematic. For example, there could be heavy rainfall in Agra (west Uttar Pradesh), but sunny skies in Lucknow (central Uttar Pradesh). If the weather data is only received for Lucknow, the forecasting system would have no knowledge of the rainfall elsewhere.
(Include a relevant radar image once available)
The geography and climatology varies across Uttar Pradesh.

Methodology

In order to give an aggregated state load forecast, a virtual weather station must be formed, which best represents the weather across the entire state.
Initially, the weather forecast locations you do choose must have data availability, reliability and accuracy. All weather stations that do not, must be ignored. Let’s assume that we begin with 20 weather station locations across Uttar Pradesh that have reliable and accurate data, and provide a good geographical spread across the state.
The virtual weather station can then be built by using multiple point-locations. But how many point locations should be used? Too little will not represent the area well enough, but too many point locations may actually increase the forecast error and increases the amount of data, which has its own issues.
Once you’ve decided how many point locations to use, how do you decide which point locations should be used? Do you choose a roughly even spread of locations across the entire area? Or do you choose more in the west because population is higher there? Ultimately, you want to represent the impact of weather on load, not just the average weather of the state. This will vary with population density, economic and demographic diversity, as well as the end-use diversity of the residential consumers.
Another challenge is that weather forecasts are rarely perfect. Inaccuracies are lower the closer in time we are to the forecast time, but will often still contain significant errors. To rectify this, real-time observational data can be used to compare the forecasted values with the actual observed values and adjust the weather forecasts accordingly. This is particularly effective during days of extreme weather, because the forecasting system is able to quickly react to sudden weather changes.

Methods for aggregated weather forecasts


a)      Simple Averaging

The first method simply takes an average of every forecast variable, from all 20 weather locations. Although this is a simple method, it can be very effective at capturing the impact of weather on load across a large area, as shown by multiple studies (Lloyd 2014). However, there are of course issues to using this simple average method. Firstly, averaging may actually reduce accuracy - many studies have shown that sometimes you can use too many weather locations and cause unwanted smoothing of the data. Additionally, taking the average fails to deal with:
·         the effect of geographical diversity – differing weather across the state;
·         demographic diversity – differing economic and classes across the state;
·         end-use diversity – differing efficiencies of appliances, difference uses of electricity, etc.
Also, why 20 weather locations? Why not 5 or 10, or 100? This number has been heuristically (randomly) picked and is unlikely to be the optimal number of weather locations to use.

b)      Best-fit to Load

In this second method, we progress on the simple averaging technique, by building a load model for every weather location (20 in this case). The accuracy of each model is then tested against actual load and the individual models are ranked in order of their accuracies. Next, the weather data from the top 5 best-fit models are combined, perhaps through simple averaging. This method only uses weather locations that have a strong impact on load, and so reduces the chance of errors creeping in from unimportant weather locations (Charlton and Singleton, 2014). But how do we know if these 5 models have captured the effect of geographical diversity, economic diversity or end-use diversity? And why the 5 best models and not the top 3, top 7 or top 15 models? Again, these numbers have just been heuristically selected.

c)      Zonal best-fit to Load

The third method is very similar to one above, but this time we select the best model for each ‘zone’ (Nedellec et al., 2014). Zones are defined by the operators, covering a small, defined territory within the greater load area. Again, a separate load model is built, one per weather station location. The accuracy of each model is then tested against the actual load. Next, within each zone, the weather station that achieved the highest load accuracy is used. All other weather stations within that zone are ignored. Finally, the average is taken of all the zonal weather stations.
This deals with the geographical diversity problem – all zones are represented. But as a result, it may neglect the issue of capturing the population, economic or end-use diversities. And again, the number of zones and the initial number of weather locations is randomly decided.

The state can be split into zones.

d)      Ranking and Rating

All three of the above methods have heuristically selected the number of weather stations to use; they do not answer the question “How many weather locations should be used?” The Ranking and Rating method attempts to answer this question (Hong & Wang et al., 2015).
A load model is built for each individual weather location. These loads are then compared against the actual aggregate load and ranked in order of their load error, in terms of the mean average percentage error (MAPE) or root mean squared error (RMSE). Obviously, the highest ranked model contains the weather station that has the highest impact on the load. The next step then combines the weather data from x number of the highest performing load models, where x is a number ranging from 2 to the total number of locations; e.g. for x = 2, 3, 4, 5, etc. Again, we fit the virtual weather station of each combination scenario with load data, and arrange in ascending order of error (MAPE or RMSE). The virtual weather station that produces the lowest error is therefore the combination of weather stations locations to use.

Combining weather stations – weighted means

How should the weather locations be combined? A simple average may be effective, but it may be more effective to apply a weighted mean. This weighting could be by population density, giving greater influence to weather in areas with higher populations. This is often used in developed countries, where population correlates well with local electricity demand, but this may not be the case in developing countries. For example in India, highly populated areas tend to be poorer areas with less energy-use per capita, and so population density does not correlate well with local electricity demand. Therefore, a different type of weighting should be applied, such as the economic density of the area. This may correlate better than population density for developing countries, because economically powerful areas tends to be co-located with areas of high residential demand.
The population density of Uttar Pradesh, India.

An important point to make here is that weather insensitive loads should be taken out of the equation. Weather-insensitive loads are sources of load that are not affected by the weather. These are mostly from industrial sources.
Weather insensitive load centres, like this car factory, must be ignored when applying the Rating and Ranking method.

Conclusions

There are multiple methods for producing an aggregated weather forecast, applicable to electricity demand forecasting and management for large load areas. The best of these is the Rating and Ranking method, which ranks the weather station locations by their load error when inputted into a single-station load model. This answers the question “what weather station locations should be used?” Next, the question of “how many weather station locations should be used?” is answered by combining the weather data from x number of the highest performing load models. The virtual weather station that produced the lowest error is therefore the combination of weather stations locations to use.
These weather stations can be combined using a simple average or a weighted mean. It is suggested that a weighted mean by population density is suitable in developed countries, but a weighted mean by economic density is more suitable in developing countries.

References

Charlton, N., & Singleton, C. (2014). A refined parametric model for short term load forecasting. International Journal of Forecasting, 30(2), 364–368.
Hong, T., Wang, P. and White, L., 2015. Weather station selection for electric load forecasting. International Journal of Forecasting, 31(2), pp.286-295.
Lloyd, J. R. (2014). GEFCom2012 hierarchical load forecasting: Gradient boosting machines and Gaussian processes. International Journal of Forecasting, 30(2), 369–374.
Nedellec, R., Cugliari, J., & Goude, Y. (2014). GEFCom2012: Electric load forecasting and backcasting with semi-parametric models. International Journal of Forecasting, 30(2), 440–446.

Sunday, May 6, 2018

How Machines Can Learn to ‘See’ the Weather and Enhance Future Energy Management



Sight or Intuition?

How do we understand the world around us? That is a surprisingly complex question. Let’s try to explain using a real-life example. In the past, whilst enjoying a merry late night meander back from the bar, you may have experienced being barked at, or even chased by a dog.

It was obviously a dog, but did you know what kind of dog? German Shepherd, Labrador, Beagle? In that situation it didn’t matter, it was just a ‘dog’, a barking one with sharp teeth.

But how was that understood? For humans, there is a degree of intuition. Aside from the barking and teeth, it is because we all share a common abstract image of 'dog'. The internal image is likely to vary from person to person. If you grew up with a Golden Retriever, your internal ‘dog’ will look more like that, with some positive emotional attachment. And if you grew up with a Rottweiler snarling at you, your image will be one with scary teeth, and some frightful memories.

Perhaps surprisingly, this is not so dissimilar to how a visual processing AI 'sees' the world.
A Machine's Brain

Having established how humans understand the world in an abstract sense - by having representations inside our brains - enables us to start conceptually understanding how a machine ‘sees’. It obtains its own internal image of reality, which is in a sense the mission of Machine Learning (ML), to obtain a representation of whatever we want it to learn, but in a systematic way. We can encourage machines to form their own internal image of reality using ML, by giving them a mathematical foundation from which they can construct these.

So if we have a picture of a dog, how can we enable a machine to learn how to see that dog?
Figure 1: Dog - a friendly, drooling one (Image credit: Wikimedia Commons)
Figure 1 shows just one dog. But there are hundreds of different breeds, and a machine cannot obtain a general image of ‘dog’ from just looking at a single example. Figure 2 shows several dogs, and we could assemble thousands. Each a different type and in different poses.
Figure 2: Selection of different dogs (Image credit: Wikimedia Commons)
They are simple enough to differentiate. But, now squint your eyes, and you’ll see something like Figure 3.
Figure 3: Blurred dogs - but still identifiable by a human (Image credit: Wikimedia Commons)
As a human, you can probably still tell them apart, but don’t those dogs now look at least slightly similar to each other? Certainly more so than the clear pictures (Figure 2). All we did was decrease the resolution of the image (known as down-sampling). More precisely, we picked a group of pixels and took their average (Figure 4), then repeated for all areas.
Figure 4: The operation of blurring (Image credit: Wikimedia Commons and Sho Akama Images)
Now imagine if we apply the same process to all the picture several times. They will look increasingly similar. We might then only be able to obtain abstract images of ‘dog’, containing some distinct features, like the shape of its head and body, its saucer bowl eyes, wet button nose, drooling tongue, floppy ears, and maybe sharp white teeth. But whereas human intuition can still make sense of these incomplete datasets, machines typically cannot, they lack that intuition.

Convolutional Neural Networks

We all see the world through our own personal 'lens'. Elderly people tend to have slightly blurrier eyes than young people in a physical sense. More conceptually, a politician has a better lens to understand parliamentary dynamics. And a physicist has a lens to see the invisible laws of the universe.

But what does the sight lens of a machine look like? Within the ML domain, this lens is called a filter (or kernel). For any computer, an image is a 2-dimensional array with numbers. It cannot recognise a complex object like a dog in one go. It starts from understanding edges and lines, and by stacking different filters, it can eventually ‘see’ a dog. An ML model specifically designed for computer vision is called a Convolutional Neural Network (CNN).

Traditionally the set of filters is handcrafted by specific ML-area experts. For a human face, you may want to consider a filter that can detect the general geometric shape of faces, like the ‘T-line’ between the nose and eyes (Figure 5).
Figure 5: T-Line, on just some random guy (Image credit: Wikimedia Commons)
To identify a chameleon, you need a very different set of filters than to identify a human (maybe less so if they are of the Karma variety). This means every time you want to detect something new, you need a group of experts for R&D. Imagine trying to develop the best filters for a chameleon. What would they be? To detect rounded edges and stack those up to find two big eyeballs? Why not a filter to find their signature long tongue? Now how are those filters different from the filters for a human? Can our filter distinguish the two creatures in figure 6?
Figure 6: Chameleon vs human (......probably) (Image credit: Wikimedia Commons)
So there are a seemingly infinite number of possibilities, and even with years of research, it is just not practicable to obtain a ‘best’ set of filters. This is where CNN enters the frame and excels - a machine capable of learning filters relative to given objects.
Figure 7: How a Convolutional Neural Network recognises an object (Image credit: Wikimedia Commons)
In Figure 7, each blue block is applying a filter, then extracting core information relative to that filter (also known as ‘convolution’). The size of subsequent blocks decreases because each time a filter is applied, the resulting image shrinks.

Each sequential group of blue blocks is called a layer. By exploiting sets and combinations of filters, a CNN can recognise complex shapes, like dogs (or chameleons).

CNN for Weather Prediction and Better Energy Management

So now we have understood CNN and how a machine can ‘see’. But the link between recognition of a dog and Energy management probably still feels very abstract.

Rather than jump to how machines can ‘see’ and interpret weather patterns to make forecasts. Let's also consider how applications to forecasting are very different to simple object recognition. There are some important future possibilities that could provide compelling solutions to energy industry problems, but it is important to first smooth that gap of understanding.

ML-enhanced visual recognition can be used to recognise, and more crucially interpret, specific visual weather patterns, in much the same way that they can learn how to recognise dogs. Then, these machines can combine image interpretation data with other weather data-sets, to perform some of the higher level integrative decision-making tasks that a human meteorologist currently does. This automation will enable a speedier response by renewable energy generation and distribution management systems. To better match energy supply and demand, by predicting changes in weather patterns over specific local areas.

For example, CNN could be applied to imagery from various weather sources (satellite and radar images, forecast model output graphics, etc), to forecast the temperature, humidity and precipitation. Such a weather combiner would greatly help with the operations and management of electricity grid load, which is heavily dependent on the weather. So such forecasts are hugely important for balancing energy supply with demand. The use of meteorological imagery in the energy industry has started to garner more interest, yet the intersection of ML (CNN), Energy, and Meteorology remains under-explored. But with the integration of renewable generation rapidly growing across the globe, the need to focus on this is becoming more pertinent.

Taking weather satellite imagery specifically, CNN can observe and forecast solar irradiation at the surface. As it can ‘see’ everything, any weather data represented in the form of an image can have filters applied. These are both inherently complex and seemingly similar, even to the human eye. Human eyes are attentive to fine detail, in principle all-encompassing, and hugely experienced due to the constant repetition of situations. Nevertheless, humans can and do make mistakes, and one set of eyes can only see what is in front of it. This is where CNN wins over human eyes. If given enough data, a CNN can help interpolate multiple images, around the clock. In an ideal future, perhaps Meteorologists would no longer have to do night-shifts – a CNN-driven forecasting model could do the job instead.

CNN Can Bring Human Intuition to Numerical Weather Prediction Models

Within the field of Meteorology, weather forecasts are generated by Numerical Weather Prediction (NWP) models, which are then interpreted by Meteorologists to accurately communicate a weather forecast for a given location. Due to the complexity of weather, the human interpretation of these outputs is a crucial step – the model cannot be relied upon alone.
Figure 8: Typhoon Mawar - didn't require much interpretation (Image credit: Wikimedia Commons)
For example, an NWP model output may show there to be short, sharp, heavy showers over Buckingham Palace in London tomorrow. But in reality, these mesoscale-sized heavy showers may occur further to the north than the model predicted. Or maybe the convective rain clouds developed slower than forecasted and ended up causing rainfall later in the day and further downstream from Buckingham Palace. Clearly, there’s uncertainty in the NWP forecast. This is where the job of a trained meteorologist comes in.

The meteorologist would hopefully have understood the complete picture of the NWP forecast and made solid interpretations with that understanding. They see that the predicted rain clouds are small in size and so covering only a very small part of London. They also see that other NWP models are showing the rain clouds to cover different locations. With this added layer of understanding, the meteorologist can then communicate a forecast along the lines of: “Short, heavy showers expected to be scattered around central London.”
Figure 9: ‘Not raining’ at Buckingham Palace (Image credit: Wikimedia Commons)
CNN applied to NWP model graphics could replicate the meteorologists’ interpretation. The CNN can be trained to interpret the spatial relationship of components within the output image, and produce an accurate local forecast. This would be achieved by reading every model output graphic and learning their evolution. This can then be matched with historical model graphics, and the observed conditions seen at every time-step and every location. Once the CNN has enough repetition of each scenario, it will have learnt to understand the complete picture of the forecast, just like a human would. Though the above example of Buckingham Palace is light-hearted, there are far more destructive potential cases in much of the world, where weather patterns are highly volatile and severe.

Similarly, CNN visual recognition can be applied to radar imagery. Radars typically scan the atmosphere every 15 minutes and output a value for the reflectivity that’s bounced back to it by water droplets within clouds and rainfall. Currently a meteorologist must use these radar images to decide if and when rainfall may occur, say over an airport. Use of CNN visual recognition could replicate the human interpolation, and eventually avoid equivalent human error.

Conclusion

Given how fundamental weather forecasting is for the Energy sector, especially as we rapidly integrate more renewables to the grid, there is a clear need to develop weather specific CNN. Computer-vision researchers used to hand-craft filters so that machines could see, now a CNN can learn these filters by itself. In similar spirit, a CNN can start to interpret weather images instead of a meteorologist, eventually achieving far greater accuracy and efficiency. Though development is still very nascent, it echoes what is happening across the energy sector, and many other domains. However, though machines can learn 'how' to see, they don’t yet know ‘what’ to see. So for now, machines have only gained sight. To enhance the sector and address our pressing global energy needs, we must help them gain vision.