Thursday, August 2, 2018

Harnessing the Wind and Sun to Achieve an Energy Surplus Future

Imagine a future where a 100% renewables-powered electricity grid harnesses the abundance of energy beamed down on us daily, to give 100% reliability of supply. Enabling energy prices to be a fraction of what they are today, or perhaps even universally free. Let’s consider two paths of thought, one into the far future, the other more nearer-term for India, and rather imperative.

Looking to history to inform our future-gazing, the indications are that we’ll find ever newer ways to consume energy. So we may take indeterminate time to reach ‘surplus’ in the strictest sense, though that does not preclude universal free access. Even if we developed the ability to absorb all the energy emitted by a star - perhaps through the hypothesised Dyson Sphere - and become a ‘Type 1 civilisation’. Tech advancement will likely hog whatever we produce. More likely, we’ll develop ships that are electric instead of solid-fuel run, and hyperloops to replace trains. Energy is like money, we will always find ways to consume it - our way of life will just change to match.

Today however, India already has an energy surplus, with approximately 300GW of installed capacity, and only 150GW of peak demand. But this is a technical rather than practical reality. There is still load shedding in most parts of the country after all, and our per capita consumption remains low because most people still have only limited access, in term of both connectivity and availability of power. Further, India is a tropical country blessed with ample sunshine and wind, even in the monsoon months. Yet at present, India still imports much of its energy from abroad, notably in the form of oil and coal. The government is convincingly pushing for renewables, and wants to transition us to a ‘Solar Nation’, in no small part to reduce the national import bill. Achieving this could accelerate the Make in India campaign, because national energy-independence will mean the cheapest energy possible.

Once India reaches national energy independence, another equally fundamental development and national health challenge that can be addressed, is the stark water distress. Cheap energy will allow unlimited pumping of water across immense distances, enabling this demon to finally be dispelled. Nascent water desalination technologies - which already exist, but are hamstrung by huge scaling costs - will then take off along the coast. This will enable pumping of clean water to the northern inlands, and onwards to other countries in similar need.

Technologies such as distributed energy generation and storage will take some pressure off the future central electricity grid - for example by reducing the load on transmission infrastructure - there can be no such ‘distributed desalination’ equivalent for water. The oceans are fixed, so plants must remain coastal, or possibly at some suitable inland locations, such as Sambhar lake in Rajasthan. This will mean building significant new pumping infrastructure, with massive energy demands to fully service the country,

Thus, a prerequisite for clean water is vast amounts of cheap energy, and this can only be possible from solar and wind, which are inherently variable, so will increase grid-volatility. Today, even with just a small penetration of renewable generation, the intermittency has caused a significant rise in balancing costs, and in the future this could be huge.

Given this, we must identify solutions with the highest benefit-to-cost ratio across the energy-ecosystem, and software will always be cheaper and less cumbersome than physical buildout. This is where artificial intelligence and machine learning (AI & ML) come into frame and excel - crunching the exponentially growing volumes of data required to balance gird-volatility, forecast future trends, optimise existing assets, and plan future infrastructure. We are already seeing AI & ML starting to handle these issues in the real-world, from forecasting of generation and demand, to scheduling of supply. This bodes well for the future water-energy nexus.

So, there are a multitude of ways we can imagine using an envisaged future energy surplus, but solving clean water access for all, with the help of AI & ML, seems to be the one that will most benefit the country and its people.


Credit to Niladri Roy for co-writing this article.

Thursday, June 28, 2018

Aggregated Weather Forecasts for Large Load Areas


Introduction

When checking the weather forecast in the mornings to see whether you’ll need an umbrella that day, you’ll probably be looking at the forecast for your current point-location, giving data specific to your neighbourhood. If you’re planning a weekend away, the forecast display can easily be changed to another specific point-location. But what if your business need to know what is happening across an area much greater than what can be represented by a single point location? For large areas, obtaining weather forecasts from point-locations is unlikely to suffice and may lead to strategic errors for the business, and ultimately, financial losses.

One example of where aggregated weather forecasts are required is in the energy industry. Electricity demand (load) is strongly dependent on the weather conditions, and so weather data is a vital component of electricity load forecasting. Aggregate-level load forecasts must consider the weather conditions across the entire area, as opposed to using just one location to represent the weather conditions everywhere. This article aims to address the challenges of weather location selection and proposes the best methodologies for determining how many, and which, weather locations should be used when forecasting load over large areas.
The Indian state of Uttar Pradesh, India’s most populous state, is taken as an example case study. Uttar Pradesh covers 243,290 km2, equalling the size of the United Kingdom. Weather therefore varies significantly across the state – there are higher altitude areas in the north, and more rainfall occurs in the north and eastern areas. As a result, the impact of weather on the state’s load will also vary significantly across the area.
Uttar Pradesh - India’s most populous stare.

In particular, localised extreme weather events can be hugely problematic. For example, there could be heavy rainfall in Agra (west Uttar Pradesh), but sunny skies in Lucknow (central Uttar Pradesh). If the weather data is only received for Lucknow, the forecasting system would have no knowledge of the rainfall elsewhere.
(Include a relevant radar image once available)
The geography and climatology varies across Uttar Pradesh.

Methodology

In order to give an aggregated state load forecast, a virtual weather station must be formed, which best represents the weather across the entire state.
Initially, the weather forecast locations you do choose must have data availability, reliability and accuracy. All weather stations that do not, must be ignored. Let’s assume that we begin with 20 weather station locations across Uttar Pradesh that have reliable and accurate data, and provide a good geographical spread across the state.
The virtual weather station can then be built by using multiple point-locations. But how many point locations should be used? Too little will not represent the area well enough, but too many point locations may actually increase the forecast error and increases the amount of data, which has its own issues.
Once you’ve decided how many point locations to use, how do you decide which point locations should be used? Do you choose a roughly even spread of locations across the entire area? Or do you choose more in the west because population is higher there? Ultimately, you want to represent the impact of weather on load, not just the average weather of the state. This will vary with population density, economic and demographic diversity, as well as the end-use diversity of the residential consumers.
Another challenge is that weather forecasts are rarely perfect. Inaccuracies are lower the closer in time we are to the forecast time, but will often still contain significant errors. To rectify this, real-time observational data can be used to compare the forecasted values with the actual observed values and adjust the weather forecasts accordingly. This is particularly effective during days of extreme weather, because the forecasting system is able to quickly react to sudden weather changes.

Methods for aggregated weather forecasts


a)      Simple Averaging

The first method simply takes an average of every forecast variable, from all 20 weather locations. Although this is a simple method, it can be very effective at capturing the impact of weather on load across a large area, as shown by multiple studies (Lloyd 2014). However, there are of course issues to using this simple average method. Firstly, averaging may actually reduce accuracy - many studies have shown that sometimes you can use too many weather locations and cause unwanted smoothing of the data. Additionally, taking the average fails to deal with:
·         the effect of geographical diversity – differing weather across the state;
·         demographic diversity – differing economic and classes across the state;
·         end-use diversity – differing efficiencies of appliances, difference uses of electricity, etc.
Also, why 20 weather locations? Why not 5 or 10, or 100? This number has been heuristically (randomly) picked and is unlikely to be the optimal number of weather locations to use.

b)      Best-fit to Load

In this second method, we progress on the simple averaging technique, by building a load model for every weather location (20 in this case). The accuracy of each model is then tested against actual load and the individual models are ranked in order of their accuracies. Next, the weather data from the top 5 best-fit models are combined, perhaps through simple averaging. This method only uses weather locations that have a strong impact on load, and so reduces the chance of errors creeping in from unimportant weather locations (Charlton and Singleton, 2014). But how do we know if these 5 models have captured the effect of geographical diversity, economic diversity or end-use diversity? And why the 5 best models and not the top 3, top 7 or top 15 models? Again, these numbers have just been heuristically selected.

c)      Zonal best-fit to Load

The third method is very similar to one above, but this time we select the best model for each ‘zone’ (Nedellec et al., 2014). Zones are defined by the operators, covering a small, defined territory within the greater load area. Again, a separate load model is built, one per weather station location. The accuracy of each model is then tested against the actual load. Next, within each zone, the weather station that achieved the highest load accuracy is used. All other weather stations within that zone are ignored. Finally, the average is taken of all the zonal weather stations.
This deals with the geographical diversity problem – all zones are represented. But as a result, it may neglect the issue of capturing the population, economic or end-use diversities. And again, the number of zones and the initial number of weather locations is randomly decided.

The state can be split into zones.

d)      Ranking and Rating

All three of the above methods have heuristically selected the number of weather stations to use; they do not answer the question “How many weather locations should be used?” The Ranking and Rating method attempts to answer this question (Hong & Wang et al., 2015).
A load model is built for each individual weather location. These loads are then compared against the actual aggregate load and ranked in order of their load error, in terms of the mean average percentage error (MAPE) or root mean squared error (RMSE). Obviously, the highest ranked model contains the weather station that has the highest impact on the load. The next step then combines the weather data from x number of the highest performing load models, where x is a number ranging from 2 to the total number of locations; e.g. for x = 2, 3, 4, 5, etc. Again, we fit the virtual weather station of each combination scenario with load data, and arrange in ascending order of error (MAPE or RMSE). The virtual weather station that produces the lowest error is therefore the combination of weather stations locations to use.

Combining weather stations – weighted means

How should the weather locations be combined? A simple average may be effective, but it may be more effective to apply a weighted mean. This weighting could be by population density, giving greater influence to weather in areas with higher populations. This is often used in developed countries, where population correlates well with local electricity demand, but this may not be the case in developing countries. For example in India, highly populated areas tend to be poorer areas with less energy-use per capita, and so population density does not correlate well with local electricity demand. Therefore, a different type of weighting should be applied, such as the economic density of the area. This may correlate better than population density for developing countries, because economically powerful areas tends to be co-located with areas of high residential demand.
The population density of Uttar Pradesh, India.

An important point to make here is that weather insensitive loads should be taken out of the equation. Weather-insensitive loads are sources of load that are not affected by the weather. These are mostly from industrial sources.
Weather insensitive load centres, like this car factory, must be ignored when applying the Rating and Ranking method.

Conclusions

There are multiple methods for producing an aggregated weather forecast, applicable to electricity demand forecasting and management for large load areas. The best of these is the Rating and Ranking method, which ranks the weather station locations by their load error when inputted into a single-station load model. This answers the question “what weather station locations should be used?” Next, the question of “how many weather station locations should be used?” is answered by combining the weather data from x number of the highest performing load models. The virtual weather station that produced the lowest error is therefore the combination of weather stations locations to use.
These weather stations can be combined using a simple average or a weighted mean. It is suggested that a weighted mean by population density is suitable in developed countries, but a weighted mean by economic density is more suitable in developing countries.

References

Charlton, N., & Singleton, C. (2014). A refined parametric model for short term load forecasting. International Journal of Forecasting, 30(2), 364–368.
Hong, T., Wang, P. and White, L., 2015. Weather station selection for electric load forecasting. International Journal of Forecasting, 31(2), pp.286-295.
Lloyd, J. R. (2014). GEFCom2012 hierarchical load forecasting: Gradient boosting machines and Gaussian processes. International Journal of Forecasting, 30(2), 369–374.
Nedellec, R., Cugliari, J., & Goude, Y. (2014). GEFCom2012: Electric load forecasting and backcasting with semi-parametric models. International Journal of Forecasting, 30(2), 440–446.

Sunday, May 6, 2018

How Machines Can Learn to ‘See’ the Weather and Enhance Future Energy Management



Sight or Intuition?

How do we understand the world around us? That is a surprisingly complex question. Let’s try to explain using a real-life example. In the past, whilst enjoying a merry late night meander back from the bar, you may have experienced being barked at, or even chased by a dog.

It was obviously a dog, but did you know what kind of dog? German Shepherd, Labrador, Beagle? In that situation it didn’t matter, it was just a ‘dog’, a barking one with sharp teeth.

But how was that understood? For humans, there is a degree of intuition. Aside from the barking and teeth, it is because we all share a common abstract image of 'dog'. The internal image is likely to vary from person to person. If you grew up with a Golden Retriever, your internal ‘dog’ will look more like that, with some positive emotional attachment. And if you grew up with a Rottweiler snarling at you, your image will be one with scary teeth, and some frightful memories.

Perhaps surprisingly, this is not so dissimilar to how a visual processing AI 'sees' the world.
A Machine's Brain

Having established how humans understand the world in an abstract sense - by having representations inside our brains - enables us to start conceptually understanding how a machine ‘sees’. It obtains its own internal image of reality, which is in a sense the mission of Machine Learning (ML), to obtain a representation of whatever we want it to learn, but in a systematic way. We can encourage machines to form their own internal image of reality using ML, by giving them a mathematical foundation from which they can construct these.

So if we have a picture of a dog, how can we enable a machine to learn how to see that dog?
Figure 1: Dog - a friendly, drooling one (Image credit: Wikimedia Commons)
Figure 1 shows just one dog. But there are hundreds of different breeds, and a machine cannot obtain a general image of ‘dog’ from just looking at a single example. Figure 2 shows several dogs, and we could assemble thousands. Each a different type and in different poses.
Figure 2: Selection of different dogs (Image credit: Wikimedia Commons)
They are simple enough to differentiate. But, now squint your eyes, and you’ll see something like Figure 3.
Figure 3: Blurred dogs - but still identifiable by a human (Image credit: Wikimedia Commons)
As a human, you can probably still tell them apart, but don’t those dogs now look at least slightly similar to each other? Certainly more so than the clear pictures (Figure 2). All we did was decrease the resolution of the image (known as down-sampling). More precisely, we picked a group of pixels and took their average (Figure 4), then repeated for all areas.
Figure 4: The operation of blurring (Image credit: Wikimedia Commons and Sho Akama Images)
Now imagine if we apply the same process to all the picture several times. They will look increasingly similar. We might then only be able to obtain abstract images of ‘dog’, containing some distinct features, like the shape of its head and body, its saucer bowl eyes, wet button nose, drooling tongue, floppy ears, and maybe sharp white teeth. But whereas human intuition can still make sense of these incomplete datasets, machines typically cannot, they lack that intuition.

Convolutional Neural Networks

We all see the world through our own personal 'lens'. Elderly people tend to have slightly blurrier eyes than young people in a physical sense. More conceptually, a politician has a better lens to understand parliamentary dynamics. And a physicist has a lens to see the invisible laws of the universe.

But what does the sight lens of a machine look like? Within the ML domain, this lens is called a filter (or kernel). For any computer, an image is a 2-dimensional array with numbers. It cannot recognise a complex object like a dog in one go. It starts from understanding edges and lines, and by stacking different filters, it can eventually ‘see’ a dog. An ML model specifically designed for computer vision is called a Convolutional Neural Network (CNN).

Traditionally the set of filters is handcrafted by specific ML-area experts. For a human face, you may want to consider a filter that can detect the general geometric shape of faces, like the ‘T-line’ between the nose and eyes (Figure 5).
Figure 5: T-Line, on just some random guy (Image credit: Wikimedia Commons)
To identify a chameleon, you need a very different set of filters than to identify a human (maybe less so if they are of the Karma variety). This means every time you want to detect something new, you need a group of experts for R&D. Imagine trying to develop the best filters for a chameleon. What would they be? To detect rounded edges and stack those up to find two big eyeballs? Why not a filter to find their signature long tongue? Now how are those filters different from the filters for a human? Can our filter distinguish the two creatures in figure 6?
Figure 6: Chameleon vs human (......probably) (Image credit: Wikimedia Commons)
So there are a seemingly infinite number of possibilities, and even with years of research, it is just not practicable to obtain a ‘best’ set of filters. This is where CNN enters the frame and excels - a machine capable of learning filters relative to given objects.
Figure 7: How a Convolutional Neural Network recognises an object (Image credit: Wikimedia Commons)
In Figure 7, each blue block is applying a filter, then extracting core information relative to that filter (also known as ‘convolution’). The size of subsequent blocks decreases because each time a filter is applied, the resulting image shrinks.

Each sequential group of blue blocks is called a layer. By exploiting sets and combinations of filters, a CNN can recognise complex shapes, like dogs (or chameleons).

CNN for Weather Prediction and Better Energy Management

So now we have understood CNN and how a machine can ‘see’. But the link between recognition of a dog and Energy management probably still feels very abstract.

Rather than jump to how machines can ‘see’ and interpret weather patterns to make forecasts. Let's also consider how applications to forecasting are very different to simple object recognition. There are some important future possibilities that could provide compelling solutions to energy industry problems, but it is important to first smooth that gap of understanding.

ML-enhanced visual recognition can be used to recognise, and more crucially interpret, specific visual weather patterns, in much the same way that they can learn how to recognise dogs. Then, these machines can combine image interpretation data with other weather data-sets, to perform some of the higher level integrative decision-making tasks that a human meteorologist currently does. This automation will enable a speedier response by renewable energy generation and distribution management systems. To better match energy supply and demand, by predicting changes in weather patterns over specific local areas.

For example, CNN could be applied to imagery from various weather sources (satellite and radar images, forecast model output graphics, etc), to forecast the temperature, humidity and precipitation. Such a weather combiner would greatly help with the operations and management of electricity grid load, which is heavily dependent on the weather. So such forecasts are hugely important for balancing energy supply with demand. The use of meteorological imagery in the energy industry has started to garner more interest, yet the intersection of ML (CNN), Energy, and Meteorology remains under-explored. But with the integration of renewable generation rapidly growing across the globe, the need to focus on this is becoming more pertinent.

Taking weather satellite imagery specifically, CNN can observe and forecast solar irradiation at the surface. As it can ‘see’ everything, any weather data represented in the form of an image can have filters applied. These are both inherently complex and seemingly similar, even to the human eye. Human eyes are attentive to fine detail, in principle all-encompassing, and hugely experienced due to the constant repetition of situations. Nevertheless, humans can and do make mistakes, and one set of eyes can only see what is in front of it. This is where CNN wins over human eyes. If given enough data, a CNN can help interpolate multiple images, around the clock. In an ideal future, perhaps Meteorologists would no longer have to do night-shifts – a CNN-driven forecasting model could do the job instead.

CNN Can Bring Human Intuition to Numerical Weather Prediction Models

Within the field of Meteorology, weather forecasts are generated by Numerical Weather Prediction (NWP) models, which are then interpreted by Meteorologists to accurately communicate a weather forecast for a given location. Due to the complexity of weather, the human interpretation of these outputs is a crucial step – the model cannot be relied upon alone.
Figure 8: Typhoon Mawar - didn't require much interpretation (Image credit: Wikimedia Commons)
For example, an NWP model output may show there to be short, sharp, heavy showers over Buckingham Palace in London tomorrow. But in reality, these mesoscale-sized heavy showers may occur further to the north than the model predicted. Or maybe the convective rain clouds developed slower than forecasted and ended up causing rainfall later in the day and further downstream from Buckingham Palace. Clearly, there’s uncertainty in the NWP forecast. This is where the job of a trained meteorologist comes in.

The meteorologist would hopefully have understood the complete picture of the NWP forecast and made solid interpretations with that understanding. They see that the predicted rain clouds are small in size and so covering only a very small part of London. They also see that other NWP models are showing the rain clouds to cover different locations. With this added layer of understanding, the meteorologist can then communicate a forecast along the lines of: “Short, heavy showers expected to be scattered around central London.”
Figure 9: ‘Not raining’ at Buckingham Palace (Image credit: Wikimedia Commons)
CNN applied to NWP model graphics could replicate the meteorologists’ interpretation. The CNN can be trained to interpret the spatial relationship of components within the output image, and produce an accurate local forecast. This would be achieved by reading every model output graphic and learning their evolution. This can then be matched with historical model graphics, and the observed conditions seen at every time-step and every location. Once the CNN has enough repetition of each scenario, it will have learnt to understand the complete picture of the forecast, just like a human would. Though the above example of Buckingham Palace is light-hearted, there are far more destructive potential cases in much of the world, where weather patterns are highly volatile and severe.

Similarly, CNN visual recognition can be applied to radar imagery. Radars typically scan the atmosphere every 15 minutes and output a value for the reflectivity that’s bounced back to it by water droplets within clouds and rainfall. Currently a meteorologist must use these radar images to decide if and when rainfall may occur, say over an airport. Use of CNN visual recognition could replicate the human interpolation, and eventually avoid equivalent human error.

Conclusion

Given how fundamental weather forecasting is for the Energy sector, especially as we rapidly integrate more renewables to the grid, there is a clear need to develop weather specific CNN. Computer-vision researchers used to hand-craft filters so that machines could see, now a CNN can learn these filters by itself. In similar spirit, a CNN can start to interpret weather images instead of a meteorologist, eventually achieving far greater accuracy and efficiency. Though development is still very nascent, it echoes what is happening across the energy sector, and many other domains. However, though machines can learn 'how' to see, they don’t yet know ‘what’ to see. So for now, machines have only gained sight. To enhance the sector and address our pressing global energy needs, we must help them gain vision.


Wednesday, December 20, 2017

How Weather Forecasting Will Drive Down Electricity Bills in the Renewable Energy Future

Renewable energy sources like wind and solar power are inherently variable due to their dependence on natural phenomena. For renewable systems to be integrated into our energy mix, comprehensive system management must be in place, where energy sources are accurately scheduled ahead of time to ensure electricity demand is always matched by supply. This therefore requires precise forecasting of renewable generation. The dependence of these systems on the weather means that accurate meteorological data is essential as an input into renewables forecast modelling. As renewables increase in the energy mix globally, the need for accurate and valid weather data is becoming increasingly more important.

The Rise of Renewable Energy
The renewables revolution is well under way. Today, renewable energy (excluding hydro) only accounts for 8% of global electricity generation [1], but its current growth is rapid - almost 40% of the additional global power generation in 2016 was from renewables. The fastest rises compared to 1990 levels have been from solar photovoltaic (PV) (45.5%) and wind power (24%) [2], due to national and international policies, and sharply falling costs. Record-low solar PV auction prices below 3 cents per kilowatt-hour have occurred across the globe, including in Argentina, Chile, India, Jordan, Saudi Arabia and the United States. Similar records have been seen for electricity from wind developments, spearheaded in Chile, India, Mexico, Morocco, Denmark and the Netherlands.

The Rise of Electricity Bills
But does lower renewable kWh prices result in cheaper electricity bills for everyday consumers? End-user electricity prices have been on a continuous upward trend in many countries. In the UK, there was a 63% rise in electricity prices between 2004 and 2014 [3]. A similar picture has been seen across Europe and further afield. According to the US Energy Information Administration, US utility residential electricity rates have increased by around 20% in the last 10 years [4].  The most influential driver of this has been the rising price of natural gas, and this trend is expected to continue. For electricity bills to come down, cheaper energy sources must be realised – can renewables provide this?

The Renewable Energy Future
The growth of renewables is expected to continue. The International Energy Agency (IEA) predict that by 2040, the installed capacity of solar and wind will increase by a factor of 14 and 4, respectively. This will result in the share of global electricity generation accounted for by solar and wind rising from 5% today to 34% in 2040 (figure 1) [1].
Figure 1. Global electricity generation mix to 2040. Source: Bloomberg New Energy Finance, New Energy Outlook 2017. 
But as governments remove feed-in-tariffs and other supportive incentives, renewables need to be able to stand on their own feet. The only way this can happen is if they are cheaper than conventional sources. Ever more efficient solar PV modules and wind turbines will reduce costs, but price decreases must also come from more efficient operating and maintenance procedures. Yet due to the uncontrollable variability of wind and solar PV systems, their integration into power systems poses challenges.

Balancing Supply and Demand
Supply must always balance demand. This is achieved by grid operators scheduling power generation in advance to match the expected demand at all times. The limited ability to anticipate variable energy sources on a minute-by-minute basis results in supply and demand deviating from each other in real-time and as a result, grid operators must have additional power plants (usually fossil-based) on continual standby.  These balancing costs can be substantial, and ultimately, greater imbalancing costs lead to higher electricity prices for consumers. Ofgem, the UK’s electricity regulator, notes that balancing costs of around £850,000 per year adds £9 to the average UK electricity bill [5]. Worse still, getting the balance wrong leads to blackouts or power surges, further impacting energy costs.

Power system flexibility is therefore required – the capability to maintain continuous grid balance in the face of rapid and large swings in supply or demand. Without it, wholesale prices of renewable energy become depressed, particularly when wind and solar production exceeds the demand on sunny, windy days.

Renewable Energy Forecasting
To achieve power system flexibility, comprehensive forecasting of renewable systems is essential. Traditional forecasting approaches in the energy sector have mainly been statistical techniques, comprising of time-series methods or regressions. However, with the advances in big data, processing capacity and tools to manipulate these datasets, artificial intelligence-driven modelling is starting to take more precedence. This is due to its flexible nature and ability to handle non-linearity, which is inherent in many energy forecasting problems, especially those required at a granular time-scale. Figure 2 displays some of the inputs for solar generation forecasting using artificial intelligence-driven modelling.
Figure 2. System architecture of solar PV generation forecasting using artificial intelligence-driven modelling.
The Need for Weather Data
The power produced by a solar plant is strongly dependent on the global horizontal irradiance (GHI) – the total amount of shortwave radiation received from above by a horizontal surface, at an instantaneous point in time. GHI is measured by on-site pyranometers and expressed in Watt per square metre (Wm-2). GHI values are then adjusted as per the tilt and orientation of the solar panels – the global tilted irradiance, GTI (figure 3).
Figure 3. Global Horizontal Irradiance (GHI) and Global Tilted Irradiance (GTI).
GHI, and therefore solar PV output, has seasonal and diurnal cycles. GHI also dramatically fluctuates on much shorter timescales due to sharp changes in cloud cover and atmospheric conditions.

The power produced by wind turbines is dependent on the cube of wind speed. Turbines have a lower limit at which no power is generated, and an upper limit at which turbines are shut down to avoid damage and failure. Although less pronounced compared to solar, wind power may fluctuate on seasonal timescales, down to very short-term fluctuations due to localised wind speed changes.

All fluctuations in GHI and wind speed must be captured to high precisions – renewable energy forecasting is only good if the weather data feeding into the forecast models is good.

Different Weather Sources for Different Forecast Lead Time
The best source of weather data for forecasting of renewables depends upon the forecast lead time required, particularly for solar power forecasting (figure 4).
Figure 4. Conceptual plot of forecast lead time vs. forecast skill for different sources of weather data: ground observations, satellite imagery or NWP model output.
At sub-hourly lead times, the highest solar forecast skill is provided by methods based on on-site ground observations, such as from real-time infrared imagery.
But forecasts based on these local observations become less skilful as the forecast lead time increases, which is when satellite-based methods for computing GHI become useful. Geostationary satellite imagery (e.g. from Meteosat-10 over Europe and Africa) allows clouds further afield to be captured, and cloud motion vectors enables the future positions of clouds over the site of interest to be inferred.
Obviously, clouds evolve, so the use of satellite imagery to compute solar radiation is limited to around 7 hours ahead. After this, weather forecasts from numerical weather prediction (NWP) models become most effective for solar forecasting. NWP models incorporate our physical and dynamical understanding of the atmosphere to forecast the future state of weather using mathematical equations and huge computational power.

NWP Forecast Uncertainty
However, NWP forecasts are not perfect. It is impossible to predict with certainty for every point in space and time, but there is still plenty of room for improvement.

The main limit to their accuracy is the spatial resolution of the NWP model. The finer the spatial resolution, the smaller the grid cells that divide up the atmosphere, and therefore the more representative the predicted variables will be to every point in that cell. This itself is limited by computational power, despite the NWP models like the GFS and ECMWF running on the most super of supercomputers. Consequently, there’s a lot of information the models struggle to infinitely resolve. This includes all atmospheric phenomena that are smaller in size than the grid resolution, such as turbulence and small cloud features like cumulus clouds. Surface conditions are also not fully resolvable: is the surface cover vegetation or water or concrete? What is the topography of the surface?

Anything that cannot be resolved are parameterised in the model, which means that they are approximated using over-simplified assumptions. A cumulus cloud, which has an average width of about a 1km, must be parameterised, and hence may be poorly represented by the model. This provides a large source of uncertainty in solar forecasting; solar plant output can dramatically fall within seconds as a cloud moves over and blocks out the Sun’s radiation.

Good observational data is also essential – it’s a case of garbage in, garbage out. Where observational data is not available, interpolation is used to fill the grid with initial conditions. But any errors in the initial observational data will result in errors in the interpolated data too, and this may lead to large uncertainties creeping in. Chaos theory (the butterfly effect) means that the forecast output is highly sensitive to the initial and boundary conditions and so may have very different outcomes with just slight differences to the model inputs. Chaos will increase with time, so forecasts become increasingly more uncertain the further out they are made for.

High model generation frequency (how often the model runs) helps to eliminate this issue by re-running the model with the latest observational data. For example, the GFS runs every 6 hours, whilst the Met Office’s UKV model runs every 3 hours. As with the spatial resolution, model run frequency is limited by computer power.

Reducing Overall Costs
The savings due to good energy forecasting can be substantial - five-fold reductions in the costs of hourly imbalance penalties levied on generation have been seen in the past. The UK's Central Electricity Generating Board claimed that if perfect wind power prediction was possible, it would add 10% to the value of wind power on the system. 

Conclusion
With an increasing share of wind and solar in the generation mix across the globe, the overall generation variability is set to rise, and hence the challenges of balancing supply and demand will become greater. But while wind and solar power cannot be dispatched, they are schedulable, with the support of accurate weather forecasting. Ground instruments, satellite imagery and NWP models are all essential when forecasting renewable generation in real-time. As accurate forecasting of renewables becomes more prominent, renewable energy prices will fall, and this should lead to lower electricity bills for consumers.

References
[1]
Bloomberg New Energy Finance, “New Energy Outlook,” 2017.
[2]
International Energy Agency, “Renewables information 2017,” 2017.
[3]
Ofgem, “Annual Report 2015-2016,” 2016.
[4]
U.S. Energy Information Administration, “Annual Energy Outlook,” 2017.
[5]
Ofgem, “Balancing and Settlement Code (BSC) P333,” 2016.


Monday, January 30, 2017

Red sky at night, shepherd's delight?

Imagine a world where immediate access to reliable weather forecasts did not exist. How many layers would you wear today? Would you pack an umbrella? It's likely you'd get caught out by the weather a little more, but after living in a pre-weather-forecast-world for a little while, you may start to develop your own predictions for the weather. It was from human experience that old forecasting phrases were born, and we still know these sayings today because some of them often worked and helped our ancestors plan for the upcoming weather. So is there any meteorological foundation to the saying "red sky at night, shepherd's delight; red sky in the morning, shepherd's warning"?

The sky appears a glorious pink/red when aerosols (small particles) in the atmosphere preferentially scatter the blue part of the Sun's light spectrum, meaning our eyes receive a more reddish colour from the sky. The preferential scattering requires a low incident angle from the Sun, hence why this phenomena occurs at sunrise or sunset. Also required are the small atmospheric particles, which often exist in greater concentrations when high pressure dominates.

If there is red sky whilst the sun sets in the west, there is likely to be high pressure to the west of our location. If the high pressure is moving east it will soon reach our location, often bringing pleasant, stable conditions with it - a shepherd's delight. Because weather systems predominantly come from the west in the UK, red sky at night is often followed by high pressure and calm weather.

Red sky at night over the University of Leeds on 30/11/16.
However, red sky as the sun rises in the east tells us at there is likely to be high pressure to the east of our location. Under the predominantly easterly-moving weather systems over the UK, the high pressure departs further east of our location and as it moves out, low pressure often takes it place. Low pressure systems are associated with more wet, windy weather, so the morning red sky comes as a warning - poor weather conditions are coming!

Red sky in the morning over Leeds on 05/12/16
Unfortunately, the complex nature of the atmosphere makes this old saying not so straight forward. Weather systems reach the UK from every direction, not just from the west. Small particles do not only exist in the atmosphere under high pressure, either, so it is possible to have a red sky when low pressure is present in the direction of the Sun.

Nevertheless, this saying, like other old weather proverbs and stories, give us an idea of what weather forecasting consisted of before today's meteorological understanding and accuracy.