Ensemble forecasting
Based on Wikipedia: Ensemble forecasting
In 1963, a meteorologist named Edward Lorenz ran a simulation on a Royal McBee LGP-30 computer, a machine the size of a small room that required punch cards and took minutes to produce a single result. He wanted to extend a weather forecast by restarting the calculation halfway through, but to save time, he entered the initial values from a printout rather than the full internal memory. The printout rounded the numbers to three decimal places, whereas the computer stored six. That difference, a mere one part in a thousand, should have been negligible. Instead, the new forecast diverged wildly from the original, producing a completely different weather pattern within weeks. Lorenz had stumbled upon the butterfly effect, the realization that the atmosphere is a chaotic system where minute errors in initial conditions amplify exponentially over time, rendering long-range deterministic predictions impossible. This discovery shattered the dream of a single, perfect forecast and forced the scientific community to abandon the quest for certainty in favor of a new philosophy: ensemble forecasting.
For decades, weather prediction operated on a deterministic model. Scientists believed that if they could just measure the state of the atmosphere with enough precision and solve the complex fluid dynamics equations accurately enough, they could predict the weather days or even weeks in advance. The reality, as Lorenz demonstrated, is that the atmosphere is a nonlinear, chaotic system. The equations governing it are sensitive to the starting point. If you cannot measure the current wind speed in the middle of the Pacific Ocean with absolute perfection—an impossibility given our sparse network of observation buoys, satellites, and stations—then your starting point is wrong by a fraction. In a chaotic system, that fraction does not stay small. It grows. By the time the forecast reaches six or seven days out, the error has magnified until the prediction is indistinguishable from noise.
To understand why we no longer ask "Will it rain on Tuesday?" but rather "What is the probability it will rain on Tuesday?", one must look at the mechanics of Monte Carlo analysis. Ensemble forecasting is the practical application of this statistical method to the chaotic fluid of our atmosphere. Instead of running the model once to produce a single "best guess" scenario, forecasters run the model dozens, sometimes hundreds, of times. Each run, known as an ensemble member, starts with slightly different initial conditions or uses slightly different model physics. The result is not a single line on a graph, but a fan of possibilities. This set of forecasts aims to map the range of possible future states of the atmosphere, providing a visual and statistical representation of uncertainty.
The power of this approach lies in its honesty. A single deterministic forecast is a lie of omission; it pretends to know the future with a precision it cannot possess. An ensemble forecast admits its own limitations. Ideally, the verified future state of the atmosphere should fall within the spread of the predicted ensemble members. If the ensemble spread is wide, it tells the forecaster, "We are very uncertain; anything could happen." If the spread is tight, with all members agreeing on a specific outcome, it suggests a high degree of confidence. This is not just a theoretical exercise; it is the backbone of modern meteorology. Today, ensemble predictions are standard at virtually every major operational weather prediction facility worldwide. The National Centers for Environmental Prediction (NCEP) in the United States, the European Centre for Medium-Range Weather Forecasts (ECMWF), the UK Met Office, Météo-France, Environment Canada, the Japan Meteorological Agency, and the Bureau of Meteorology in Australia all rely on these systems. The China Meteorological Administration, the Korea Meteorological Administration, CPTEC in Brazil, and India's Ministry of Earth Sciences have integrated ensemble methods into their daily operations. Even the US Navy and Air Force generate their own ensembles for tactical decision-making.
The history of this shift is a story of intellectual resistance followed by inevitable acceptance. In 1969, Edward Epstein recognized that the atmosphere could not be completely described by a single forecast run due to inherent uncertainty. He proposed a stochastic dynamic model that would produce not just a mean state, but also the variance—the spread—of the atmosphere. His work showed promise, but the technology and computational power were not yet there to make it operational. In 1974, Cecil Leith further refined the theory, revealing that these Monte Carlo simulations only produced adequate forecasts when the ensemble probability distribution was a true representative sample of the actual atmospheric distribution. It was a high bar to clear. It took nearly two decades of theoretical development and the exponential growth of computing power before the first operational ensemble forecasts were produced for sub-seasonal timescales in 1985. However, it was not until 1992 that the ECMWF and NCEP began preparing ensemble forecasts for daily use, marking the moment the industry fully embraced the philosophy that uncertainty is a feature, not a bug, of atmospheric prediction.
There are two primary sources of uncertainty that ensemble forecasting must account for: initial condition uncertainty and model uncertainty. The first is the most intuitive. We simply do not know the exact state of the atmosphere right now. Our observation networks have limited spatial and temporal resolution. We have gaps over vast oceans, in the upper atmosphere, and in remote regions. Even when we have data, such as from satellites, the measurements are indirect and require complex retrieval algorithms that introduce their own errors. To address this, ensemble systems perturb the initial conditions. They create a set of starting points that are all consistent with our imperfect knowledge of the current state. One member might assume the wind is slightly stronger in a low-pressure system; another might assume the humidity is slightly lower. By running the model from these slightly different starting points, the ensemble explores the range of possible evolutions that could result from our lack of perfect information.
The techniques for generating these perturbations are sophisticated. The ECMWF's Ensemble Prediction System (EPS) uses a combination of singular vectors and an ensemble of data assimilations (EDA). The singular vectors identify the directions in the atmosphere where errors grow the fastest, focusing the perturbations on the most sensitive areas. These are particularly active in the extra-tropics, where storm systems evolve rapidly. The EDA perturbations, on the other hand, are more active in the tropics, where different dynamics dominate. The NCEP Global Ensemble Forecasting System employs a technique known as vector breeding, which amplifies small errors from a previous forecast to create realistic perturbations for the next run. Perturbing initial states derived from satellite data is a particular challenge; the perturbed states must remain physically plausible. If a perturbation creates a state that violates the laws of physics, the forecast is useless. Recent advancements in deep learning have introduced techniques like flow matching, which can perturb complex initial states while maintaining physical consistency, further refining the accuracy of these starting points.
The second source of uncertainty, model uncertainty, is perhaps even more insidious. Even if we knew the exact state of the atmosphere right now, our computer models are imperfect representations of reality. The atmosphere is a continuous fluid, but computers can only solve equations on a discrete grid. A typical weather model might have a grid spacing of 10 to 25 kilometers. Anything smaller than that—like a single cloud, a patch of turbulence, or a localized heat island—must be represented by parametrisation schemes. These are mathematical shortcuts that approximate the average effect of these small-scale processes on the large-scale grid. These schemes introduce their own errors and uncertainties.
Consider the entrainment coefficient, a parameter used to represent the turbulent mixing of dry environmental air into a convective cloud. This is a complex, chaotic physical process reduced to a single number. That number is uncertain. In a perturbed parameter approach, forecasters identify these uncertain parameters and vary their values between different ensemble members. One member might use a higher entrainment coefficient, causing clouds to dissipate faster; another might use a lower one, allowing clouds to grow taller. This explores how sensitive the forecast is to our understanding of the physics. While in climate modeling these parameters might be held constant, modern numerical weather prediction often stochastically varies the value of these parameters in time and space, acknowledging that the uncertainty is not static. A stochastic parametrisation scheme recognizes that there may be many sub-grid scale states consistent with a particular resolved scale state. Instead of predicting the single most likely sub-grid motion, it samples from a probability distribution, introducing random numbers into the equations of motion. These stochastic schemes, first developed at the ECMWF, have significantly improved the skill of weather forecasting models and are now used globally.
The result of this massive computational effort is a vast amount of data that must be visualized and interpreted. Forecasters and the public use various tools to make sense of the ensemble. Spaghetti plots are a classic visualization where the contours of a weather feature, such as a 500mb height line, are drawn for every ensemble member. If the lines are tightly bunched together, the forecast is confident. If they are spread out like a bowl of spaghetti, the outcome is uncertain. Ensemble means provide an average of all the runs, often smoothing out the noise to reveal the most likely large-scale pattern. Postage stamp displays show individual maps from several different members side-by-side, allowing a forecaster to see the specific variations between scenarios. These tools transform abstract statistics into actionable intelligence.
The practical importance of ensemble forecasts cannot be overstated. They provide a prior estimate of state-dependent predictability. In other words, they tell us not just what the weather might be, but how sure we can be about it. This is crucial for decision-making. If a hurricane is approaching, a deterministic forecast might show a specific landfall point. An ensemble forecast might show a cone of uncertainty, with some members hitting a major city and others missing it entirely. This distinction changes everything. It shifts the conversation from a false sense of precision to a risk-based strategy. Emergency managers can prepare for the worst-case scenarios while understanding the likelihood of different outcomes. It allows society to navigate the inherent chaos of the atmosphere with a level of caution and preparedness that deterministic models could never support.
The limitations of this approach are well understood. As Lorenz predicted, long-range forecasts—those made more than two weeks in advance—cannot predict the state of the atmosphere with any degree of skill. The chaotic nature of the fluid dynamics equations ensures that errors will eventually overwhelm the signal. Current ensemble systems generally limit the useful forecast window to about six days. Beyond that, the spread of the ensemble becomes so wide that it offers little more than climatology. However, within that six-day window, the ensemble approach has revolutionized our ability to see the future. It has turned the question from "Will it happen?" to "How likely is it to happen?"
The evolution of ensemble forecasting is a testament to the scientific method's ability to adapt when faced with the limits of human knowledge. We cannot measure the atmosphere perfectly. We cannot model its physics perfectly. But by embracing these imperfections and running the numbers thousands of times, we have found a way to quantify the unknown. We have learned to live with uncertainty, not by ignoring it, but by mapping it. The ensemble forecast is a humble acknowledgment that the future is not a single line written in stone, but a cloud of possibilities, and that the most honest thing we can say about the weather is a probability.
"The atmosphere is a chaotic system. It is impossible to predict the weather with certainty beyond a certain time horizon." — Edward Lorenz, 1963
This insight, once a barrier to progress, has become the foundation of modern meteorology. The journey from Lorenz's punch cards to today's supercomputer ensembles represents a fundamental shift in how humanity interacts with the natural world. We no longer pretend to control the weather or to know it with absolute certainty. Instead, we measure the odds, we prepare for the range of possibilities, and we respect the chaos that drives our planet's climate. The ensemble forecast is the best tool we have to navigate a world that is, by its very nature, unpredictable.
The future of ensemble forecasting lies in the continued refinement of these techniques. As computing power grows, the resolution of the models increases, and the number of ensemble members can expand. New techniques in machine learning are being integrated to better perturb initial conditions and to model the stochastic behavior of sub-grid processes. The goal is not to eliminate uncertainty—that is impossible—but to shrink the cone of uncertainty and to better characterize the risks that remain. Every day, thousands of ensemble members run around the clock, simulating a thousand different futures, so that when the storm comes, we are not caught off guard. We are ready, not because we know exactly what will happen, but because we know what could happen. In a chaotic universe, that is the closest thing to wisdom we possess.