Early warning of grid load sheds, scored like weather warnings
Since 2020, US grid operators have had to cut power for lack of supply eight times. The public was warned beforehand in four. GridShedWatch is a shared test of whether a warning method would have caught them in time: all eight events, public hourly data, and one fixed scoring procedure any method can be run through.
Why a benchmark
Plenty of methods claim to spot grid trouble early. None of them can be compared, because there has been no common test.
What exists
- Seasonal reliability assessments (NERC) and operators' own alert levels, set by judgment.
- Anomaly-detection studies on simulated grids, or on one event, scored per hour.
- No public list of real load sheds with a fixed way to score warnings against them.
What GridShedWatch adds
- Every qualifying load shed since 2020, chosen by a stated rule, not by hand.
- The same data, rules and alert budget for every method, so only the method differs.
- The operators' own public notices as the baseline to beat.
How it is scored
The National Weather Service scores every warning on three numbers read together: events warned in advance, lead time, and false alarms. GridShedWatch asks the same of grid warnings, at two levels.
Something unusual is building
About one a month per grid. Counts if an alert starts in the 7 days before the load shed begins.
Act now
Two a year per grid. Counts from 7 days before onset to 48 hours after, since a warning during the event still helps.
The rules every method shares
- A method only scores each hour. For each of five signals from EIA-930 (demand forecast error, demand minus generation, and seasonal anomalies of demand, generation and their gap) it says how unusual the hour is, from 0 to 1, using data up to that hour only.
- One shared rule turns scores into alerts. Both signal groups must be unusual at once and stay so for a few hours. The rule is the same for every method.
- Same alert budget, set without the events. Each method's threshold and persistence are chosen so it raises at most 12 watches and 2 warnings per grid per year, looking only at alert counts.
- Read against chance. Each method's own alerts are shifted to random times 2,000 times to show how many events it would catch by luck. Intervals come from resampling the events.
Full protocol: docs/scoring.md.
The eight events
Every operator-initiated load shed caused by too little supply in a grid that reports to EIA-930, January 2020 to September 2026, of at least 250 MW at its worst. Three events were used to design the shared rules; five were held out and scored once.
| Event | Grid | Shed at worst | Onset (local) | Split |
|---|
Selection rule, onsets and exclusions: docs/events.md.
What the operators said beforehand
Every notice each operator issued in the week before its load shed. Public notices (aimed at customers) are the status-quo baseline: they count as a watch if issued before onset.
Hover a marker for the notice. Sources for every time: operator_notices.yaml.
Results, v1.0
Five reference methods, one from each family: rules, statistics, machine learning (LSTM, HTM) and a foundation model (Chronos-Bolt, zero-shot). All at the same alert budget.
Event by event
Share of runs that caught each event, with the median lead time in hours. The LSTM and HTM have five seeds each.
Full table
With eight events, differences of about one event are not meaningful. Per-event and per-seed results: results/v1.0.
Add your method
Two ways in. Both are scored the same way, at the same alert budget.
Scores track
The controlled comparison. Your method scores how unusual each of the five signals is, every hour. The shared rules, the budget step and the scoring are done for you, so it cannot be tuned toward the eight events. All five reference methods are here.
Alerts track
Any method that decides on its own when to alert: a rule, a classifier, a weather-based model, a forecaster's judgment. List when each watch and warning started. Any data available at the time is allowed.
# install pip install gridshedwatch # scores track: my_scores/s0/<grid>.csv with columns p_<signal> in [0, 1] gridshedwatch score my_scores --name "My method" # alerts track: one row per alert, e.g. tx,watch,2021-02-11 03:00 gridshedwatch score-alerts my_alerts.csv --name "My method"
- The budget: about 12 watches and 2 warnings per grid per year (median grid); in the alerts track, also no grid above 3 times that.
- Templates for both tracks are in examples/, and the five reference methods' alerts are published in the alerts format, where they reproduce the table above.
- Random methods submit several seeds; results are averaged.
- To be listed, open a pull request with your scores or alerts, the code that made them, the
--jsonoutput, and short answers on the data used and, for the alerts track, which events you looked at while developing. Every result is added, including ones that beat the reference methods. Methods that use extra data, such as weather, are listed separately.
Limits
- Eight events, seven from two storms (Uri and Elliott). One storm of a new kind could change the picture.
- Three events shaped the shared rules; five were held out and scored once.
- Reference methods see only five signals from EIA-930: no weather, outage or market data. The alerts track is open to methods that use more.
- The baseline covers operators' public notices. Their notices to the industry and their false alarms in ordinary years are not yet compiled.
GridShedWatch grows as new events happen: each load shed that meets the rule is added in a new version and scored with the v1.0 rules.
Cite
Heiserman, S. (2026). GridShedWatch: a benchmark for early warning of grid load sheds (Version 1.1.1). Zenodo. https://doi.org/10.5281/zenodo.23626223
Code: MIT. Event list, timelines, documentation and results: CC BY 4.0. EIA-930 data: U.S. public domain. HTM's entry is run with htm-monitor, scored like any other submission.