Performance Metrics - user guide
- brett gebert
pd4castr 7 DAY Summary
7 DAY Performance is measured for forecasts beyond the predispatch time horizon only.
pd4castr 7d performance metrics page measures the historical accuracy of the latest pd4castr 7d model as of the model’s release date and compares performance with AEMO’s forecasts over the same period. The performance metrics used are Absolute Mean Error and Average Error.
The user also has the option to apply conditions to the metrics by limiting either the rundatetime range, forecast settlement hours and/or actual rrp range.
Table contents
All fields in the table are averages for the sample size shown.
Absolute Mean Error is the average of the magnitude of the difference between pd4castr or AEMO and the actual price outcome. Note that we don’t use mean squared error because price outcomes do not follow a normal distribution and volatile price outliers would otherwise dominate the metric.
Average Error is the average of the difference between pd4castr or AEMO and the actual price outcome.
Average Price provides a frame of reference for the error metrics.
Sample size reflects both the length of time the model has been operating and the filters applied in Options (below).
Note that actual RRP for settlementdates between the half hour are not factored into the calculation.
Options
Options apply filters to the performance metrics.
By default all model forecasts after the release date are included in the metric calculations.
Rundatetime will measure performance for forecasts made between the rundatetimes (inclusive).
By default Min Date is set to the beginning of the pd4castr 7d service not from the release date of the model.
Settlement Date Hours will measure performance for forecast settlementdate hours between Min Hour and Max Hour (inclusive). Note that forecasts on the half hour are included in the calculations. E.g. MinHour=14, MaxHour=20 will include settlementdates from 14:00 to 20:30.
Actual RRP will measure performance only for settlementdates where the actual RRP is between Min RRP and Max RRP.
PD performance metrics
To be released soon. Note that AEMO’s PD forecast errors are currently better than pd4castr when the time horizon is less than 8 hours (give or take). As the forecast time horizon approaches, AEMO’s forecasts generally become more accurate since they hold all of the information that is consumed by the National Energy Market Dispatch Engine (NEMDE). One way to think about this is AEMO’s actual dispatch outcome is effectively a forecast at zero time in the future.