8th August 2023 release
Summary
The retraining process resulted in a new models, one for PD and one for 7day price forecast. After considerable experimentation we retrained the price forecasting models with a considerable increase in accuracy (see validation results table below comparing the previous models with the new models).
Training
Both PD and 7day price forecast models were trained on data between 01-10-2020 and 20-08-2023.
The market suspension period and a period after the Callide explosion were removed from the training data.
New features (input data) exceptions
A new “recent prices feature” was added to give the model some context for the current pricing environment. This feature is the median price for a window around the forecast time for the most recent three days. For example, for a forecast occurring in 2 days at 11am, the model generated the median value for prices between 10am-12pm for the three days before the forecast was run.
A number of additional experiments were conducted on the 7 day forecast model, with the major modifications including:
The 50th percent probability of demand features were augmented with 10th percent probability of demand data
The new recent prices features mentioned previously
A new time of day feature
A new weekday/weekend feature.
e Cup did not have a significant impact on the performance metrics.
Predispatch model | p7day model | |
|---|---|---|
Model_id | 20230729_213406_CB_2 | 20230213_012319_CB_7D_1 |
Previous model release date | 20th April 2023 | 20th April 2023 |
Latest release date (after retraining) | 8th August 2023 | 8th August 2023 |
Machine Learning Algorithm | Ensemble predictor: gradient boosting | Ensemble predictor: gradient boosting |
Training Period | 1st October 2020 to 20th July 2023 | 1st October 2020 to 20th July 2023 |
Hyper-parameter type | A | A |
Hyper-parameter notes | No weighting across time periods | No weighting across time periods |
Target | Dispatchprice (30min resolution) | Dispatchprice (30min resolution) |
Input (and training) variables | PREDISPATCHPRICE rrp DISPATCHPRICE last 3 day average actual RRP for same time PREDISPATCHREGIONSUM (intervention = 0) hours_out time_of_day day_type (weekday_or_weekend) dispatchableload netinterchage (availablegeneration - uigf) (dispatchablegeneration - scheduled_clearedmw) ss_solar_clearedmw ss_solar_uigf ss_wind_clearedmw ss_wind_uigf totaldemand totalintermittentgeneration | STPASA_REGIONSOLUTION hours_out time_of_day day_type (weekday_or_weekend) aggregatepasaavailability aggregatescheduledload netinterchangeunderscarcity aggregatecapacityavailable ss_solar_cleared ss_wind_uigf ss_wind_cleared ss_solar_uigf demand10 demand50 totalintermittentgeneration demand_and_nonschedgen |
Validation Results
These are the results of the best model generated through our experimentation process and compares the results with the previous model across our validation set (our test period).
Average error is equal to Predicted Prices minus Actual Prices.
All error metrics are in $/MWhr.
PD Results
Metric/Region | Absolute Mean Error New Model | Absolute Mean Error Previous Model | Average Error New Model | Average Error Previous Model |
|---|---|---|---|---|
NSW1 | 37.16954 | 63.55691 | 19.63544 | 38.2957 |
VIC1 | 29.91769 | 37.21212 | 6.468355 | 2.998254 |
QLD1 | 36.46522 | 132.3778 | 15.79482 | 112.1593 |
SA1 | 45.14167 | 65.13762 | 19.56986 | 35.5676 |
TAS1 | 19.53866 | 17.71054 | 9.396361 | 6.877853 |
7 Day Results
Metric/Region | Absolute Mean Error New Model | Absolute Mean Error Previous Model | Average Error New Model | Average Error Previous Model |
|---|---|---|---|---|
NSW1 | 34.06431 | 48.10104 | 15.01693 | 37.08693 |
VIC1 | 31.66891 | 44.83715 | 1.215206 | 19.20936 |
QLD1 | 39.10014 | 53.91976 | 17.32682 | 35.32128 |
SA1 | 45.46252 | 104.0136 | 21.62486 | 78.54663 |
TAS1 | 24.07958 | 45.70449 | 14.92562 | 39.71673 |
Variables that imply binding interconnectors (such as limits and marginal values) are currently not included as features.
Previous release history
6th June 2023
Summary
The current model was trained using data up until 30th March 2023. Retraining the model included a months data after the retirement of Liddell. We also conducted many experiments by changing the feature set (input variables) as well as boosting (weighting) more recent training periods.
Much to our surprise the performance of all retrained and new models demonstrated that the original model either performed better or more or less equal to all new models when measured against the validation set (which was a week of recent data not used in the training set). As a result we felt it best to keep the current model. Therefore we did not change the model as scheduled for the 6th of June.
Why retraining did not result in a better model (for 6th June)
There are a number of reasons why the original model performed better than all other retrained models however we believe that the this was due to the volatile market dynamics post Liddell retirement and that the market behaviour has since settled. Models that included the weeks after the retirement of Liddell are less representative of current market dynamics and now overstate forecast prices relative to actual market outcomes.