Results for 6th June 2023 release
Summary
The current model consumed training data up until 30th March 2023. The retraining process to update the model included a months worth of data after the retirement of Liddell. We also conducted many experiments including changing the feature set (input variables) as well as boosting (weighting) more recent training periods.
Much to our surprise the performance metrics, when measured against the validation set (a weeks worth of recent data not used in the training set), demonstrated that the original model either performed better or more or less equal to all new models. As a result we felt it best to preserve the consistency afforded by the current model hence no update to the model was made for the 6th of June.
Comments on why retraining did not result in a better model
There are a number of reasons why the original model performed better than all other retrained models:
The validation set (which is the test period used to assess performance) may have contained unexpected market outcomes, particularly contingencies in which case the validation set was not representative of “normal” market dynamics.
The market dynamics post Lidell retirement were volatile and undergoing shifts in behaviour in which case the model was being trained (and weighted) to include a volatile period which manifests itself in overstating forecast prices after the volatility died down.
Retraining schedule
Retraining a model is scheduled every 2 months or after 3 or 4 weeks after a significant change in market dynamics. The current retraining schedule for both PD and P7DAY follows:
Retrained model release dates | Retraining process and model changes |
|---|---|
6th June 2023 | No material improvements to the model could be made, hence the model has not changed. |
1st August 2023 | |
Maybe October 2023 |
Retraining an existing model
Once a model is released to pd4castr it will be periodically retrained to include the most recent historical data. Hyper parameters and weighting variables used by the MLA in the training process may also be adjusted. Note that the model_id will not change.
Training a new model
If a model is released that was trained using new features (input training variables) or that uses a different MLA then a new model will be released with a new model_id.