Forecasting

Scheduling is about the future, and you need some knowledge / expectations about the future to do it.

In FlexMeasures, this knowledge often comes in the form of forecasts — data-driven estimates of what’s likely to happen next.

https://github.com/FlexMeasures/screenshots/raw/main/tut/PV-forecasting-example.png

Example of a 24-hour horizon forecast for solar power production.

Of course, the nicest forecasts are the ones you don’t have to make yourself (it’s not an easy field), so do use price or usage forecasts from third parties if available, and load them into FlexMeasures. There are even existing plugins for importing weather forecasts or market data.

If you need to make your own predictions, forecasting algorithms can be used within FlexMeasures, for instance, to arrive at an expected profile of future solar power production at the site.

FlexMeasures provides a CLI command and an API endpoint to generate forecasts (see below).

FlexMeasures provides a fixed viewpoint forecasting infrastructure. This means that from one point in time (the fixed viewpoint), we forecast a range of events into the future (e.g. 24 hourly events for a span of one day). While the first forecast (one hour ahead) has a small horizon (1H), the last one has a large horizon (24H) and the accuracy between the two will usually differ (it is easier to forecast small horizons).

At the same time, the design we implemented in FlexMeasures is inspired by rolling forecasts, as training and prediction can be repeated in cycles until a user-specified end date is reached. If you ask FlexMeasures for a fixed viewpoint forecast (one cycle), the model is trained once on the most recent applicable historical period and then produces predictions for the requested future period in one go. This is controlled by the forecast-frequency parameter, which specifies how often predictions are generated during the forecast period.

How a forecasting cycle works

A single forecasting cycle consists of the following steps:

Training: Fit the model on a historical window defined by train-start and train-end.
Prediction: Produce forecasts for a time window defined by predict-start and predict-end.
Repeat: Extend the training window and move the prediction window forward, both by forecast-frequency.

Cycles repeat until predict-end reaches the global to-date. This way, forecasts can cover long ranges while still being based on updated training data in each cycle.

CLI Command

You can create forecasts from the command line using:

flexmeasures add forecasts --from-date 2024-02-02 --to-date 2024-02-02 --max-forecast-horizon 6 --sensor 12 --as-job

This command asks FlexMeasures to generate forecasts for one day (2 February 2024) with a forecast horizon of 6 hours for the sensor with ID 12. If you include --as-job, the forecasting task is added to the job queue to be processed by a worker.

The main CLI parameters that control this process are:

from-date: Defines the first predict-start.
to-date: The global cutoff point. Training and prediction cycles continue until the predict-end reaches this date.
max-forecast-horizon: The maximum length of a forecast into the future.
forecast-frequency: Determines the number of prediction cycles within the forecast period (e.g. daily, hourly).
train-period: Define a window of historical data to use for training.

Note that:

forecast-frequency together with max-forecast-horizon determine how the forecasting cycles advance through time. train-period, from-date and to-date allow precise control over the training and prediction windows in each cycle.

Forecast post-processing

Forecaster configuration can clip and snap forecast values before they are stored. Use lower and upper to clip values to bounds, and snap to replace values inside a configured interval with a target value. Units are optional; unitless values are interpreted in the output sensor unit.

For example, this configuration clips forecasts to the 0-20 kW range and snaps values in [0 kW, 4 kW) to 0 kW:

{
  "lower": "0 kW",
  "snap": {
    "0 kW": ["0 kW", "4 kW"]
  },
  "upper": "20 kW"
}

The snap target must lie within its interval (on a bound or inside it), so values never snap to a value outside the interval. A target inside the interval snaps the whole band to that value, for example {"2 kW": ["0 kW", "4 kW"]} snaps everything in [0 kW, 4 kW) to 2 kW. The first bound is inclusive and the second is exclusive, so an interval reads as [first, second). Listing the bounds in reverse order flips the closed side: ["10 kW", "4 kW"] means (4 kW, 10 kW]. This keeps adjacent intervals unambiguous — a value on a shared boundary belongs to the interval that opens at it. For instance, given {"0 kW": ["0 kW", "4 kW"], "10 kW": ["4 kW", "10 kW"]}, a forecast of exactly 4 kW snaps to 10 kW.

Snapping runs first and clipping runs afterwards, so lower/upper always take precedence: a snap target outside the bounds is clipped back into range.

Pass the same object as the API config payload, or place it in a JSON or YAML file and pass it to flexmeasures add forecasts with --config.

Forecasting via the UI

The quickest way to create a one-off forecast is the Create forecast button on the sensor page (see Creating a forecast). The button is available to users with permission to record data on sensors, provided at least two days of historical data exist. The forecast duration defaults to 48 hours (configured via FLEXMEASURES_PLANNING_HORIZON) but can be adjusted up to 7 days in the panel. No further configuration is needed — one click queues the job and the page shows progress messages until the forecast is ready.

For more control over what and how to forecast, use the API.

Forecasting via the API

In addition to the CLI command, FlexMeasures provides API endpoints for triggering forecasts and retrieving their results.

These endpoints live under the Sensor API (/api/v3_0/sensors).

A typical workflow is:

Trigger a forecasting job for a sensor.
Poll the job until it finishes.
Retrieve the forecast results.

For the exact API endpoints, parameters, and response formats, refer to the API v3 documentation:

or try the endpoints interactively with the Swagger UI at /api/v3_0/docs.

Technical specs

In a nutshell, FlexMeasures uses a LightGBM regression model (darts.models.LightGBMModel) as its base model to forecast future values.

Note that the most important factor is often the features provided to the model ― lagged values (e.g., the value at the same time yesterday) and regressors (e.g., wind speed prediction to forecast wind power production). Most assets have yearly seasonality (e.g. wind, solar) and therefore forecasts benefit from at least two years of historical data.

Here are more details:

The main model is a LightGBM model, which can be wrapped to produce probabilistic forecasts if required.
Lagged outcome variables are selected based on the periodicity of the asset (e.g. hourly, daily and/or weekly).
Missing data is filled using linear interpolation (via the Darts MissingValuesFiller, which wraps pandas.DataFrame.interpolate).
The model is trained once per cycle for each asset and can forecast up to the maximum forecast horizon in a single run.
Forecasts are fixed viewpoint forecasts — the model is trained on a given history and produces predictions for a future window in one go. Training and prediction can then be repeated in cycles until a user-specified end date is reached. This cycle-based design is inspired by rolling forecasts while keeping a fixed viewpoint. The forecast-frequency parameter controls how often predictions are generated during the forecast period.

A use case: automating solar production prediction

We’ll consider an example that FlexMeasures supports ― forecasting an asset that represents solar panels. Here is how you can ask for forecasts to be made in the CLI:

flexmeasures add forecasts --from-date 2024-02-02 --to-date 2024-02-02 --max-forecast-horizon 6 --sensor 12  --as-job  # add train-start

Sensor 12 would represent the power readings of your solar power, and here you ask for forecasts for one day (2 February, 2024), with a forecast of 6 hours.

The --as-job parameter is optional. If given, the computation becomes a job which a worker needs to pick up. There is some more information at How forecasting jobs are queued.

Fixed viewpoint vs rolling viewpoint

Unlike previous rolling forecasts, where each prediction covers the same relative forecast horizon (but the origin keeps moving forward), the new infrastructure generates fixed viewpoint forecasts:

One reference timestamp.
Predictions are made for multiple future horizons from that point.
Periodic retraining ensures forecasts remain accurate.

Regressors

If you want to take regressors into account, in addition to merely past measurements (e.g. weather forecasts, see above).

past regressors : sensors that only have realizations (historical data).
future regressors : sensors that only have forecasts (e.g. weather forecasts).
regressors : sensors that have both historical data and forecasts (e.g. weather forecasts).

Including regressors can significantly improve forecasting accuracy, especially when they are highly correlated with the target variable. For example, using irradiation forecasts as regressors can substantially improve solar production predictions. In this weather forecast plugin, we enable you to collect regressor data for ["temperature", "wind speed", "cloud cover", "irradiance"], at a location you select.

Automating forecasts

Instead of asking for forecasts one at a time, you can set up an automation: a recurring task defined on an asset. On each run, the automation queues forecasting jobs (so make sure a worker is processing the forecasting queue, see Redis Queues). When the automation was created, its forecast parameters (see above) were stored, and validated with the same schema that the CLI and API use. Timing parameters are resolved on each run — for instance, the forecast start defaults to the time the automation runs, so each run produces fresh forecasts.

Here is how you create an automation in the CLI, asking for daily (at 6 AM) forecasts of sensor 12:

flexmeasures add automation --asset 3 --name "Daily PV forecasts" --cron "0 6 * * *" --sensor 12

The recurrence is defined by a cron string (interpreted in the FLEXMEASURES_TIMEZONE). Automations are active by default (use --inactive to create them in deactivated state). Use flexmeasures edit automation to rename, re-schedule (--cron), activate or deactivate an automation, and flexmeasures delete automation to remove one. These changes are recorded in the asset’s audit log.

For automations to actually run, let a cron job execute the following command once per minute:

* * * * * flexmeasures jobs run-automations

Each due automation then queues its forecasting jobs. The jobs record how they were created, which is shown on the asset’s status page (UI), where recent jobs are listed.

The runner remembers (in Redis) the last minute it processed. If your cron job misses some minutes (e.g. due to host downtime or overload), the next invocation catches up: it also matches automations against the missed minutes, up to a maximum catch-up window (--max-catchup, 60 minutes by default; set it to 0 to disable catching up). An automation that was due in several missed minutes still runs only once per invocation — its timing parameters (such as the forecast start) are resolved against the current time, so queueing multiple identical jobs would be wasteful. Minutes missed for longer than the catch-up window are skipped.

Automations defined on an asset can be viewed on the asset’s Automations page in the UI, and listed with the API endpoint [GET] /assets/(id)/automations.