Skip to content

Entity-level evaluation

This section documents evaluation utilities for computing metrics and diagnostics at the individual entity level.

Entity-level evaluation supports analysis of forecast performance and readiness for a single unit (e.g. store, item, or asset) over time.

eb_evaluation.dataframe.entity

Entity-level evaluation utilities (DataFrame helpers).

This module contains DataFrame-oriented evaluation helpers that operate on panel data: entity-by-interval observations containing actuals and forecasts.

The primary helper evaluates each entity using entity-specific cost asymmetry parameters (cost ratios) that are typically estimated upstream (for example, via a balance-based estimator). The result is one row per entity containing cost-weighted and service-oriented Electric Barometer metrics, plus familiar symmetric error metrics.

evaluate_panel_with_entity_R(df, entity_R, *, entity_col='entity', y_true_col='actual_qty', y_pred_col='forecast_qty', R_col='R', co_col='co', tau=2.0, sample_weight_col=None)

Evaluate an entity-interval panel using entity-level cost ratios.

This helper evaluates each entity slice of a panel using entity-specific cost asymmetry parameters. It is designed to pair naturally with a table that provides, for each entity, a cost ratio:

\[ R_e = \frac{c_{u,e}}{c_o} \]

and an overbuild cost coefficient \(c_o\). For each entity \(e\), the implied shortfall (underbuild) cost coefficient is:

\[ c_{u,e} = R_e \cdot c_o \]
Evaluation flow

For each entity:

  1. Join the entity-level values \((R_e, c_o)\) onto all intervals for that entity.
  2. Construct per-row arrays \(c_u\) and \(c_o\) that are constant within the entity slice.
  3. Compute the EB metric suite (CWSL, NSL, UD, HR@\(\tau\), FRS) using the entity-specific cost parameters, plus common symmetric metrics.

Parameters:

Name Type Description Default
df DataFrame

Panel of interval-level data containing at least:

  • entity_col (entity identifier)
  • y_true_col (actuals)
  • y_pred_col (forecasts)

If sample_weight_col is provided, it must also exist in df.

 required
entity_R DataFrame

Table with one row per entity containing at least:

  • entity_col (entity identifier)
  • R_col (cost ratio \(R_e\))
  • co_col (overbuild cost coefficient \(c_o\))

Typically produced by an upstream calibration step (for example, an entity ratio estimator).

 required
entity_col str

Column identifying the entity (for example, "store_id", "sku", "location").

 "entity"
y_true_col str

Column containing realized demand / actual values.

 "actual_qty"
y_pred_col str

Column containing baseline forecast values.

 "forecast_qty"
R_col str

Column in entity_R containing the cost ratio \(R_e\).

 "R"
co_col str

Column in entity_R containing the overbuild cost coefficient \(c_o\).

 "co"
tau float

Absolute-error tolerance parameter for the hit-rate metric HR@\(\tau\).

 2.0
sample_weight_col str | None

Optional column in df of non-negative sample weights.

 None

Returns:

Type Description
DataFrame

One row per entity with columns:

  • entity_col : entity identifier
  • R : entity ratio \(R_e\)
  • cu : implied underbuild cost coefficient \(c_{u,e} = R_e \cdot c_o\)
  • co : overbuild cost coefficient \(c_o\)
  • CWSL : cost-weighted service loss
  • NSL : no-shortfall level (service-oriented)
  • UD : underbuild depth
  • wMAPE : weighted mean absolute percentage error (per eb_metrics definition)
  • HR@tau : hit rate within tolerance \(\tau\)
  • FRS : forecast readiness score
  • MAE : mean absolute error
  • RMSE : root mean squared error
  • MAPE : mean absolute percentage error

If a metric is undefined for a given entity slice (for example, due to a metric-specific validation failure), that metric value is returned as NaN for that entity.

Raises:

Type Description
KeyError

If required columns are missing from df or entity_R.
ValueError

If the merge between df and entity_R produces no rows (no overlapping entities).
Notes
  • The join uses an inner merge on entity_col. Entities present in df but missing from entity_R are dropped. This is intentional: evaluation requires cost parameters.
  • Cost arrays are constructed per entity as constants, enabling vectorized evaluation calls.
  • Some metrics in :mod:eb_metrics.metrics may not accept sample weights; this function calls those metrics unweighted to match their signatures.