M5 - Nixtla

`M5`

M5(source_url='https://github.com/Nixtla/m5-forecasts/raw/main/datasets/m5.zip')

`M5.download`

download(directory)

Downloads M5 Competition Dataset. Parameters:

Name	Type	Description	Default
`directory`	`str`	Directory path to download dataset.	required

`M5.load`

load(directory, cache=True)

Downloads and loads M5 data. Parameters:

Name	Type	Description	Default
`directory`	`str`	Directory where data will be downloaded.	required
`cache`	`bool`	If `True` saves and loads.	`True`

Returns:

Type	Description
`Tuple[DataFrame, DataFrame, DataFrame]`	Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]: Target time series with columns [‘unique_id’, ‘ds’, ‘y’], Exogenous time series with columns [‘unique_id’, ‘ds’, ‘y’], Static exogenous variables with columns [‘unique_id’, ‘ds’] and static variables.

`M5.source_url`

source_url: str = 'https://github.com/Nixtla/m5-forecasts/raw/main/datasets/m5.zip'

Evaluation class

`M5Evaluation`

`M5Evaluation.aggregate_levels`

aggregate_levels(y_hat, categories=None)

Aggregates the 30_480 series to get 42_840. Parameters:

Name	Type	Description	Default
`y_hat`	`DataFrame`	Forecasts as wide pandas dataframe with columns [‘unique_id’].	required
`categories`	`DataFrame`	Categories of M5 dataset (not used). Defaults to None.	`None`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Aggregated forecasts as wide pandas dataframe with columns [‘unique_id’].

`M5Evaluation.evaluate`

evaluate(directory, y_hat, validation=False)

Evaluates y_hat according to M4 methodology. Parameters:

Name	Type	Description	Default
`directory`	`str`	Directory where data will be downloaded.	required
`validation`	`bool`	Wheter perform validation evaluation. Default False, return test evaluation.	`False`
`y_hat`	`Union[DataFrame, str]`	Forecasts as wide pandas dataframe with columns [‘unique_id’] and forecasts or benchmark url from https://github.com/Nixtla/m5-forecasts/tree/main/forecasts.	required

Returns:

Type	Description
`DataFrame`	pd.DataFrame: DataFrame with columns OWA, SMAPE, MASE and group as index.

Examples:

m5_winner_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0001 YJ_STU.zip'
winner_evaluation = M5Evaluation.evaluate('data', m5_winner_url)

m5_second_place_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0002 Matthias.zip'
m5_second_place_forecasts = M5Evaluation.load_benchmark('data', m5_second_place_url)
second_place_evaluation = M5Evaluation.evaluate('data', m5_second_place_forecasts)

`M5Evaluation.levels`

levels: dict = dict(Level1=['total'], Level2=['state_id'], Level3=['store_id'], Level4=['cat_id'], Level5=['dept_id'], Level6=['state_id', 'cat_id'], Level7=['state_id', 'dept_id'], Level8=['store_id', 'cat_id'], Level9=['store_id', 'dept_id'], Level10=['item_id'], Level11=['state_id', 'item_id'], Level12=['item_id', 'store_id'])

`M5Evaluation.load_benchmark`

load_benchmark(directory, source_url=None, validation=False)

Downloads and loads a bechmark forecasts. Parameters:

Name	Type	Description	Default
`directory`	`str`	Directory where data will be downloaded.	required
`source_url`	`str`	Optional benchmark url obtained from https://github.com/Nixtla/m5-forecasts/tree/master/forecasts. If `None` returns the M5 winner.	`None`
`validation`	`bool`	Wheter return validation forecasts. Default False, return test forecasts.	`False`

Returns:

Type	Description
`ndarray`	np.ndarray: Numpy array of shape (n_series, horizon).

Example:

winner_benchmark = M5Evaluation.load_benchmark('data')
winner_evaluation = M5Evaluation.evaluate('data', winner_benchmark)

URL-based evaluation

The method evaluate from the class M5Evaluation can receive a url of a submission to the M5 competiton. The results compared to the on-the-fly evaluation were obtained from the official evaluation.

m5_winner_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0001 YJ_STU.zip'
winner_evaluation = M5Evaluation.evaluate('data', m5_winner_url)
# Test of the same evaluation as the original one
test_close(winner_evaluation.loc['Total'].item(), 0.520, eps=1e-3)
winner_evaluation

Pandas-based evaluation

Also the method evaluate can recevie a pandas DataFrame of forecasts.

m5_second_place_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0002 Matthias.zip'
m5_second_place_forecasts = M5Evaluation.load_benchmark('data', m5_second_place_url)
second_place_evaluation = M5Evaluation.evaluate('data', m5_second_place_forecasts)
# Test of the same evaluation as the original one
test_close(second_place_evaluation.loc['Total'].item(), 0.528, eps=1e-3)
second_place_evaluation

By default you can load the winner benchmark using the following.

winner_benchmark = M5Evaluation.load_benchmark('data')
winner_evaluation = M5Evaluation.evaluate('data', winner_benchmark)
# Test of the same evaluation as the original one
test_close(winner_evaluation.loc['Total'].item(), 0.520, eps=1e-3)
winner_evaluation

Validation evaluation

You can also evaluate the official validation set.

winner_benchmark_val = M5Evaluation.load_benchmark('data', validation=True)
winner_evaluation_val = M5Evaluation.evaluate('data', winner_benchmark_val, validation=True)
winner_evaluation_val

Kaggle-Competition-M5 References

The evaluation metric of the Favorita Kaggle competition was the normalized weighted root mean squared logarithmic error (NWRMSLE). Perishable items have a score weight of 1.25; otherwise, the weight is 1.0.

NWRMSLE = \sqrt{\frac{\sum^{n}_{i=1} w_{i}\left(log(\hat{y}_{i}+1) - log(y_{i}+1)\right)^{2}}{\sum^{n}_{i=1} w_{i}}}

Kaggle Competition Forecasting Methods	16D ahead NWRMSLE
LGBM [1]	0.5091
Seq2Seq WaveNet [2]	0.5129

​

​M5

​M5.download

​M5.load

​M5.source_url

​Evaluation class

​M5Evaluation

​M5Evaluation.aggregate_levels

​M5Evaluation.evaluate

​M5Evaluation.levels

​M5Evaluation.load_benchmark

​URL-based evaluation

​Pandas-based evaluation

​Validation evaluation

​Kaggle-Competition-M5 References

`M5`

`M5.download`

`M5.load`

`M5.source_url`

Evaluation class

`M5Evaluation`

`M5Evaluation.aggregate_levels`

`M5Evaluation.evaluate`

`M5Evaluation.levels`

`M5Evaluation.load_benchmark`

URL-based evaluation

Pandas-based evaluation

Validation evaluation

Kaggle-Competition-M5 References