CODEWITHSUNDEEP
amazon-web-servicesamazon-forecast
Khalid Hamim
Khalid Hamim

Reputation: 1

AWS backtest what does 'Summary' and 'Computed' mean in backtest exports

In the accuracy-metric-values folder that is created by the export job I usually get 2 to 3 files, part0 is always empty with only column names (I wonder why?) for the other ones I noticed that the column 'backtest_window' takes two values either 'Summary' or 'Computed' what do they mean?

Upvotes: 0

Views: 63

Answers (1)

dingus
dingus

Reputation: 1001

In cases where you've configured multiple backtest windows (which as far as I know is only possible with legacy predictors but not the newer AutoPredictors), you'll see separate metrics Computed for each test window. Forecast then aggregates these together to give Summary metrics across all the test windows, for a nice overall description of model performance.

So in short the Summary is probably what you want if you're looking for a single view of metrics, with the Computed being useful if you're using a legacy predictor and trying to dive in to how performance changed between different backtest windows.

I believe the output files are always sharded because there's some parallel processing being used under the hood for scalability... As for the empty file, maybe the dataset just wasn't big enough to utilize all the workers? But really your guess is as good as mine 😅

Upvotes: 0

Related Questions