Run #15 success proof_build_manual

Run detail

Monitor the academic pipeline and ML actions from one page.

Open Grafana Back to batch All runs

Started

2026-03-20 14:22

Finished

2026-03-20 14:22

Assets processed

Days processed

3000

Operator guide

Model run #15 is available. Next actions: Validate, Backtest, then Forecast.

Stage flow

Each stage shows its summary, errors, and payload details.

Load

success

Reused existing load output

source	reused_existing_artifacts
source_batch_id	11

→ 2026-03-19 20:59:02

Cycles / Baseline / FSI

success

Reused existing cycle / baseline / FSI output

source	reused_existing_artifacts
source_batch_id	11

→ 2026-03-19 20:59:02

Features / TFT Dataset

success

Reused existing feature / TFT dataset

source	reused_existing_artifacts
source_batch_id	11

→ 2026-03-19 20:59:02

ML Train

success

TFT training completed

id	15
params	{'dropout': 0.1, 'val_days': 90, 'batch_size': 64, 'max_epochs': 40, 'hidden_size': 64, 'feature_spec': 'v1_log1p_fsi', 'learning_rate': 0.001, 'attention_heads': 4, 'target_transform': 'log1p', 'max_encoder_length': 60, 'max_prediction_length': 30}
metrics	{'rows': 2662, 'qa_summary': {'rows': 2662, 'assets': 3, 'by_asset': [{'rows': 831, 'fsi_max': 3.690835432401493, 'fsi_min': 0.0, 'time_max': 904, 'time_min': 0, 'asset_code': 'E102A', 'neg_raw_fsi': 0}, {'rows': 1000, 'fsi_max': 29.31267248493691, 'fsi_min': 0.0, 'time_max': 1080, 'time_min': 0, 'asset_code': 'E102B', 'neg_raw_fsi': 0}, {'rows': 831, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 904, 'time_min': 0, 'asset_code': 'E102C', 'neg_raw_fsi': 0}], 'time_idx_max': 1080, 'time_idx_min': 0, 'top_null_counts': {'day': 0, 'fsi': 0, 'flow': 0, 'day_cos': 0, 'day_sin': 0, 'delta_p': 0, 'baseline': 0, 'cycle_id': 0, 'time_idx': 0, 'is_weekend': 0, 'cleaning_event': 0, 'pdi_normalized': 0, 'steam_feed_ratio': 0, 'system_t_skin_avg': 0, 'system_t_skin_max': 0, 'total_vc5_rate_m3h': 0, 'days_since_cleaning': 0, 'furnaces_in_service': 0, 'total_ngl_rate_km3h': 0, 'pws_feedwater_rate_m3h': 0}, 'insufficient_assets': [], 'negative_target_count': 0, 'negative_raw_fsi_count': 0}, 'max_time_idx': 1080, 'cutoff_time_idx': 990, 'best_model_score': 0.09676852822303772, 'target_transform': 'log1p', 'zero_target_count': 1703, 'negative_raw_fsi_count': 0}
checkpoint_path	/app/models/run_15_modelrun_15/best.ckpt

2026-03-19 21:01:07 → 2026-03-19 21:55:24

Validate

success

TFT validation completed

error

no_forecasts

2026-03-19 21:40:19 → 2026-03-19 21:40:19

Backtest

success

TFT backtest completed

rows	2662
assets	3
h1_mae	0.0864828337030369
h7_mae	0.026710458619044822
h14_mae	0.02669476674960941
h1_rmse	0.16008918882063614
h30_mae	0.20727598128904784
h7_rmse	0.04708550783194005
h14_rmse	0.049185168948107795
h1_smape	193.68154647511076
h30_rmse	0.2775245149431807
h7_smape	190.27469004843724
h14_smape	193.92238158594304
h30_smape	125.61743897381616
used_cutoffs	[1027, 1034, 1041, 1048]
skipped_cutoffs	[{'cutoff': 1020, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}]
num_used_cutoffs	4
requested_cutoffs	[1020, 1027, 1034, 1041, 1048]
negative_pred_rate	0.0
num_prediction_rows	16
num_skipped_cutoffs	1
num_requested_cutoffs	5

2026-03-19 21:41:17 → 2026-03-19 21:41:21

Forecast

success

TFT forecast generation completed

rows_written	12
assets_predicted	3
forecast_end_day	2024-07-29
forecast_start_day	2024-06-30

2026-03-20 14:22:48 → 2026-03-20 14:22:51

Run parameters

mode	proof_build_manual
source	ui
batch_id	11
latest_model_params	{'dropout': 0.1, 'val_days': 90, 'batch_size': 64, 'max_epochs': 40, 'hidden_size': 64, 'feature_spec': 'v1_log1p_fsi', 'learning_rate': 0.001, 'attention_heads': 4, 'target_transform': 'log1p', 'max_encoder_length': 60, 'max_prediction_length': 30}
latest_model_run_id	15
latest_forecast_rows_written	12
latest_forecast_assets_predicted	3

Recent events

STAGE_FORECAST_SUCCESS

2026-03-20 14:22:51

TFT forecast generation completed

STAGE_FORECAST_START

2026-03-20 14:22:48

TFT forecast generation started

UI_FORECAST_START

2026-03-20 14:22:48

Forecast action started from UI.

STAGE_TRAIN_SUCCESS

2026-03-19 21:55:24

TFT training completed

STAGE_TRAIN_SUCCESS

2026-03-19 21:48:53

TFT training completed

STAGE_FORECAST_SUCCESS

2026-03-19 21:42:20

TFT forecast generation completed

STAGE_FORECAST_START

2026-03-19 21:42:11

TFT forecast generation started

UI_FORECAST_START

2026-03-19 21:42:11

Forecast action started from UI.

STAGE_BACKTEST_SUCCESS

2026-03-19 21:41:21

TFT backtest completed

STAGE_BACKTEST_START

2026-03-19 21:41:17

TFT backtest started

UI_BACKTEST_START

2026-03-19 21:41:17

Backtest action started from UI.

STAGE_VALIDATE_SUCCESS

2026-03-19 21:40:19

TFT validation completed

STAGE_VALIDATE_START

2026-03-19 21:40:19

TFT validation started

UI_VALIDATE_START

2026-03-19 21:40:19

Validate action started from UI.

STAGE_TRAIN_SUCCESS

2026-03-19 21:33:56

TFT training completed

STAGE_TRAIN_START

2026-03-19 21:01:07

TFT training started

UI_TRAIN_START

2026-03-19 21:01:07

Train action started from UI.

STAGE_TRAIN_START

2026-03-19 21:00:08

TFT training started

UI_TRAIN_START

2026-03-19 21:00:08

Train action started from UI.

STAGE_TRAIN_START

2026-03-19 20:59:02

TFT training started

STAGE_FEATURES_REUSED

2026-03-19 20:59:02

Reused existing feature / TFT dataset

STAGE_CYCLES_REUSED

2026-03-19 20:59:02

Reused existing cycle / baseline / FSI output

STAGE_LOAD_REUSED

2026-03-19 20:59:02

Reused existing load output

UI_TRAIN_START

2026-03-19 20:59:01

Train action started from UI.

Model runs

ID	Status	Checkpoint
#15	success	/app/models/run_15_modelrun_15/best.ckpt
#14	success	/app/models/run_15_modelrun_14/best.ckpt
#13	success	/app/models/run_15_modelrun_13/best.ckpt

Latest model details

Model run	#15
Status	success
Checkpoint	/app/models/run_15_modelrun_15/best.ckpt

Training params

dropout	0.1
val_days	90
batch_size	64
max_epochs	40
hidden_size	64
feature_spec	v1_log1p_fsi
learning_rate	0.001
attention_heads	4
target_transform	log1p
max_encoder_length	60
max_prediction_length	30

Training metrics

rows	2662
qa_summary	{'rows': 2662, 'assets': 3, 'by_asset': [{'rows': 831, 'fsi_max': 3.690835432401493, 'fsi_min': 0.0, 'time_max': 904, 'time_min': 0, 'asset_code': 'E102A', 'neg_raw_fsi': 0}, {'rows': 1000, 'fsi_max': 29.31267248493691, 'fsi_min': 0.0, 'time_max': 1080, 'time_min': 0, 'asset_code': 'E102B', 'neg_raw_fsi': 0}, {'rows': 831, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 904, 'time_min': 0, 'asset_code': 'E102C', 'neg_raw_fsi': 0}], 'time_idx_max': 1080, 'time_idx_min': 0, 'top_null_counts': {'day': 0, 'fsi': 0, 'flow': 0, 'day_cos': 0, 'day_sin': 0, 'delta_p': 0, 'baseline': 0, 'cycle_id': 0, 'time_idx': 0, 'is_weekend': 0, 'cleaning_event': 0, 'pdi_normalized': 0, 'steam_feed_ratio': 0, 'system_t_skin_avg': 0, 'system_t_skin_max': 0, 'total_vc5_rate_m3h': 0, 'days_since_cleaning': 0, 'furnaces_in_service': 0, 'total_ngl_rate_km3h': 0, 'pws_feedwater_rate_m3h': 0}, 'insufficient_assets': [], 'negative_target_count': 0, 'negative_raw_fsi_count': 0}
max_time_idx	1080
cutoff_time_idx	990
best_model_score	0.09676852822303772
target_transform	log1p
zero_target_count	1703
negative_raw_fsi_count	0