Fouling AI
Run #17 success proof_build_manual

Run detail

Monitor the academic pipeline and ML actions from one page.

Open Grafana Back to batch All runs
Started
2026-03-22 03:22
Finished
2026-03-22 03:22
Assets processed
3
Days processed
3000
Operator guide
Model run #17 is available. Next actions: Validate, Backtest, then Forecast.

Stage flow

Each stage shows its summary, errors, and payload details.

Load
success

Reused existing load output

sourcereused_existing_artifacts
source_batch_id13

→ 2026-03-22 02:17:18

Cycles / Baseline / FSI
success

Reused existing cycle / baseline / FSI output

sourcereused_existing_artifacts
source_batch_id13

→ 2026-03-22 02:17:18

Features / TFT Dataset
success

Reused existing feature / TFT dataset

sourcereused_existing_artifacts
source_batch_id13

→ 2026-03-22 02:17:18

ML Train
success

TFT training completed

id17
params{'dropout': 0.1, 'val_days': 90, 'batch_size': 64, 'max_epochs': 40, 'hidden_size': 64, 'feature_spec': 'v1_log1p_fsi', 'learning_rate': 0.001, 'attention_heads': 4, 'target_transform': 'log1p', 'max_encoder_length': 60, 'max_prediction_length': 30}
metrics{'rows': 2662, 'qa_summary': {'rows': 2662, 'assets': 3, 'by_asset': [{'rows': 831, 'fsi_max': 3.690835432401493, 'fsi_min': 0.0, 'time_max': 904, 'time_min': 0, 'asset_code': 'E102A', 'neg_raw_fsi': 0}, {'rows': 1000, 'fsi_max': 29.31267248493691, 'fsi_min': 0.0, 'time_max': 1080, 'time_min': 0, 'asset_code': 'E102B', 'neg_raw_fsi': 0}, {'rows': 831, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 904, 'time_min': 0, 'asset_code': 'E102C', 'neg_raw_fsi': 0}], 'time_idx_max': 1080, 'time_idx_min': 0, 'top_null_counts': {'day': 0, 'fsi': 0, 'flow': 0, 'day_cos': 0, 'day_sin': 0, 'delta_p': 0, 'baseline': 0, 'cycle_id': 0, 'time_idx': 0, 'is_weekend': 0, 'cleaning_event': 0, 'pdi_normalized': 0, 'steam_feed_ratio': 0, 'system_t_skin_avg': 0, 'system_t_skin_max': 0, 'total_vc5_rate_m3h': 0, 'days_since_cleaning': 0, 'furnaces_in_service': 0, 'total_ngl_rate_km3h': 0, 'pws_feedwater_rate_m3h': 0}, 'insufficient_assets': [], 'negative_target_count': 0, 'negative_raw_fsi_count': 0}, 'max_time_idx': 1080, 'cutoff_time_idx': 990, 'best_model_score': 0.09676852822303772, 'target_transform': 'log1p', 'zero_target_count': 1703, 'negative_raw_fsi_count': 0}
checkpoint_path/app/models/run_17_modelrun_17/best.ckpt

2026-03-22 02:17:18 → 2026-03-22 02:27:38

Validate
success

TFT validation completed

errorno_forecasts

2026-03-22 03:21:47 → 2026-03-22 03:21:47

Backtest
success

TFT backtest completed

rows2662
assets3
h1_mae0.09150402341526905
h7_mae0.03168751260114098
h14_mae0.034220284396271755
h1_rmse0.1650415408253389
h30_mae0.32511792757666597
h7_rmse0.04881575514333678
h14_rmse0.049535170933054956
h1_smape194.8174410178323
h30_rmse0.37289611647379955
h7_smape191.525406798097
h14_smape188.7888050177969
h30_smape157.97227682359417
used_cutoffs[1027, 1034, 1041, 1048]
skipped_cutoffs[{'cutoff': 1020, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}]
num_used_cutoffs4
requested_cutoffs[1020, 1027, 1034, 1041, 1048]
negative_pred_rate0.0625
num_prediction_rows16
num_skipped_cutoffs1
num_requested_cutoffs5

2026-03-22 03:22:07 → 2026-03-22 03:22:11

Forecast
success

TFT forecast generation completed

rows_written12
assets_predicted3
forecast_end_day2024-07-29
forecast_start_day2024-06-30

2026-03-22 03:22:33 → 2026-03-22 03:22:36

Run parameters

modeproof_build_manual
sourceui
batch_id13
latest_model_params{'dropout': 0.1, 'val_days': 90, 'batch_size': 64, 'max_epochs': 40, 'hidden_size': 64, 'feature_spec': 'v1_log1p_fsi', 'learning_rate': 0.001, 'attention_heads': 4, 'target_transform': 'log1p', 'max_encoder_length': 60, 'max_prediction_length': 30}
latest_model_run_id17
latest_backtest_summary{'rows': 2662, 'assets': 3, 'h1_mae': 0.09150402341526905, 'h7_mae': 0.03168751260114098, 'h14_mae': 0.034220284396271755, 'h1_rmse': 0.1650415408253389, 'h30_mae': 0.32511792757666597, 'h7_rmse': 0.04881575514333678, 'h14_rmse': 0.049535170933054956, 'h1_smape': 194.8174410178323, 'h30_rmse': 0.37289611647379955, 'h7_smape': 191.525406798097, 'h14_smape': 188.7888050177969, 'h30_smape': 157.97227682359417, 'used_cutoffs': [1027, 1034, 1041, 1048], 'skipped_cutoffs': [{'cutoff': 1020, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}], 'num_used_cutoffs': 4, 'requested_cutoffs': [1020, 1027, 1034, 1041, 1048], 'negative_pred_rate': 0.0625, 'num_prediction_rows': 16, 'num_skipped_cutoffs': 1, 'num_requested_cutoffs': 5}
latest_validation_summary{'error': 'no_forecasts'}
latest_forecast_rows_written12
latest_forecast_assets_predicted3

Recent events

STAGE_FORECAST_SUCCESS
2026-03-22 03:22:36

TFT forecast generation completed

STAGE_FORECAST_START
2026-03-22 03:22:33

TFT forecast generation started

UI_FORECAST_START
2026-03-22 03:22:33

Forecast action started from UI.

STAGE_BACKTEST_SUCCESS
2026-03-22 03:22:11

TFT backtest completed

STAGE_BACKTEST_START
2026-03-22 03:22:07

TFT backtest started

UI_BACKTEST_START
2026-03-22 03:22:07

Backtest action started from UI.

STAGE_VALIDATE_SUCCESS
2026-03-22 03:21:47

TFT validation completed

STAGE_VALIDATE_START
2026-03-22 03:21:47

TFT validation started

UI_VALIDATE_START
2026-03-22 03:21:47

Validate action started from UI.

STAGE_TRAIN_SUCCESS
2026-03-22 02:27:38

TFT training completed

STAGE_TRAIN_START
2026-03-22 02:17:18

TFT training started

STAGE_FEATURES_REUSED
2026-03-22 02:17:18

Reused existing feature / TFT dataset

STAGE_CYCLES_REUSED
2026-03-22 02:17:18

Reused existing cycle / baseline / FSI output

STAGE_LOAD_REUSED
2026-03-22 02:17:18

Reused existing load output

UI_TRAIN_START
2026-03-22 02:17:18

Train action started from UI.

Model runs

IDStatusCheckpoint
#17 success /app/models/run_17_modelrun_17/best.ckpt

Latest model details

Model run#17
Statussuccess
Checkpoint/app/models/run_17_modelrun_17/best.ckpt

Training params

dropout0.1
val_days90
batch_size64
max_epochs40
hidden_size64
feature_specv1_log1p_fsi
learning_rate0.001
attention_heads4
target_transformlog1p
max_encoder_length60
max_prediction_length30

Training metrics

rows2662
backtest{'rows': 2662, 'assets': 3, 'h1_mae': 0.09150402341526905, 'h7_mae': 0.03168751260114098, 'h14_mae': 0.034220284396271755, 'h1_rmse': 0.1650415408253389, 'h30_mae': 0.32511792757666597, 'h7_rmse': 0.04881575514333678, 'h14_rmse': 0.049535170933054956, 'h1_smape': 194.8174410178323, 'h30_rmse': 0.37289611647379955, 'h7_smape': 191.525406798097, 'h14_smape': 188.7888050177969, 'h30_smape': 157.97227682359417, 'used_cutoffs': [1027, 1034, 1041, 1048], 'skipped_cutoffs': [{'cutoff': 1020, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}], 'num_used_cutoffs': 4, 'requested_cutoffs': [1020, 1027, 1034, 1041, 1048], 'negative_pred_rate': 0.0625, 'num_prediction_rows': 16, 'num_skipped_cutoffs': 1, 'num_requested_cutoffs': 5}
qa_summary{'rows': 2662, 'assets': 3, 'by_asset': [{'rows': 831, 'fsi_max': 3.690835432401493, 'fsi_min': 0.0, 'time_max': 904, 'time_min': 0, 'asset_code': 'E102A', 'neg_raw_fsi': 0}, {'rows': 1000, 'fsi_max': 29.31267248493691, 'fsi_min': 0.0, 'time_max': 1080, 'time_min': 0, 'asset_code': 'E102B', 'neg_raw_fsi': 0}, {'rows': 831, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 904, 'time_min': 0, 'asset_code': 'E102C', 'neg_raw_fsi': 0}], 'time_idx_max': 1080, 'time_idx_min': 0, 'top_null_counts': {'day': 0, 'fsi': 0, 'flow': 0, 'day_cos': 0, 'day_sin': 0, 'delta_p': 0, 'baseline': 0, 'cycle_id': 0, 'time_idx': 0, 'is_weekend': 0, 'cleaning_event': 0, 'pdi_normalized': 0, 'steam_feed_ratio': 0, 'system_t_skin_avg': 0, 'system_t_skin_max': 0, 'total_vc5_rate_m3h': 0, 'days_since_cleaning': 0, 'furnaces_in_service': 0, 'total_ngl_rate_km3h': 0, 'pws_feedwater_rate_m3h': 0}, 'insufficient_assets': [], 'negative_target_count': 0, 'negative_raw_fsi_count': 0}
max_time_idx1080
cutoff_time_idx990
best_model_score0.09676852822303772
target_transformlog1p
zero_target_count1703
negative_raw_fsi_count0

Validation metrics

HorizonMetricValue
1mae0.09150402341526905
1rmse0.1650415408253389
1smape194.8174410178323
7mae0.03168751260114098
7rmse0.04881575514333678
7smape191.525406798097
14mae0.034220284396271755
14rmse0.049535170933054956
14smape188.7888050177969
30mae0.32511792757666597
30rmse0.37289611647379955
30smape157.97227682359417

Physics metrics

MetricValue
actual_fsi_max3.690835432401493
actual_fsi_max5.04221501862956
actual_fsi_max29.31267248493691
actual_fsi_mean0.12270322168700852
actual_fsi_mean0.09984170613976978
actual_fsi_mean0.46839388210382393
negative_pred_rate0.0625
rows_in_eval_df1000.0
rows_in_eval_df831.0
rows_in_eval_df831.0