Fouling AI
Run #9 success proof_build_manual

Run detail

Monitor the academic pipeline and ML actions from one page.

Open Grafana Back to batch All runs
Started
2026-03-08 20:40
Finished
2026-03-08 20:40
Assets processed
3
Days processed
1742
Operator guide
Model run #9 is available. Next actions: Validate, Backtest, then Forecast.

Stage flow

Each stage shows its summary, errors, and payload details.

Load
success

Reused existing load output

sourcereused_existing_artifacts
source_batch_id6

→ 2026-03-08 20:36:56

Cycles / Baseline / FSI
success

Reused existing cycle / baseline / FSI output

sourcereused_existing_artifacts
source_batch_id6

→ 2026-03-08 20:36:56

Features / TFT Dataset
success

Reused existing feature / TFT dataset

sourcereused_existing_artifacts
source_batch_id6

→ 2026-03-08 20:36:56

ML Train
success

TFT training completed

id9
params{'dropout': 0.1, 'val_days': 90, 'batch_size': 64, 'max_epochs': 40, 'hidden_size': 64, 'feature_spec': 'v1_log1p_fsi', 'learning_rate': 0.001, 'attention_heads': 4, 'target_transform': 'log1p', 'max_encoder_length': 60, 'max_prediction_length': 30}
metrics{'rows': 1478, 'qa_summary': {'rows': 1478, 'assets': 3, 'by_asset': [{'rows': 358, 'fsi_max': 1.4368447795979469, 'fsi_min': 0.0, 'time_max': 524, 'time_min': 0, 'asset_code': 'E102A', 'neg_raw_fsi': 0}, {'rows': 740, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 1047, 'time_min': 0, 'asset_code': 'E102B', 'neg_raw_fsi': 0}, {'rows': 380, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 675, 'time_min': 0, 'asset_code': 'E102C', 'neg_raw_fsi': 0}], 'time_idx_max': 1047, 'time_idx_min': 0, 'top_null_counts': {'day': 0, 'fsi': 0, 'flow': 0, 'day_cos': 0, 'day_sin': 0, 'delta_p': 0, 'baseline': 0, 'cycle_id': 0, 'time_idx': 0, 'is_weekend': 0, 'cleaning_event': 0, 'pdi_normalized': 0, 'steam_feed_ratio': 0, 'system_t_skin_avg': 0, 'system_t_skin_max': 0, 'total_vc5_rate_m3h': 0, 'days_since_cleaning': 0, 'furnaces_in_service': 0, 'total_ngl_rate_km3h': 0, 'pws_feedwater_rate_m3h': 0}, 'insufficient_assets': [], 'negative_target_count': 0, 'negative_raw_fsi_count': 0}, 'max_time_idx': 1047, 'cutoff_time_idx': 957, 'best_model_score': 0.05200118198990822, 'target_transform': 'log1p', 'zero_target_count': 741, 'negative_raw_fsi_count': 0}
checkpoint_path/app/models/run_9_modelrun_9/best.ckpt

2026-03-08 20:36:56 → 2026-03-08 20:39:01

Validate
success

TFT validation completed

errorno_forecasts

2026-03-08 20:39:40 → 2026-03-08 20:39:40

Backtest
success

TFT backtest completed

rows1478
assets3
h1_mae0.029753006617158555
h7_mae0.037790607677026106
h14_mae0.014034801302518718
h1_rmse0.03945307024123112
h30_mae0.00825027145886525
h7_rmse0.0508465555576532
h14_rmse0.014113207662959724
h1_smape200.0
h30_rmse0.014289889342990017
h7_smape200.0
h14_smape143.99320702339304
h30_smape200.0
used_cutoffs[1001, 1008, 1015]
skipped_cutoffs[{'cutoff': 987, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}, {'cutoff': 994, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}]
num_used_cutoffs3
requested_cutoffs[987, 994, 1001, 1008, 1015]
negative_pred_rate0.16666666666666666
num_prediction_rows12
num_skipped_cutoffs2
num_requested_cutoffs5

2026-03-08 20:40:13 → 2026-03-08 20:40:16

Forecast
success

TFT forecast generation completed

rows_written12
assets_predicted3
forecast_end_day2024-06-26
forecast_start_day2024-05-08

2026-03-08 20:40:53 → 2026-03-08 20:40:56

Run parameters

modeproof_build_manual
sourceui
batch_id6
latest_model_params{'dropout': 0.1, 'val_days': 90, 'batch_size': 64, 'max_epochs': 40, 'hidden_size': 64, 'feature_spec': 'v1_log1p_fsi', 'learning_rate': 0.001, 'attention_heads': 4, 'target_transform': 'log1p', 'max_encoder_length': 60, 'max_prediction_length': 30}
latest_model_run_id9
latest_backtest_summary{'rows': 1478, 'assets': 3, 'h1_mae': 0.029753006617158555, 'h7_mae': 0.037790607677026106, 'h14_mae': 0.014034801302518718, 'h1_rmse': 0.03945307024123112, 'h30_mae': 0.00825027145886525, 'h7_rmse': 0.0508465555576532, 'h14_rmse': 0.014113207662959724, 'h1_smape': 200.0, 'h30_rmse': 0.014289889342990017, 'h7_smape': 200.0, 'h14_smape': 143.99320702339304, 'h30_smape': 200.0, 'used_cutoffs': [1001, 1008, 1015], 'skipped_cutoffs': [{'cutoff': 987, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}, {'cutoff': 994, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}], 'num_used_cutoffs': 3, 'requested_cutoffs': [987, 994, 1001, 1008, 1015], 'negative_pred_rate': 0.16666666666666666, 'num_prediction_rows': 12, 'num_skipped_cutoffs': 2, 'num_requested_cutoffs': 5}
latest_validation_summary{'error': 'no_forecasts'}
latest_forecast_rows_written12
latest_forecast_assets_predicted3

Recent events

STAGE_FORECAST_SUCCESS
2026-03-08 20:40:56

TFT forecast generation completed

STAGE_FORECAST_START
2026-03-08 20:40:53

TFT forecast generation started

UI_FORECAST_START
2026-03-08 20:40:53

Forecast action started from UI.

STAGE_BACKTEST_SUCCESS
2026-03-08 20:40:16

TFT backtest completed

STAGE_BACKTEST_START
2026-03-08 20:40:13

TFT backtest started

UI_BACKTEST_START
2026-03-08 20:40:13

Backtest action started from UI.

STAGE_VALIDATE_SUCCESS
2026-03-08 20:39:40

TFT validation completed

STAGE_VALIDATE_START
2026-03-08 20:39:40

TFT validation started

UI_VALIDATE_START
2026-03-08 20:39:40

Validate action started from UI.

STAGE_TRAIN_SUCCESS
2026-03-08 20:39:01

TFT training completed

STAGE_TRAIN_START
2026-03-08 20:36:56

TFT training started

STAGE_FEATURES_REUSED
2026-03-08 20:36:56

Reused existing feature / TFT dataset

STAGE_CYCLES_REUSED
2026-03-08 20:36:56

Reused existing cycle / baseline / FSI output

STAGE_LOAD_REUSED
2026-03-08 20:36:56

Reused existing load output

UI_TRAIN_START
2026-03-08 20:36:56

Train action started from UI.

Model runs

IDStatusCheckpoint
#9 success /app/models/run_9_modelrun_9/best.ckpt

Latest model details

Model run#9
Statussuccess
Checkpoint/app/models/run_9_modelrun_9/best.ckpt

Training params

dropout0.1
val_days90
batch_size64
max_epochs40
hidden_size64
feature_specv1_log1p_fsi
learning_rate0.001
attention_heads4
target_transformlog1p
max_encoder_length60
max_prediction_length30

Training metrics

rows1478
backtest{'rows': 1478, 'assets': 3, 'h1_mae': 0.029753006617158555, 'h7_mae': 0.037790607677026106, 'h14_mae': 0.014034801302518718, 'h1_rmse': 0.03945307024123112, 'h30_mae': 0.00825027145886525, 'h7_rmse': 0.0508465555576532, 'h14_rmse': 0.014113207662959724, 'h1_smape': 200.0, 'h30_rmse': 0.014289889342990017, 'h7_smape': 200.0, 'h14_smape': 143.99320702339304, 'h30_smape': 200.0, 'used_cutoffs': [1001, 1008, 1015], 'skipped_cutoffs': [{'cutoff': 987, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}, {'cutoff': 994, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}], 'num_used_cutoffs': 3, 'requested_cutoffs': [987, 994, 1001, 1008, 1015], 'negative_pred_rate': 0.16666666666666666, 'num_prediction_rows': 12, 'num_skipped_cutoffs': 2, 'num_requested_cutoffs': 5}
qa_summary{'rows': 1478, 'assets': 3, 'by_asset': [{'rows': 358, 'fsi_max': 1.4368447795979469, 'fsi_min': 0.0, 'time_max': 524, 'time_min': 0, 'asset_code': 'E102A', 'neg_raw_fsi': 0}, {'rows': 740, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 1047, 'time_min': 0, 'asset_code': 'E102B', 'neg_raw_fsi': 0}, {'rows': 380, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 675, 'time_min': 0, 'asset_code': 'E102C', 'neg_raw_fsi': 0}], 'time_idx_max': 1047, 'time_idx_min': 0, 'top_null_counts': {'day': 0, 'fsi': 0, 'flow': 0, 'day_cos': 0, 'day_sin': 0, 'delta_p': 0, 'baseline': 0, 'cycle_id': 0, 'time_idx': 0, 'is_weekend': 0, 'cleaning_event': 0, 'pdi_normalized': 0, 'steam_feed_ratio': 0, 'system_t_skin_avg': 0, 'system_t_skin_max': 0, 'total_vc5_rate_m3h': 0, 'days_since_cleaning': 0, 'furnaces_in_service': 0, 'total_ngl_rate_km3h': 0, 'pws_feedwater_rate_m3h': 0}, 'insufficient_assets': [], 'negative_target_count': 0, 'negative_raw_fsi_count': 0}
max_time_idx1047
cutoff_time_idx957
best_model_score0.05200118198990822
target_transformlog1p
zero_target_count741
negative_raw_fsi_count0

Validation metrics

HorizonMetricValue
1mae0.029753006617158555
1rmse0.03945307024123112
1smape200.0
7mae0.037790607677026106
7rmse0.0508465555576532
7smape200.0
14mae0.014034801302518718
14rmse0.014113207662959724
14smape143.99320702339304
30mae0.00825027145886525
30rmse0.014289889342990017
30smape200.0

Physics metrics

MetricValue
actual_fsi_max1.4368447795979469
actual_fsi_max5.04221501862956
actual_fsi_max5.04221501862956
actual_fsi_mean0.21455832557829554
actual_fsi_mean0.13704504075908946
actual_fsi_mean0.1616484376933917
negative_pred_rate0.16666666666666666
rows_in_eval_df740.0
rows_in_eval_df358.0
rows_in_eval_df380.0