Fouling AI
Run #10 success proof_build_manual

Run detail

Monitor the academic pipeline and ML actions from one page.

Open Grafana Back to batch All runs
Started
2026-03-08 18:17
Finished
2026-03-08 18:17
Assets processed
3
Days processed
1742
Operator guide
Model run #8 is available. Next actions: Validate, Backtest, then Forecast.

Stage flow

Each stage shows its summary, errors, and payload details.

Load
pending

Waiting

Cycles / Baseline / FSI
pending

Waiting

Features / TFT Dataset
pending

Waiting

ML Train
success

TFT training completed

id8
params{'dropout': 0.1, 'val_days': 90, 'batch_size': 64, 'max_epochs': 40, 'hidden_size': 64, 'feature_spec': 'v1_log1p_fsi', 'learning_rate': 0.001, 'attention_heads': 4, 'target_transform': 'log1p', 'max_encoder_length': 60, 'max_prediction_length': 30}
metrics{'rows': 1478, 'qa_summary': {'rows': 1478, 'assets': 3, 'by_asset': [{'rows': 358, 'fsi_max': 1.4368447795979469, 'fsi_min': 0.0, 'time_max': 524, 'time_min': 0, 'asset_code': 'E102A', 'neg_raw_fsi': 0}, {'rows': 740, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 1047, 'time_min': 0, 'asset_code': 'E102B', 'neg_raw_fsi': 0}, {'rows': 380, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 675, 'time_min': 0, 'asset_code': 'E102C', 'neg_raw_fsi': 0}], 'time_idx_max': 1047, 'time_idx_min': 0, 'top_null_counts': {'day': 0, 'fsi': 0, 'flow': 0, 'day_cos': 0, 'day_sin': 0, 'delta_p': 0, 'baseline': 0, 'cycle_id': 0, 'time_idx': 0, 'is_weekend': 0, 'cleaning_event': 0, 'pdi_normalized': 0, 'steam_feed_ratio': 0, 'system_t_skin_avg': 0, 'system_t_skin_max': 0, 'total_vc5_rate_m3h': 0, 'days_since_cleaning': 0, 'furnaces_in_service': 0, 'total_ngl_rate_km3h': 0, 'pws_feedwater_rate_m3h': 0}, 'insufficient_assets': [], 'negative_target_count': 0, 'negative_raw_fsi_count': 0}, 'max_time_idx': 1047, 'cutoff_time_idx': 957, 'best_model_score': 0.05225546285510063, 'target_transform': 'log1p', 'zero_target_count': 741, 'negative_raw_fsi_count': 0}
checkpoint_path/app/models/run_10_modelrun_8/best.ckpt

2026-03-08 18:12:21 → 2026-03-08 18:14:45

Validate
success

TFT validation completed

errorno_forecasts

2026-03-08 18:12:39 → 2026-03-08 18:12:39

Backtest
success

TFT backtest completed

rows1478
assets3
h1_mae0.029753006617158555
h7_mae0.037790607677026106
h14_mae0.014034801302518718
h1_rmse0.03945307024123112
h30_mae0.00825027145886525
h7_rmse0.0508465555576532
h14_rmse0.014113207662959724
h1_smape200.0
h30_rmse0.014289889342990017
h7_smape200.0
h14_smape143.99320702339304
h30_smape200.0
used_cutoffs[1001, 1008, 1015]
skipped_cutoffs[{'cutoff': 987, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}, {'cutoff': 994, 'reason': 'assertion_error: filters should not remove entries all entries - check encoder/decoder lengths and lags'}]
num_used_cutoffs3
requested_cutoffs[987, 994, 1001, 1008, 1015]
negative_pred_rate0.16666666666666666
num_prediction_rows12
num_skipped_cutoffs2
num_requested_cutoffs5

2026-03-08 18:13:00 → 2026-03-08 18:13:03

Forecast
success

TFT forecast generation completed

rows_written12
assets_predicted3
forecast_end_day2024-06-26
forecast_start_day2024-05-08

2026-03-08 18:17:18 → 2026-03-08 18:17:21

Run parameters

modeproof_build_manual
sourceui
batch_id6
latest_model_params{'dropout': 0.1, 'val_days': 90, 'batch_size': 64, 'max_epochs': 40, 'hidden_size': 64, 'feature_spec': 'v1_log1p_fsi', 'learning_rate': 0.001, 'attention_heads': 4, 'target_transform': 'log1p', 'max_encoder_length': 60, 'max_prediction_length': 30}
latest_model_run_id8
latest_forecast_rows_written12
latest_forecast_assets_predicted3

Recent events

STAGE_FORECAST_SUCCESS
2026-03-08 18:17:21

TFT forecast generation completed

STAGE_FORECAST_START
2026-03-08 18:17:18

TFT forecast generation started

UI_FORECAST_START
2026-03-08 18:17:18

Forecast action started from UI.

STAGE_TRAIN_SUCCESS
2026-03-08 18:14:45

TFT training completed

STAGE_FORECAST_SUCCESS
2026-03-08 18:13:19

TFT forecast generation completed

STAGE_FORECAST_START
2026-03-08 18:13:16

TFT forecast generation started

UI_FORECAST_START
2026-03-08 18:13:16

Forecast action started from UI.

STAGE_BACKTEST_SUCCESS
2026-03-08 18:13:03

TFT backtest completed

STAGE_BACKTEST_START
2026-03-08 18:13:00

TFT backtest started

UI_BACKTEST_START
2026-03-08 18:13:00

Backtest action started from UI.

STAGE_VALIDATE_SUCCESS
2026-03-08 18:12:39

TFT validation completed

STAGE_VALIDATE_START
2026-03-08 18:12:39

TFT validation started

UI_VALIDATE_START
2026-03-08 18:12:39

Validate action started from UI.

STAGE_TRAIN_START
2026-03-08 18:12:21

TFT training started

UI_TRAIN_START
2026-03-08 18:12:21

Train action started from UI.

STAGE_TRAIN_SUCCESS
2026-03-08 18:11:53

TFT training completed

STAGE_TRAIN_START
2026-03-08 18:09:42

TFT training started

UI_TRAIN_START
2026-03-08 18:09:41

Train action started from UI.

Model runs

IDStatusCheckpoint
#8 success /app/models/run_10_modelrun_8/best.ckpt
#7 success /app/models/run_10_modelrun_7/best.ckpt

Latest model details

Model run#8
Statussuccess
Checkpoint/app/models/run_10_modelrun_8/best.ckpt

Training params

dropout0.1
val_days90
batch_size64
max_epochs40
hidden_size64
feature_specv1_log1p_fsi
learning_rate0.001
attention_heads4
target_transformlog1p
max_encoder_length60
max_prediction_length30

Training metrics

rows1478
qa_summary{'rows': 1478, 'assets': 3, 'by_asset': [{'rows': 358, 'fsi_max': 1.4368447795979469, 'fsi_min': 0.0, 'time_max': 524, 'time_min': 0, 'asset_code': 'E102A', 'neg_raw_fsi': 0}, {'rows': 740, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 1047, 'time_min': 0, 'asset_code': 'E102B', 'neg_raw_fsi': 0}, {'rows': 380, 'fsi_max': 5.04221501862956, 'fsi_min': 0.0, 'time_max': 675, 'time_min': 0, 'asset_code': 'E102C', 'neg_raw_fsi': 0}], 'time_idx_max': 1047, 'time_idx_min': 0, 'top_null_counts': {'day': 0, 'fsi': 0, 'flow': 0, 'day_cos': 0, 'day_sin': 0, 'delta_p': 0, 'baseline': 0, 'cycle_id': 0, 'time_idx': 0, 'is_weekend': 0, 'cleaning_event': 0, 'pdi_normalized': 0, 'steam_feed_ratio': 0, 'system_t_skin_avg': 0, 'system_t_skin_max': 0, 'total_vc5_rate_m3h': 0, 'days_since_cleaning': 0, 'furnaces_in_service': 0, 'total_ngl_rate_km3h': 0, 'pws_feedwater_rate_m3h': 0}, 'insufficient_assets': [], 'negative_target_count': 0, 'negative_raw_fsi_count': 0}
max_time_idx1047
cutoff_time_idx957
best_model_score0.05225546285510063
target_transformlog1p
zero_target_count741
negative_raw_fsi_count0