MLOps Financial Sentinel

Abstract

MLOps Financial Sentinel is an institutional-grade, event-driven Machine Learning platform built from bare compute on Oracle Cloud Infrastructure (OCI) and lightweight Kubernetes (k3s). It orchestrates point-in-time market data lineage, distributed deep learning time-series forecasting (AutoGluon TimeSeries with Chronos, PatchTST, and DeepAR), Katib Bayesian AutoML gates, and scale-to-zero model serving for the Indonesia Stock Exchange (XIDX).

🏗️ End-to-End System Architecture

┌────────────────────────────────────────────────────────────────────────┐
│ 1. Point-in-Time Data Lineage & Replay Layer                          │
│    • Ingestion: Daily XIDX EOD trades & market screen replays          │
│    • Deterministic Processing: Python uv + pyarrow + Calendar alignment│
│    • S3 Object Storage: MinIO with RFC 8785 canonical SHA-256 digests  │
│      (Buckets: xidx-market-data, feature-store, model-registry)        │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ 2. Declarative Orchestration & Tracking (Kubeflow Pipelines v2 SDK)    │
│    ├── Step 1: Snapshot Ingestion & Point-in-Time Cutoff Validation   │
│    ├── Step 2: Baseline Feature Transforms (ATR(14), Log-Returns)      │
│    ├── Step 3: AutoGluon TimeSeries Fit (Ensemble Deep Models)         │
│    │           ├── Katib Controller (Bayesian Hyperparameter Search)   │
│    │           └── MLflow Server (Live WQL / CRPS Loss Curve Streams)  │
│    ├── Step 4: Held-Out Cold-Start Certification Check                 │
│    └── Step 5: Benchmark Evaluation Gate (Coverage >= 95%)             │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                         ┌──────────┴──────────┐
                         │ Gate PASS: Cov >=95%│
                         ▼                     ▼ (Gate FAIL -> Quarantine Alert)
┌────────────────────────────────────────────────────────────────────────┐
│ 3. Production Serving, Automation & Drift Sentries                     │
│    • Registry: Blessed model artifacts promoted to MLflow Registry     │
│    • Serving Engine: KServe InferenceService (v2 DataPlane protocol)   │
│    • Serverless Autoscaling: Knative Scale-to-Zero during idle hours   │
│    • Dispatch: Real-time candidate ranking via Discord Webhook alerts  │
│    • Drift Sentry: KS-test on volatility & Real-Time Quantile Breaches │
└────────────────────────────────────────────────────────────────────────┘

📑 Zero-to-One MLOps Article Series

This project is documented as an in-depth, multi-part engineering series detailing how each layer is built from first principles:

1. Zero-to-One MLOps (Part 1): Why Build by Hand and the 5-Phase Architecture Why toy tutorials and managed SaaS obscure fundamental systems physics, the 3 physical failure modes of distributed ML, and our 5-phase architectural blueprint.

2. Zero-to-One MLOps (Part 2): Hands-on Guide to k3s, MinIO Helm, and Traefik Ingress Step-by-step runbook for provisioning a production-grade MLOps base on bare Linux: k3s v1.31, MinIO S3 via Helm, dynamic Terraform manifest loaders, and Traefik HTTPS Ingress backed by Cloudflare.

3. Part 3: Point-in-Time Market Data Lineage & Cryptographic Manifests (Upcoming) Eliminating lookahead bias with point-in-time Parquet partitions, calendar-aware alignment, and RFC 8785 SHA-256 manifests on MinIO.

4. Part 4: Declarative Kubeflow Pipelines (KFP v2) & MLflow Experiment Tracking (Upcoming) Containerized multi-stage DAG orchestration, ephemeral artifact passing, and streaming live validation loss curves.

5. Part 5: Katib Bayesian AutoML & Automated Benchmark Certification Gates (Upcoming) Automated hyperparameter optimization, held-out cold-start qualification, and strict model promotion criteria.

6. Part 6: Production KServe Scale-to-Zero Serving & Statistical Drift Sentries (Upcoming) Serverless inference economics (saving 128 idle hours/week), Knative scale-to-zero, and Kolmogorov-Smirnov drift alerts.

🛠️ Technology Stack & Platform Components

🎯 Engineering Invariants

1. Storage Decoupled from Pod Lifecycles: Kubernetes pods are ephemeral worker units; state is permanent. All datasets, weights, and metrics flow directly to S3.
2. Zero Lookahead Tolerance: Data ingested for training must strictly reflect information available at prediction time $T$, enforced via cryptographic hash manifests.
3. Automate from First Principles: Avoid vendor lock-in by sticking to open cloud-native standards (Kubernetes CRDs, S3 API, OCI container images).