MLOps for OT: Versioning, Drift, and Model Monitoring on the Edge

By Articles for AutomationInside.com
Posted on Oct 28, 2025

MLOps for OT: Versioning, Drift, and Model Monitoring on the Edge

Industrial AI now sits inside control rooms, not just in data centers. When models influence maintenance decisions or trigger alarms, plants need a disciplined way to version, deploy, monitor, and roll back models across fleets of gateways and HMIs. This article explains a practical MLOps-for-OT blueprint that respects plant constraints—deterministic timing, safety, and segmented networks—while giving data teams the tools they expect from modern software delivery.

Why OT Needs Its Own Flavor of MLOps

Traditional MLOps assumes abundant cloud compute and uniform environments. In operations technology (OT), we face three hard realities:

Heterogeneous edge: From fanless IPCs to ARM gateways, often without GPUs and with limited storage.
Determinism & safety: Models run beside PLCs and robots; jitter and surprise updates are unacceptable.
Network segmentation: Many cells operate with intermittent or firewalled connectivity. Updates must survive outages and prove integrity.

As discussed in Edge vs Cloud Inference, inference typically stays on the edge for latency. That increases the importance of governed deployments and monitoring at the edge.

Core Principles

Artifact immutability: Freeze models as versioned, signed artifacts (e.g., ONNX/TensorRT package with a manifest and hash).
Reproducibility: Every prediction ties back to model version, feature pipeline version, and configuration snapshot.
Safety-first rollouts: Stage to canary cells, enforce break-glass rollback, and never hot-swap without validation.
Edge-first monitoring: Collect lightweight health, data drift, and performance KPIs locally; sync summaries upstream when bandwidth allows.

A Reference Architecture for MLOps on the Edge

Model Registry (central): Stores artifacts, schemas, and approval state. Think of it as the “single source of truth”.
Deployment Orchestrator: Pushes signed bundles over a pull-based agent to edge nodes; supports staged rings (dev → pilot → prod).
Edge Runtime: Containerized inference plus a feature service that guarantees consistent preprocessing.
Telemetry Pipeline: Local store (ring buffer) for predictions and sample payloads; periodic backhaul to a data lake or historian.
Monitoring & Alerting: On-edge heartbeats, latency histograms, and drift metrics; central dashboards aggregate fleets.

Version Everything That Affects a Prediction

In maintenance and quality, the model is only part of the story. To make decisions auditable:

Model version: Semantic tag (e.g., bearing-detector:2.3.1), hash, training data snapshot ID.
Feature pipeline version: Signal conditioning, filtering, window sizes, and normalization constants.
Threshold pack: Alarm tiers and hysteresis by asset class or SKU.
Environment: Runtime (TensorRT/OpenVINO), hardware target, and driver versions.

Embed these in the inference header that accompanies every decision. When a planner queries an alert in the CMMS, they should see the exact versions that produced it.

Data & Concept Drift: What to Measure at the Edge

Drift catches the slow mismatch between training assumptions and reality. On constrained gateways, start with metrics that are cheap and effective:

Input drift: Monitor distribution shifts in key features (e.g., RMS, kurtosis, temperature) using running means/variances or lightweight histograms.
Prediction drift: Track class balance and confidence. A steady rise in “uncertain” predictions is a leading indicator.
Performance drift proxy: For PdM, correlate alerts with operator dispositions and work orders to compute precision/recall over rolling windows.
Timing drift: Watch capture→inference latency and its P95/P99; rising jitter often precedes missed actuations.

When drift exceeds guardrails, the edge agent raises a retrain request or auto-queues samples for labeling. See Predictive Maintenance 2025 for feature choices that remain stable across lines.

Deployment Rings and Safe Rollback

Avoid big-bang deployments. Use rings to stage risk:

Dev ring: Test with recorded data and simulated I/O.
Pilot ring: 1–3 cells in production with shadow mode or advisory-only.
Prod ring: Gradual ramp with automatic rollback if KPIs regress.

Define go/no-go gates between rings: minimum precision, latency ceiling, operator intervention rate, and absence of new nuisance alarms. Rollback should be a button, not a project.

Monitoring Pack: The Minimum You Need on Day 1

Health: CPU/GPU utilization, temps, disk pressure, inference throughput, queue sizes.
Model KPIs: Alert counts by tier, precision/recall from operator feedback, confidence histograms.
Data Quality: Feature completeness, sensor dropout rate, time sync checks.
Change Log: Last model/threshold update, approver, and checksum.

Expose these on an HMI page for operators and forward summaries to a central dashboard when connectivity allows. For deterministic transport and consistent timestamps, OPC UA over TSN can coexist with standard IP telemetry.

Labeling and Feedback in the Plant

Most failures are rare. The best labels come from technicians who inspect assets or parts after an alert:

Operator prompts: Simple “Confirm/Reject” with reason codes at disposition time.
Work-order linkage: Each alert attaches to the CMMS ticket; outcomes feed back to training sets.
Sampling policy: Periodically store raw windows or image crops around alerts to enrich datasets without excessive storage cost.

Security & Compliance

Models are intellectual property and may imply safety functions. Treat them accordingly:

Code signing: Only run signed artifacts from the registry. Verify signatures at boot and before activation.
Least privilege: Edge agents cannot write PLC logic; they publish results via well-defined interfaces.
Air-gapped updates: Support USB or offline bundles with the same verification path for sites without WAN access.
Audit trails: Immutable logs of who approved and when the change took effect.

KPIs That Matter for MLOps in OT

MTTR (Model Time To Restore): Time from regression detection to safe rollback.
Drift Mean Time to Detection: Average time between actual change and alert, by asset class.
Deployment Lead Time: Commit → artifact → edge activation under change-control.
On-edge latency P95/P99: Budget adherence across the fleet.
Alert Precision/Recall: Verified by operator dispositions and work orders, not just offline tests.

Example: Bearing Fault Detector Rollout

A plant deploys a lightweight autoencoder to 120 motors. The registry tags bearing-ae:1.4.0 with feature pack feat-v3 and thresholds thres-2025-02. The pilot ring shows a small precision drop on high-speed assets; drift monitors reveal elevated kurtosis at high ambient temperatures. Engineers add temperature normalization and release 1.4.1. Rollout proceeds, with canaries on each line. A week later, a storage outage causes intermittent backhaul—but on-edge monitoring keeps alerts flowing and backfills once connectivity returns. All decisions remain auditable to versioned artifacts.

Lightweight Q&A

Do I need a full Kubernetes stack on the edge?

No. Many plants run a small agent that pulls signed bundles and supervises a single container. Keep it simple and deterministic.

How do I handle multi-vendor sensors and historians?

Standardize at the feature layer. Map vendor differences to a common feature registry so models remain portable across sites.

What about sites with no internet?

Use offline bundles and a local registry mirror. The same signatures and approval workflows apply; synchronization happens via removable media during maintenance windows.

Conclusion

MLOps for OT is about repeatability and safety at the edge. Version every artifact that shapes a prediction, monitor for drift with lightweight metrics, and roll out through controlled rings with instant rollback. Do this, and your models will earn trust with technicians—converting analytics into reliable maintenance actions across your fleet.