Part 3: Hard Real-Time Edge AI for Automotive Inspection: Designing the Inference and Control-Plane Split

With the architectural pattern chosen, the next engineering challenge is building the edge inference stack itself  a deterministic pipeline that must complete reliably within a physical actuation window on every single production cycle, including the rare ones.

‍

This part walks through all five layers of the stack in sequence: frame acquisition and timestamping, preprocessing, inference runtime selection, decision logic and failure semantics, and PLC integration. Each section goes beyond component selection to cover the design decisions that determine whether your system meets its p99 latency budget in production  not just in a benchmark. If you are implementing or reviewing an edge inspection stack, this is the engineering reference for each layer.

Edge inference stack: from camera trigger to PLC bit

Design the edge stack as a pipeline with strict timing guarantees and well-defined failure modes.

1. Frame acquisition and triggering

Use industrial cameras (GigE Vision, CoaXPress, USB3) with hardware triggers tied to encoders, photo-eyes, or PLC outputs so that frames align with part position. Fix optics (aperture, focal length) and lighting conditions to minimize variation in image intensity and glare, which directly affects model stability.

‍

Timestamping and correlation treat these as first-class requirements at acquisition, not as an afterthought at logging.

‍

Every frame must be assigned a unique, immutable frame ID at the moment of hardware trigger not at preprocessing, not at inference, and not at the PLC write. This frame ID must propagate unchanged through every subsequent stage of the pipeline: preprocessing, inference, decision logic, PLC handshake, and local log entry.  

‍

Without this, post-incident debugging becomes a manual correlation exercise across disconnected log files with no reliable join key.

• Frame ID generation: Assign a frame ID at trigger time, combining a monotonic sequence number with a high-resolution timestamp (e.g., line-A_cam-01_20260326T141523.004_seq00412). The sequence number catches dropped frames; the timestamp enables absolute latency attribution per stage.

• Hardware timestamp at acquisition: Where the camera or frame grabber supports it (GigE Vision chunk data, CoaXPress auxiliary channel), embed a hardware-generated timestamp directly in the frame metadata. Hardware timestamps are far more reliable than software timestamps for jitter measurement because they are not subject to OS scheduling delays or thread preemption.

• Correlation ID propagation: The frame ID must be carried as a first-class field through every pipeline stage and appear in every log entry related to that frame: preprocessing duration, inference score, decision outcome, PLC write timestamp, and actuator confirmation. This creates a complete, queryable audit trail per part, per frame, per cycle.

• End-to-end latency attribution: By recording entry and exit timestamps at each stage boundary (acquisition → preprocessing → inference → decision → PLC write), you can reconstruct the exact latency breakdown for any frame after the fact. This is essential for: (a) identifying which stage caused a p99 latency breach, (b) correlating missed rejections with specific pipeline delays during incident reviews, and (c) validating that your latency budget is being met in production, not just in benchmarks.

• Correlation with downstream quality events:  If your line has a re-inspection station or end-of-line measurement system, the frame ID enables you to join edge inference decisions back to ground-truth quality outcomes for the same physical part. Without a shared correlation key, this join is impossible and in-field precision/recall cannot be computed.

A minimal frame context object passed through the pipeline might look like:

‍

@dataclass 

class FrameContext: 

    frame_id: str          # e.g. "lineA_cam01_20260326T141523004_seq00412" 

    hw_timestamp_ns: int   # hardware trigger timestamp in nanoseconds 

    sw_acquired_ns: int    # software receive timestamp for jitter measurement 

    camera_id: str 

    line_id: str 

    part_carrier_id: str   # from encoder or RFID if available 

  

    # Populated as frame progresses through pipeline 

    preprocess_start_ns: int = 0 

    preprocess_end_ns: int = 0 

    inference_start_ns: int = 0 

    inference_end_ns: int = 0 

    decision_ts_ns: int = 0 

    plc_write_ts_ns: int = 0 

 

‍

This context object travels with the frame from acquisition to PLC write, is serialized to the local log on completion, and can be forwarded to the cloud as part of telemetry batches for fleet-wide latency analysis

2. Preprocessing service

Implement deterministic transforms (resize, crop ROIs, color normalize, and, when needed, de-warp or de-skew images) to standardize geometry. How you deploy this preprocessing stage as a separate process, a separate container, or an in-process pipeline step is a design trade-off with real performance implications, not a blanket recommendation.

Process separation: benefits and costs

‍

Running preprocessing in a separate process or container offers genuine advantages:

• Fault isolation: A crash or memory leak in the preprocessing stage does not take down the inference runtime. The inference process remains alive and can respond with a safe failure mode (e.g., forced reject or line stop) rather than going dark entirely.

• Independent scaling and profiling : CPU-bound preprocessing and GPU-bound inference can be profiled, tuned, and resource-constrained independently (e.g., via cgroup CPU quotas), making it easier to identify which stage is the bottleneck.

• Restart and recovery: The preprocessing process can be restarted independently without interrupting the inference runtime, reducing mean time to recovery for preprocessing-specific faults.

However, process separation introduces a non-trivial cost that must be explicitly accounted for in your latency budget:

• Memory copy overhead: When preprocessing and inference run in separate processes, the preprocessed tensor (e.g., a 3 × 512 × 512 float32 array = 3 MB) must be transferred across the process boundary via shared memory, a Unix socket, or a message queue. Even with shared memory (the fastest option), this involves at least one memory copy and synchronization overhead. On a constrained edge device, repeated large tensor copies can consume measurable CPU cycles and add 1–5 ms of inter-process communication latency per frame.

• GPU memory staging: If preprocessing runs on CPU in one process and inference runs on GPU in another, the tensor must be copied from CPU RAM to GPU VRAM as a separate step, rather than being placed directly into a pinned GPU memory buffer in a single in-process pipeline. This can add GPU transfer overhead that an in-process design avoids entirely using CUDA pinned memory or zero-copy buffers.

• Pipeline stall risk: In a separated design, the preprocessing process must signal the inference process that a new tensor is ready. Under high frame rates or when OS scheduling causes preemption, this handoff can introduce jitter that compounds with other pipeline stages.

How to decide:

The right answer depends on your hardware tier, frame rate, tensor size, and tolerance for operational complexity. Measure the inter-process transfer overhead on your target hardware before committing to a separated architecture on a Jetson Orin NX at 30 fps with 1080p inputs; the copy cost may be acceptable; on a lower-tier device at 60 fps, it may not be.

3. Inference Runtime

Use a runtime optimized for your hardware footprint. The three primary options for automotive edge deployments are distinct in capability and trade-offs choose based on your hardware target, how much optimization control you need, and your model export format:

• ONNX Runtime (ORT) with execution providers — The recommended starting point for most deployments. ORT supports multiple hardware backends via execution providers: the CUDA Execution Provider (CUDA EP) runs inference on NVIDIA GPUs using standard CUDA, while the TensorRT Execution Provider (TensorRT EP) transparently applies TensorRT engine optimization under the ONNX Runtime API surface. Using TensorRT EP gives you TensorRT-level performance with less manual engine management, at the cost of some flexibility in quantization and layer fusion control.

• TensorRT directly — When you need the maximum degree of optimization control — custom quantization calibration, explicit layer fusion, fine-grained memory pool management, or engine serialization for a specific GPU SKU — use TensorRT directly rather than through the ONNX Runtime abstraction layer. This is the right choice for high-throughput, latency-critical stations where squeezing the last few milliseconds out of the inference budget is justified, and where the team has the expertise to manage engine build pipelines, calibration datasets, and version-locked engine artifacts per device class.

• OpenVINO — Intel's dedicated inference runtime for deploying models across supported Intel hardware targets: CPU (with AVX-512 optimizations), Intel integrated GPU, Intel Arc discrete GPU, Intel Movidius VPU, and Intel FPGA. OpenVINO is not a generic "low-power" option — it is purpose-built for Intel silicon and achieves best performance through IR (Intermediate Representation) conversion: models are converted from their source format (ONNX, PyTorch, TensorFlow) to OpenVINO IR using the Model Optimizer, which applies Intel-specific graph optimizations, layer fusion, and quantization (INT8 via Post-Training Optimization Toolkit). Using ONNX models directly in OpenVINO without IR conversion leaves significant performance on the table. This is the right choice for Intel-based industrial PCs where NVIDIA GPU hardware is not present or not desired.

Model patterns for automotive quality control

‍

The right model architecture depends on the inspection task. Automotive quality-control use cases span a wider range than detection and classification alone:

• Detection models (e.g., YOLO, EfficientDet, RT-DETR) Best for localizing multiple discrete defects per frame (scratches, dents, missing fasteners) when part position varies or multiple defect classes must be localized simultaneously. Outputs bounding boxes with class labels and confidence scores.

• Classification models: Best for fixed, fixtured parts where the question is binary or multi-class at the frame level: presence/absence, correct/incorrect orientation, marking vs. no-marking. More computationally efficient than detection on constrained hardware when localization is not required.

• Segmentation models (e.g., Mask R-CNN, YOLOv8-seg, DeepLabV3+): Best when defect area, shape, or boundary matters for the pass/fail decision for example, weld bead geometry, surface crack propagation extent, or coating coverage percentage. Segmentation outputs pixel-level masks that enable area measurement and morphology analysis beyond what bounding boxes allow. Computationally heavier than detection; requires careful quantization and input resolution trade-offs on edge hardware.

• Anomaly detection models (e.g., PatchCore, FastFlow, EfficientAD): Best for unsupervised or semi-supervised inspection where labelled defect examples are scarce or the defect distribution is open-ended. These models learn a normality distribution from defect-free samples and flag frames that deviate from it, without requiring explicit defect class labels. This pattern is particularly valuable in automotive for: (a) new part variants where defect data has not yet accumulated, (b) surface finish inspection where defect morphology is highly variable, and (c) early-line deployment where you need inspection capability before a labelled dataset is available. Anomaly detection models typically require threshold calibration per product family and careful monitoring of score distribution drift.

Choosing the right model pattern:

4. Decision logic and safety layer

Threshold raw model scores and aggregate across multiple views, cameras, or time steps depending on the station design. The decision logic layer is not just a score comparator it is the point where your inspection system makes legally and operationally consequential commitments about part quality. It must be designed with the same rigour as the rest of your safety architecture.

Score thresholding and aggregation

‍

Raw model scores must be mapped to discrete pass/fail/indeterminate outcomes using thresholds that are:

• Calibrated per product family: not carried over from a different part variant or training run.

• Validated against your precision/recall requirements: the threshold that minimises false rejects is not the same as the threshold that minimises missed defects. Which matters more depends on defect criticality and downstream cost, and this must be an explicit, documented decision.

• Reviewed when the model is updated: a new model version with different score distributions requires threshold recalibration, not just redeployment.

Where multiple cameras or time steps cover the same station, define an explicit aggregation rule (e.g., reject if any camera scores above threshold, or reject if majority vote across N frames exceeds threshold) and document the rationale. Aggregation rules have direct false-reject and miss-rate implications and must be part of the station's quality specification.

‍

Failure semantics a documented safety and quality strategy, not a configurable default

‍

When preprocessing or inference fails to produce a valid result within the budgeted time window due to a timeout, hardware fault, model error, or corrupt frame  the system must take a defined action. This is your failure semantic, and it is one of the most consequential design decisions in the entire stack. It must not be treated as a configurable option or a sensible default  it must be the output of a documented safety and quality analysis, specific to each station and each defect class.

‍

The two most common failure responses are:

• Forced reject: The part is treated as defective and ejected. Appropriate when the defect class being inspected is safety-critical (e.g., structural welds, brake components, airbag housings) and the cost of a missed defect field failure, recall, injury vastly exceeds the cost of discarding a potentially good part.

• Line stop: The line halts and requires operator intervention before resuming. Appropriate when the inspection result is required for traceability or downstream process decisions, and when an uninspected part must not proceed under any circumstances regardless of whether it is likely good or bad.  

However, neither forced reject nor line stop is universally correct, and assuming one or the other without analysis is a design error:

• On a high-volume line producing low-criticality cosmetic parts, a forced reject on every inference timeout would cause unacceptable throughput loss and scrap cost, with no safety justification.

• On a line with a manual re-inspection buffer downstream, an uninspected part routing to re-inspection may be a more appropriate response than either forced reject or line stop.

• On a safety-critical station with no downstream safeguard, line stop may be mandatory regardless of throughput impact.  

The failure semantic for each station must be determined by:

• Defect criticality: Is the defect class safety-critical, quality-critical, or cosmetic? Safety-critical defects (those with a path to field injury or recall) warrant the most conservative failure response.

• Product flow and downstream safeguards: Is there a re-inspection station, a manual audit step, or a containment buffer downstream that can catch an uninspected part? If yes, routing to re-inspection may be acceptable. If no, the part must not pass.

• FMEA and safety analysis: The failure mode, its effect, and the recommended detection and response must be formally documented in your Failure Mode and Effects Analysis. The FMEA output not engineering intuition should drive the failure semantic choice.

• Line stop impact assessment: For each station, quantify the cost and operational impact of a line stop (mean time to restart, downstream starvation, scrap generated during restart). This informs whether line stop is operationally viable as a failure response or whether a softer response (route to re-inspection) is more appropriate at that station.

Failure semantic specification per station:

‍

Document the failure semantic explicitly in the station's quality and safety specification, covering at minimum:

This specification must be reviewed and signed off by both the quality engineering and process engineering teams  not decided unilaterally by the software team because the consequences of getting it wrong fall outside the software domain.

5. PLC integration and deterministic I/O

The PLC integration layer is where the soft, service-oriented world of the inference stack meets the deterministic, hard real-time world of industrial control. The goal is not simply to avoid a particular protocol it is to enforce a well-defined adapter boundary that preserves the determinism guarantees of the control domain regardless of what happens in the software layer above it.

Why the boundary matters

‍

The inference stack — FastAPI service, ONNX Runtime, preprocessing containers — is built from components that are deliberately non-deterministic in their timing: garbage-collected runtimes, dynamic memory allocation, OS-scheduled threads, and service frameworks with variable request handling latency. The PLC, by contrast, executes its scan cycle with microsecond-level determinism and expects signals to arrive within precisely bounded windows. Allowing the inference stack to communicate directly with the PLC — via HTTP, a shared database write, or an unstructured socket — imports the non-determinism of the software layer into the control domain. The adapter boundary exists specifically to prevent this.

What the adapter boundary must enforce

‍

The IO adapter is the sole interface between the inference stack and the PLC. It must enforce the following properties not as implementation conveniences, but as non-negotiable interface contracts:

• Protocol isolation : The adapter accepts decisions from the inference stack over an internal, process-local interface (shared memory, Unix socket, or bounded in-process queue). It translates these decisions into the PLC's native communication protocol digital I/O, Profinet, EtherNet/IP, or Modbus without exposing the PLC to any service-oriented protocol. The PLC never initiates or receives HTTP, REST, or message-broker traffic. This is not a preference it is an interface discipline that keeps the control domain's timing model intact.

• Bounded decision queue: Decisions from the inference stack are placed into a fixed-capacity queue at the adapter boundary. The queue size is determined by the maximum number of in-flight parts between the inspection station and the actuator. If the queue is full because the inference stack is producing decisions faster than the PLC can consume them, or because a backlog has developed new decisions are dropped with a logged warning, not buffered indefinitely. Unbounded queuing in the control path creates the risk of the PLC acting on a stale decision for a part that has already passed the actuator.

• Explicit staleness timeout: Every decision entering the adapter carries the frame ID and acquisition timestamp from the FrameContext established at trigger time. The adapter checks the age of each decision before writing it to the PLC. If the decision is older than the configured staleness threshold (derived from your cycle time and part-to-actuator travel distance), it is discarded and logged not forwarded. A stale pass/fail signal is worse than no signal: it actuates on the wrong part.

• Deterministic write timing: The adapter's PLC write loop runs on a dedicated, real-time-priority thread (or process) with CPU affinity pinned to an isolated core where possible. It must not share execution resources with the inference service's thread pool, HTTP handlers, or logging subsystems. On Linux, use SCHED_FIFO or SCHED_RR scheduling for the write thread and explicitly disable CPU frequency scaling on the pinned core to avoid latency spikes from power management.

• Fail-safe mode on adapter fault: If the adapter itself faults (queue overflow, IPC read error, watchdog timeout), it must transition to a defined fail-safe state before stopping. The fail-safe output whether that is a forced reject signal, a line-stop signal, or a safe-state hold is determined by the station's documented failure semantic (as defined in Step 4), not by the adapter's implementation defaults.

Implementation guidance

‍

Implement the IO adapter in C, C++, or Rust — languages that offer deterministic memory allocation (no garbage collector), fine-grained thread scheduling control, and direct access to system calls needed for real-time priority assignment. Avoid Python or JVM-based implementations for the adapter's write loop, even if the inference stack itself uses them, because GC pauses and JIT compilation events introduce exactly the kind of non-deterministic latency spikes the adapter boundary is designed to contain.

‍

A minimal adapter design:

‍

// Rust pseudocode — adapter write loop 

loop { 

    if let Some(decision) = queue.try_pop() { 

        let age_ms = now_ns().saturating_sub(decision.frame_ts_ns) / 1_000_000; 

        if age_ms > STALENESS_THRESHOLD_MS { 

            log_stale_drop(&decision); 

            continue; // discard — do not forward to PLC 

        } 

        plc_io.write(decision.frame_id, decision.outcome, decision.model_version); 

        metrics.record_plc_write(age_ms); 

    } 

    thread::sleep(Duration::from_micros(SCAN_INTERVAL_US)); 

} 

 

‍

This pattern keeps the control surface narrow, testable, and fully isolated from the service layer above it the PLC sees only bounded, timestamped, staleness-checked signals, regardless of what the inference stack is doing.

Key Takeaways

A deterministic edge inference stack is not a single component it is a pipeline of five tightly coupled layers, each with its own latency budget, failure mode, and jitter envelope. The most consequential design decisions in this stack are not model or runtime choices: they are the failure semantics at the decision logic layer (forced reject vs. line stop, and why) and the adapter boundary at the PLC integration layer (what prevents the soft, non-deterministic inference world from contaminating the hard real-time control domain).

‍

Every frame should carry a unique, immutable correlation ID from hardware trigger to PLC write. Without it, post-incident debugging and in-field accuracy measurement are guesswork.

‍

Building a correct inference stack is a necessary condition  but it is not sufficient. Part 4 covers how to package, validate, and deliver that stack reliably to a fleet of edge devices: the FastAPI inference service implementation, CI/CD pipeline design, hardware-in-the-loop validation gates, and orchestrated rollout strategies that prevent a bad model update from reaching a production line.