← Back to writing

· 14 min read

Edge, Cloud, or Hybrid: Placing AI Workloads Across Trust Boundaries

A practical placement framework for synthetic data and enterprise AI: data gravity, latency, local autonomy, cloud capabilities, identity, synchronization, and recovery.

By Shamiul Shifat
Lead AI Platform Engineer · Betterdata AI

In this article

“Should this run at the edge or in the cloud?” is usually too broad a question to produce a useful architecture. An AI system contains several workloads with different constraints: data preparation, model training, inference, evaluation, policy decisions, artifact distribution, and operational reporting. They do not all need the same home.

As a Lead AI Platform Engineer at Betterdata AI, I approach placement by asking which information crosses which boundary, who controls the operation on each side, and what must continue when the connection disappears. That perspective matters in synthetic data and privacy technology, where moving a workload can change both its operating cost and its exposure.

The framework below describes how I would make those decisions. Examples and calculations are hypothetical, not accounts of Betterdata’s infrastructure, customer deployments, or measured performance.

Distinguish embedded edge, local infrastructure, and cloud

Embedded edge means compute close to a device or physical process: a gateway, industrial computer, vehicle, camera, or other constrained endpoint. Power, thermal limits, intermittent connectivity, physical access, and a limited service lifetime may dominate the design.

On-premises infrastructure means servers or clusters operated in an organization’s facilities or another controlled local environment. It may have substantial compute, storage, networking, and operational staff. It is not automatically resource-constrained simply because it is outside a public cloud.

Cloud means infrastructure and managed capabilities delivered through a provider’s service model. It can reduce some infrastructure responsibilities and offer flexible capacity, but region, service availability, network dependencies, contractual conditions, and shared responsibilities still matter.

Hybrid means responsibilities span those environments. It is not a guarantee of portability, resilience, or cost savings. A hybrid system can combine their advantages, or inherit the operating complexity of all of them. Start with the required behavior rather than treating hybrid as the default compromise.

Place four planes independently

I separate the data plane, which handles source data and derived artifacts; the workload plane, which runs preprocessing, training, generation, inference, and evaluation; the control plane, which handles identity, policy, scheduling, and lifecycle decisions; and the telemetry plane, which carries operational evidence.

This is a reasoning model, not a requirement to build four services. Its value is revealing assumptions. A local inference worker may still depend on a cloud identity service, a remote model registry, or an external logging endpoint before it can start. Its compute is local, but its availability is not autonomous.

PlaneQuestions that determine placementA possible hybrid choice
DataWhich raw and derived data may leave, for what purpose, and under whose control?Keep raw records local; release only specifically approved artifacts
WorkloadWhat are the latency, capacity, memory, and evaluation requirements?Run local preprocessing and generation; use permitted cloud capacity for selected workloads
ControlWhich decisions must remain available during an outage?Coordinate inventory centrally while enforcing a scoped policy locally
TelemetryWhat evidence is necessary, sensitive, or too large to transmit?Retain detailed local logs and synchronize minimized status records

Document every cross-plane dependency. Then test whether the promised local operation still works when each remote dependency is unavailable.

Start with trust and purpose, then assess locality

Data residency describes where data is stored or processed. Jurisdiction, contractual obligations, administrator access, subprocessors, key control, support access, backups, and the intended processing purpose create additional questions. A regional deployment alone does not establish compliance with every requirement.

Map raw records, temporary files, generated datasets, model checkpoints, evaluation reports, logs, and backups. A system can keep its primary database local while exporting sensitive information through traces, crash reports, or model artifacts. Treat those flows as part of the architecture rather than incidental implementation detail.

NIST’s zero-trust architecture guidance rejects implicit trust based solely on network location.1 Apply that principle in both directions: a device inside a factory is not automatically trusted, and a workload in an approved cloud account still needs scoped identity and authorization.

For synthetic data, avoid a shortcut in which every generated output is deemed safe to move. Generation does not automatically establish anonymity. Where differential privacy is part of the protection, its guarantee depends on the mechanism, privacy unit, accounting, and implementation assumptions described in NIST SP 800-226.2 Release permission should still consider the artifact, recipient, purpose, and evaluation evidence.

Compare provider capability classes, not slogans

AWS, Azure, and Google Cloud all have offerings relevant to distributed enterprise workloads, but similar terminology does not imply identical lifecycle or connectivity behavior. Evaluate an actual supported configuration against the requirements.

Provider referenceCapability class to investigateQuestions to resolve before choosing
AWS OutpostsAWS infrastructure and selected services deployed at a local siteWhich services and accelerators are supported, what connectivity is required, and what continues during disconnection?
Azure ArcManagement and governance of resources across environmentsWhich agents and control functions require connectivity, and which workload functions remain local?
Google Distributed CloudDistributed infrastructure and platform offerings for local or restricted environmentsWhich edition and operating model meets the connected or disconnected requirement, and who manages updates and support?

The official Outposts, Azure Arc, and Distributed Cloud materials are starting points for those capability investigations.345 These are different product categories, not an apples-to-apples benchmark. Azure Arc management, for example, should not be confused with automatically supplying a complete local AI runtime.

Ask for a dependency inventory, supported hardware matrix, update process, outage behavior, licensing conditions, and evidence from a representative workload. Check regional availability and the exact edition under consideration. Do not choose a platform on an assumed price advantage, a generic “air-gapped” label, or a feature that exists in another configuration.

Measure data gravity before planning cloud bursts

Cloud capacity is only useful when the required data and artifacts can reach it within the permitted time and cost. Estimate source volume, transfer bandwidth, preprocessing location, compression, synchronization frequency, and how often the same data will be reread.

Consider a hypothetical site producing two decimal terabytes per day with a 100-megabit-per-second uplink. At the link’s theoretical maximum, transferring one day’s raw data takes:

2,000,000,000,000 bytes × 8 bits/byte
---------------------------------- = 160,000 seconds ≈ 44.4 hours
       100,000,000 bits/second

That cannot keep pace with a new batch every 24 hours, even before protocol overhead, competing traffic, or downtime. More cloud GPUs do not solve the input bottleneck. Local preprocessing, an approved smaller representation, a faster connection, or a different execution location is required.

Do not assume a smaller representation is automatically permitted to leave. Features, gradients, summaries, and model checkpoints can retain sensitive information. Evaluate the actual transformation and intended recipient. If a cloud burst requires a new data release, make that approval part of the workflow rather than an infrastructure-only scaling decision.

Also examine movement in the other direction: model downloads, dependency bundles, container layers, evaluation artifacts, and disaster-recovery copies. A fleet update may saturate a link even when normal inference traffic is tiny. Stage distribution and retain known-good local artifacts.

Use a latency and capacity budget together

Break the request path into local preprocessing, network transit, queueing, execution, postprocessing, and any required downstream action. Measure the whole journey under representative conditions. A fast model benchmark does not describe the user experience when authentication or artifact loading adds remote calls.

Here is an illustrative comparison, with invented deterministic times rather than measured percentiles:

ComponentLocal executionCloud execution
Processing time40 ms15 ms
Added network round tripNegligible for this simplified example80 ms
Total before queueing and other overhead40 ms95 ms

If the illustrative deadline is 60 ms, local execution fits this simplified budget while the cloud path does not. Real acceptance requires end-to-end tail-latency measurements, including network variation, cold starts, queueing, and failures. Adding component percentiles together is not a substitute for measuring the resulting distribution.

Capacity creates another constraint. Two equivalent local workers processing one request every 40 ms provide a theoretical maximum of 50 requests per second. At an arrival rate of 30 requests per second, nominal utilization is 60%. If one worker fails, capacity falls to 25 requests per second and backlog grows by five requests per second while that rate persists.

A third equivalent worker could preserve 50 requests per second after one worker fails, subject to CPU, memory, and other shared constraints. Three processes on one host do not protect against losing that host. Choose failure domains deliberately and test the remaining capacity rather than assuming extra replicas establish availability.

Fit the model to the operating envelope

Measure peak memory, model loading time, sustained throughput, power draw where relevant, and resource use during simultaneous tasks. A device that can load a model may still be unable to evaluate outputs, buffer requests, and apply an update safely at the same time.

Include a CPU-based baseline when it supports the task. Compare smaller models, batching strategies, quantization, and acceleration only through a consistent utility and privacy evaluation. Do not assume a converted or quantized model preserves every property of the original mechanism. Where a formal guarantee is claimed, review whether the transformation and surrounding workflow preserve its assumptions.

Account for model copies. If an illustrative model requires four gigabytes for weights in each independent worker process, three copies require twelve gigabytes for weights alone. Activations, runtime buffers, the operating system, and update staging need additional space. Shared-weight or specialized serving designs may change that calculation, so measure the actual runtime.

Licensing is part of fit. Verify rights for commercial use, redistribution into customer environments, offline operation, modification, and the intended model artifacts. A technically portable model is not automatically licensed for every deployment pattern.

Field guide A placement decision for each workload Choose an execution location only after checking the data boundary, operating budget, and behavior when dependencies fail.
  1. Constrain the data flow

    Identify the source, purpose, permitted recipients, artifact sensitivity, and allowed processing locations.

  2. Measure the workload

    Benchmark memory, compute, transfer volume, latency, and required evaluation on representative inputs.

  3. Test local autonomy

    Remove remote identity, registry, policy, and telemetry dependencies to see what actually continues.

  4. Choose and validate placement

    Compare supported configurations, lifecycle cost, failure domains, and the evidence needed for release.

Provision offline operation as a complete capability

An offline-capable deployment needs more than a container archive. Package pinned runtime artifacts, models, configuration schemas, required drivers or a supported driver contract, migration files, and recovery instructions. Check trusted digests and provenance before accepting the bundle. SLSA’s artifact-verification guidance is a useful reference for validating what was built and by whom.6

Provide a local route for authorized provisioning and credential rotation. Avoid distributing one fleet-wide secret in an installation package. Use device or workload identities with scoped permissions, and protect local keys using appropriate hardware or operating-system controls where available.

Plan certificate validity around the supported disconnected interval and the risk of long-lived credentials. If local issuance is allowed, constrain its authority and protect the issuer. If central renewal is mandatory, specify what stops when renewal becomes impossible. A certificate-expiry exception cannot be an undocumented instruction to disable authentication.

Stage upgrades with a trusted manifest, compatibility checks, available disk verification, and a tested fallback. Keep the previous usable model and runtime where the recovery design requires it. A new application binary may not be able to roll back a migrated database, so distinguish artifact fallback from state restoration or roll-forward repair.

For more detail on disconnected delivery boundaries, see Securing Air-Gapped Synthetic Data Platforms.

Define local authority and synchronization semantics

Specify which decisions a site can make independently and for how long. A valid cached policy may authorize a limited local workload. New recipients, changed purposes, or sensitive exports may require fresh approval. The right boundary depends on the risk and service requirement, not a universal rule that every offline operation must stop or continue.

Remote revocation cannot arrive over a disconnected link. Time-bounded leases limit the period of authority only if clocks and expiry enforcement are trustworthy. Instant central revocation and indefinite disconnected operation are incompatible promises. State the maximum supported delay and what happens when local time is uncertain.

For synchronization, define ownership before choosing a database. Let one authority own a release decision or allocate non-overlapping authority to sites. Record globally distinguishable event identifiers, artifact digests, policy versions, and both event time and receipt time. Deduplicate replayed events. Do not resolve conflicting approvals by simply keeping whichever message arrived last.

Treat financial quotas, license allocations, and privacy accounting separately. If disconnected sites share a global allowance, allocate bounded local portions or require coordination before spending it. Two sites cannot safely assume each owns the same remaining budget. Reconciliation should expose uncertainty and conflicts rather than silently overwriting them.

Keep monitoring useful when the network disappears

Local operators need health, queue depth and age, storage pressure, identity expiry, active versions, and recent failure reasons without a central dashboard. Buffer a bounded amount of telemetry locally and define which signals are retained when storage fills. The monitoring system should not exhaust the disk required for production work.

Synchronize the minimum evidence needed centrally. Avoid raw records, sensitive prompts, secrets, or unnecessary customer identifiers in logs. Protect detailed local diagnostics and define retention for them. Treat telemetry outage separately from workload failure so the central view shows unknown or stale status instead of falsely reporting health.

Overload controls remain necessary even when capacity is fixed. Google’s SRE guidance describes admission control and load shedding as defenses against excess demand.7 Use bounded queues, priority classes, and explicit degraded behavior. Decide which work can wait, which can be rejected, and which requires a local safety or business fallback.

Field guide Operate locally, reconcile deliberately A disconnected site needs bounded authority and local observability. Reconnection synchronizes evidence; it does not erase conflicting decisions.
  1. Receive trusted state

    Provision verified artifacts, scoped identity, current policy, and any bounded local allocations.

  2. Execute within local authority

    Enforce expiry and workload limits while retaining the evidence needed to explain each action.

  3. Handle disconnection explicitly

    Continue only supported operations, expose local health, and buffer a bounded set of events.

  4. Reconcile on reconnection

    Deduplicate events, check policy and identity changes, and escalate conflicts or uncertain accounting.

Design recovery beyond the individual worker

Define recovery-time and recovery-point objectives for the site and for the overall service. NIST SP 800-34 provides a reference for contingency planning and recovery considerations.8 The useful engineering question is what state and capability must be restored, from where, and under whose authority.

Protect the job ledger, release records, policy state, required artifacts, keys, and backups according to their sensitivity. Test a host failure, a lost site, a damaged local store, and an unavailable central control plane. A spare worker helps with execution capacity; it does not reconstruct missing authorization history.

If the plan uses cloud failover, verify that the necessary data is allowed to move and already available or transferable within the recovery objective. If failover requires the outage’s broken uplink, it is not a dependable response to that outage. Some environments need local redundancy or a deliberately reduced local service instead.

Compare lifecycle cost and pilot the complete journey

Include hardware acquisition and replacement, idle capacity, networking, egress, storage, licensing, support, site visits, security operations, power where material, and engineering effort. A lower hourly compute rate can be outweighed by distribution and maintenance across many sites. FinOps unit economics offers a useful discipline for relating those costs to delivered outcomes.9

Compare cost per supported workload class or approved dataset while holding utility, privacy, latency, and recovery requirements constant. Local hardware is not free after purchase; cloud elasticity is not useful if minimum reservations or data movement prevent releasing cost. Record uncertainty where a pilot has not established the operating effort.

Begin with one representative site and one workload. Measure normal operation, peak demand, cold start, disconnected operation, upgrade, and recovery. Validate the exact data flows and evidence required for release. Expand only after local operators can diagnose failure and the central team can explain the site’s authority without relying on assumptions.

The placement decision is successful when the workload runs within its trust boundary, meets its service objective, and remains understandable through change and failure. That may produce an edge deployment, a cloud deployment, or a carefully limited combination of both.

A practical starting blueprint

For an illustrative enterprise with sensitive local records and intermittent connectivity, I would begin with source storage, preprocessing, generation, and required evaluation in an approved on-premises boundary. A local release service would bind approval to the exact artifact and recipient. Central services would distribute verified software and policies, maintain the enterprise inventory, and receive minimized status metadata.

Add embedded inference only where the device’s latency or connectivity requirements justify it, and distribute only a model authorized for that environment. Add cloud batch capacity only for workloads whose inputs and outputs are permitted to cross the boundary. Keep those additions optional: the initial local workflow should remain operable within its documented authority when central services are unavailable.

This blueprint deliberately identifies the release authority, dependency boundary, and permitted information flow before choosing a scheduler. The pilot then establishes which parts need more capacity or stronger isolation.

For coordinating policies across those locations, continue with Enterprise AI Control Towers.

References and further reading

  1. NIST, Zero Trust Architecture, SP 800-207, 2020. Resource-focused access and the limits of trusting network location.
  1. NIST, Guidelines for Evaluating Differential Privacy Guarantees, SP 800-226, 2025. Interpreting privacy guarantees and their implementation assumptions.
  1. AWS documentation, What is AWS Outposts?. A starting point for evaluating locally deployed AWS infrastructure and its service dependencies.
  1. Microsoft Learn, Azure Arc overview. Management capabilities across environments; confirm the dependencies of each selected Arc service.
  1. Google Cloud, Google Distributed Cloud. Product-family starting point; validate the specific edition, operating model, and connectivity requirements.
  1. SLSA, Verifying artifacts, specification v1.0. A versioned reference for checking provenance against expected artifact and producer properties.
  1. Google, Handling Overload, Site Reliability Engineering. Admission control and deliberate responses to excess demand.
  1. NIST, Contingency Planning Guide for Federal Information Systems, SP 800-34 Rev. 1. Recovery planning concepts; application to a particular enterprise requires its own requirements and exercises.
  1. FinOps Foundation, Unit Economics. Relating technology costs to meaningful units of delivered value.

Department of small adventures

Pip’s pocket passport

Six little souvenirs. One very well-travelled duck.

Kept in this browser, just for you. Visit each section and collect its stamp here—or catch one from Pip.