· 21 min read
Securing Air-Gapped Synthetic Data Platforms in the Enterprise
A practical security architecture for disconnected synthetic-data deployments: controlled imports, strong internal boundaries, offline identity, defensible exports, and local evidence that the controls work.
By Shamiul Shifat
Lead AI Platform Engineer · Betterdata AI
In this article
An enterprise can remove internet connectivity and still expose sensitive data through an overprivileged administrator, an imported model, a forgotten checkpoint, or an export approved against the wrong assumptions. An air gap changes the available attack paths. It does not remove the need to understand them.
As a Lead AI Platform Engineer at Betterdata AI, I think about synthetic-data security as a responsibility across the entire data lifecycle: how information enters, how computation uses it, what artifacts remain, and how results leave. In a disconnected environment, that responsibility also includes operating security controls without depending on services outside the boundary.
This article presents my engineering recommendations and illustrative designs. It does not disclose Betterdata’s architecture, customer environments, internal controls, or incident history. It is a companion to Building AI Platforms at Scale and Operating Synthetic Data Platforms Efficiently.
Define the boundary before choosing the controls
I use air-gapped here to mean an environment with no persistent external network connectivity, where crossing the boundary requires a controlled transfer process. A private cluster with restricted internet egress, a cloud private endpoint, or a segmented corporate network has a different connectivity model. Those designs can be appropriate; their security claims should describe their actual boundaries.
The definition must account for management interfaces, vendor support, backup replication, wireless hardware, remote consoles, and dual-connected administrator workstations. A diagram that excludes the management network can miss the most powerful access path in the system. A temporary maintenance connection changes the threat model for its entire duration and needs explicit authorization, controls, and evidence of removal.
NIST’s zero-trust architecture guidance rejects implicit trust based solely on network location. That principle remains useful inside a disconnected environment: being inside the perimeter should not grant authority to read a dataset, impersonate a workload, or approve an export. 1
Start with a boundary statement that identifies the protected data classes, permitted users, trusted administrators, approved transfer mechanisms, and excluded capabilities. Record whether the environment protects departments from accidental cross-access, separates mutually untrusted tenants, or must resist a compromised privileged operator. Those are substantially different objectives.
Threat-model the complete synthetic-data lifecycle
My threat model starts with assets and authority rather than a list of security products. Raw data is an obvious asset. Training checkpoints, learned encodings, evaluation samples, embeddings, generated datasets, credentials, audit records, and recovery material can also be sensitive.
| Threat or failure | Why the air gap is insufficient | Design response |
|---|---|---|
| Malicious or compromised insider | Authorized access already exists inside the boundary | Separate duties, narrow privileges, review privileged activity, and constrain exports |
| Compromised release or model artifact | Controlled import still introduces executable behavior | Verify provenance, inspect content, quarantine, and test before promotion |
| Cross-tenant access | A shared scheduler or storage service can misapply authorization | Enforce tenant identity at every data boundary and test denied access |
| Removable-media misuse | Media can carry malicious content or unauthorized data | Approved devices, custody records, dedicated transfer stations, and direction-specific procedures |
| Residual data | Scratch storage, checkpoints, and backups outlive the job | Inventory artifacts, assign retention rules, and validate cleanup and recovery |
| Unauthorized output release | Synthetic output may preserve sensitive information | Apply purpose-specific evaluation and an accountable export decision |
| Security-control outage | Logs, certificates, clocks, and scanners still need maintenance | Monitor control health and define safe operating limits |
Document which adversaries a control is intended to resist. Restricting a service account can contain an application compromise; it does not automatically contain a host administrator who controls that application’s memory and storage. Encryption at rest does not protect plaintext from a process authorized to decrypt it.
NIST SP 800-53 provides a broad control catalog covering areas including access control, auditing, media protection, incident response, and contingency planning. I use it as a coverage reference, not as evidence that a particular installation is secure or certified. 2
Separate internal trust zones and administrative authority
An illustrative deployment has distinct zones for transfer quarantine, release repositories, platform management, sensitive computation, approved outputs, and security evidence. The important design property is constrained communication between them, with explicit identities and authorized operations.
The quarantine system should not have a convenient route into production storage. Generation workers should not need privileges to alter the registry or security policy. An output-serving component should not inherit access to training inputs. Administrators should use managed access paths with individual identities, strong authentication, and recorded privilege elevation appropriate to the environment.
A conceptual deployment, not a Betterdata infrastructure diagram. Imports and exports are controlled transfer procedures, not persistent network connections.
Versioned release bundle, signatures, dependency inventory, vulnerability information, and installation instructions.
Controlled import · quarantine, verify, authorize, record
Enterprise air-gapped enclave
Platform control
Internal identity, authorization, admission, scheduling, policy, and run history.
Sensitive computation
Approved source access, isolated training and generation, restricted scratch space and checkpoints.
Evaluation and release
Utility and privacy evidence, recipient-specific authorization, and a controlled artifact store.
Local operations
Internal registry, keys and certificates, time services, monitoring, audit storage, and tested recovery.
Separate workload identities and explicit network rules govern internal communication. An enclosing network boundary does not grant trust between zones.
Controlled export · review contents, authorize recipient, record release
A dataset or sanitized support package may leave only through the enterprise's authorized transfer process. Generation success grants no export permission.
- Acquire and record
Obtain an approved release, verification material, dependency inventory, and custody record outside the protected boundary.
- Transfer to quarantine
Use the approved transfer mechanism; inspect artifacts without production credentials or sensitive datasets.
- Verify and promote
Validate signatures and policy, test the complete bundle, then promote immutable artifacts to internal repositories.
- Compute under constraint
Run with scoped identities, isolated storage, explicit network policy, and locally collected security evidence.
- Evaluate and authorize export
Review intended use, utility, privacy evidence, recipients, and the exact artifact before any outbound transfer.
Kubernetes namespaces are useful organizational and policy scopes. A namespace by itself is not a strong security boundary between hostile workloads. Kubernetes’ multi-tenancy guidance explains the additional isolation considerations, while its security checklist provides practical hardening guidance. 3 4
I recommend restricted pod policies, non-root processes where supported, minimal Linux capabilities, constrained service accounts, seccomp profiles, controlled volume access, and explicit network policies enforced by a compatible implementation. Validate that policy enforcement actually occurs. Creating a policy object in an unsupported networking setup does not create the intended protection.
For workloads that cannot share a trust boundary, evaluate dedicated nodes, stronger runtime isolation, or separate clusters. Consider control-plane access, storage credentials, node agents, and the host kernel as well as application traffic. GPU sharing adds device, driver, runtime, and memory-isolation considerations. The appropriate boundary depends on the hardware and vendor-supported configuration; a scheduler label is not isolation evidence.
Do not assume that ending a container erases accelerator memory or every host-side buffer. Establish supported reset, reuse, and sanitization behavior for the deployed hardware. When that behavior cannot satisfy the threat model, avoid reuse across the relevant trust boundary rather than promising an unverified wipe.
Make enterprise identity work when disconnected
Prefer enterprise identity integration where an internal identity service is available within the boundary. An OIDC integration that depends on external discovery, signing-key retrieval, multifactor services, or token validation endpoints is not fully operational offline merely because an initial login succeeded.
Test fresh logins, signing-key rotation, expired sessions, disabled accounts, and identity-provider outages. If the enclave uses a separate internal identity realm, define how joiner, mover, and leaver changes arrive and how quickly urgent access revocation must take effect. An employee’s departure from a connected directory does not automatically invalidate an offline credential.
Human and workload identities should be distinct. Give each service the minimum permissions it needs, scope authorization to the tenant and operation, and avoid shared long-lived administrative tokens in images or notebooks. Where short-lived credentials are used, issuance and renewal must remain available internally. Short lifetime is valuable only if the system can renew safely and handle expiry predictably.
Local public-key infrastructure needs owners for issuance, trust distribution, rotation, revocation, and recovery. Depending on the architecture, revocation may use internally reachable responders or refreshed revocation lists. Specify how stale revocation information affects access. Silently skipping revocation checks because the environment is offline converts a planned control into an undocumented exception.
Clocks are a security dependency. Certificate validity, token expiry, event ordering, and signed evidence depend on trustworthy time. Use an approved internal time source, monitor skew and time changes, and test restart behavior when time synchronization is unavailable. Record collection time separately from event time when delayed transfers or disconnected agents can change arrival order.
Break-glass access should be individually attributable where feasible, limited, protected, and reviewed after use. Test it before an emergency. Also test its revocation: a recovery account that becomes the everyday workaround undermines the identity model.
Import a complete, verifiable release—not a collection of files
Offline software delivery is a supply-chain problem with an additional operational constraint: missing dependencies cannot be quietly fetched later. NIST’s Secure Software Development Framework is a useful reference for secure development and software acquisition practices. SLSA’s artifact-verification guidance helps frame provenance verification. Neither a checklist nor an attestation makes an artifact harmless. 5 6
I would package an approved release with an immutable manifest identifying container images, platform packages, model artifacts, native libraries, accelerator compatibility requirements, configuration schemas, database migrations, and rollback instructions. Include an SBOM, relevant licenses, security advisories or scan evidence, and a verification procedure that works inside the enclave.
Model dependencies deserve explicit treatment. A model loader may deserialize executable objects, request custom code, download a tokenizer, contact a model registry, or create caches outside the expected directory. Treat unfamiliar artifacts as untrusted input and inspect supported loading behavior. Prefer constrained formats and disable remote-code execution where applicable; format choice does not remove the need for patched parsers and resource limits. OWASP’s Machine Learning Security Top Ten is a useful candidate reference for the broader ML attack surface. 7
The bundle should be signed, but verification must answer more than “is there a signature?” Establish the trusted signer, expected builder or release process, exact artifact digest, applicable policy, and handling of revoked or superseded trust material. Obtain trust roots through an independently approved process. A compromised archive containing its own replacement verification key should not be able to redefine who is trusted.
If the signing design relies on online certificate or transparency services, define how the necessary verification evidence is captured and validated offline. Do not resolve a connectivity problem by disabling verification. Test a modified artifact, wrong signer, mismatched provenance, and unsupported verification state; promotion should fail with an actionable reason.
Vulnerability intelligence also becomes an imported dependency. Record the database’s source, version, import time, and age. A scan using old intelligence means “no findings under this database and configuration,” not “free of known vulnerabilities.” Define a maximum acceptable age, an exception owner, and an urgent update path. Keep scanner updates compatible with the offline runtime and validate their signatures as well.
Before promotion, run the complete installation with outbound access unavailable and observe attempted connections. Check license activation, telemetry defaults, crash reporting, package managers, DNS behavior, model downloads, and update checks. The acceptance target is a documented, self-contained operating envelope. Undocumented phone-home behavior should become a finding even when the firewall blocks it.
Protect data beyond the primary dataset
I recommend maintaining an artifact inventory tied to the job lifecycle. It should include input staging, preprocessing outputs, temporary files, training state, evaluation artifacts, generated output, crash dumps, logs, and backups. Assign each category an owner, classification, authorized readers, retention rule, and disposal method.
This inventory prevents a common conceptual error: deleting the original input while retaining another artifact that still reveals information about it. A learned model or checkpoint may require sensitive handling even when it does not contain an obvious table of original rows. Lineage metadata can also disclose customer relationships, dataset names, or the existence of a sensitive project.
Use enterprise-managed encryption and key controls where available, potentially backed by an internal HSM or other approved key-management system. Separate key administration from routine workload operation where the threat model requires it. Document which identities can request decryption and what happens when the key service is unavailable.
Recovery material needs the same seriousness as production keys. Keep approved offline recovery arrangements, establish custody and access procedures, and test recovery in an isolated environment. A backup that cannot be decrypted after a key rotation is not a successful recovery strategy. An easily accessible recovery key can nullify carefully restricted production access.
Deletion claims should describe actual behavior. File removal, retention expiry, key destruction, and media sanitization are different mechanisms. Snapshots, replicated storage, exports, and backups may retain copies. Cryptographic erasure only addresses the copies covered by the relevant encryption and key lifecycle; it does not retract previously decrypted copies. Define what the platform can prove and where another owner must act.
Treat synthetic output as a release decision
Synthetic data is not automatically anonymous, safe to export, or suitable for every use. Generation can preserve useful structure while retaining disclosure risk. The right release decision depends on the source, generation process, evaluation, recipients, and intended use.
Separate utility evidence from privacy evidence. Useful checks may include downstream task performance, constraint validity, coverage of relevant groups, and failure modes for rare patterns. Privacy assessment may consider attacks and disclosures relevant to the threat model. Passing a particular empirical test establishes evidence under that test’s assumptions; it does not establish a universal absence of leakage.
If a workflow claims differential privacy, document the protected unit, mechanism, privacy parameters, accounting method, and composition assumptions. Distinguish post-processing of an already differentially private release from running additional mechanisms that access protected source data. NIST SP 800-226 addresses evaluation of differential privacy guarantees and is the relevant reference for examining such claims. 8
Make approval refer to the exact artifact digest and destination. A changed dataset, a new recipient, or a broader intended use may require a new decision. Attach restrictions, expiry where meaningful, and accountable ownership to the export record. Removing the network connection does not simplify the consequences of approving a removable-media transfer.
The transfer process must inspect the whole export package. Reports, manifests, notebook outputs, debug files, and sample rows can travel beside the intended dataset. Technical approval should confirm content and evidence; the organization’s data-governance or privacy authority should own the relevant release decision. Neither role should be represented by a generic “job succeeded” status.
Keep observability local and protect its evidence
An air-gapped deployment should provide useful operational and security visibility without SaaS telemetry. Run the required collectors, dashboards, alert routing, and storage internally. Ensure the alert reaches an accountable person during an actual incident; a dashboard visible only to the failed platform’s operators is a fragile dependency.
Collect enough metadata to reconstruct authorized actions and failures while minimizing sensitive content. Useful fields include an opaque actor identifier, action, target class, tenant scope where permitted, outcome, request identifier, policy version, and timestamps. Avoid raw input rows, prompts containing protected data, access tokens, private keys, and unfiltered exception payloads.
Kubernetes audit logging can record API activity, but audit policy and backend configuration determine what is captured and what happens under pressure. It does not automatically capture every application-level data access or export decision. Those require explicit application audit events. 9
An illustrative event might be:
{
"eventType": "artifact.export.authorization",
"eventId": "evt_example_0042",
"occurredAt": "2026-10-29T03:00:00Z",
"actorId": "actor_opaque_example",
"artifactId": "artifact_opaque_example",
"policyVersion": "export-policy-example-v3",
"decision": "denied",
"reasonCode": "RECIPIENT_SCOPE_MISMATCH",
"requestId": "req_example_019"
}
Opaque identifiers still link activity to people or projects, so treat the event store as sensitive. Keep detailed resolution information under appropriate authorization. Prefer a separate evidence destination with constrained writers and tightly controlled deletion. Append-only storage, signed records, or hash chains can improve tamper evidence under a defined trust model; they do not make a fully compromised collection and signing system trustworthy.
| Signal | Why it matters | Operational response | Interpretation pitfall |
|---|---|---|---|
| Unexpected outbound attempts | Reveals hidden dependencies, misconfiguration, or possible malicious behavior | Attribute the process, preserve context, and assess containment | A blocked connection is evidence of enforcement, not proof the workload is benign |
| Audit drops or collection gaps | Removes the evidence needed to investigate activity | Restrict affected sensitive actions according to policy and restore collection | No events can mean no activity or a broken collector |
| Evidence-store disk pressure | May cause loss of audit coverage or availability | Apply capacity and retention procedures before exhaustion | Aggressive cleanup can destroy required evidence |
| Privilege and policy changes | Can expand access or disable controls | Verify the approved change and investigate unexpected changes | Approved automation can generate legitimate high volumes |
| Vulnerability database age | Indicates the freshness of exposure assessment | Import verified intelligence or invoke a documented exception | A clean stale scan can create false confidence |
| Clock skew and certificate expiry | Threatens authentication and event reconstruction | Restore trusted time or renew credentials through the internal process | A time shift can make multiple systems fail together |
| Unusual data or export access | May indicate misuse or a compromised account | Review scope, destination, and associated approvals | A large legitimate job can resemble abuse |
Define which operations stop when evidence is unavailable. For example, an organization may require export authorization to fail closed while allowing a contained computation to finish without releasing output. The choice should be explicit, tested, and consistent with recovery needs; indiscriminately stopping all services can prevent the investigation itself.
If agents temporarily spool events locally, define a bounded, protected buffer with capacity alarms, replay behavior, duplicate handling, and a policy for exhaustion. A queue cannot preserve an unlimited outage. State when new sensitive work must stop, what already-running work may safely finish, and who can authorize an exception. Recovery should reconcile gaps and record any known evidence loss rather than silently returning the dashboard to green.
Measure coverage and response, not the absence of alerts
I would maintain a small security scorecard alongside operational service objectives. Each measure needs a definition, an owner, a review interval, and a response when it falls outside the enterprise’s agreed limit. The limits should follow the threat model and contractual obligations; a universal numerical target would hide important differences between deployments.
| Measure | How I would define it | What the result can establish |
|---|---|---|
| Required audit-event reconciliation | Compare durable records of sensitive operations with corresponding received audit events, accounting for documented delivery delay and duplicates | Identifies missing evidence for the operations included in the comparison; does not prove all access paths are instrumented |
| Revocation propagation time | Time from an authorized revocation decision to denial on each relevant API, session, and workload path | Shows whether cached credentials and disconnected identity updates meet the stated revocation objective |
| Exposure-assessment freshness | Age of the imported vulnerability intelligence, last successful scan, and inventory against which it ran | Makes stale assessment visible without representing a clean scan as absence of vulnerabilities |
| Critical control-test coverage | Relevant deployed boundaries with current passing tests divided by the documented in-scope boundaries | Shows gaps and stale test evidence; an incomplete boundary inventory still produces misleading coverage |
| Detection and response latency | Time from a safely injected test event to its receipt, actionable alert, acknowledgement, and defined containment action | Tests the whole response path instead of measuring collector uptime alone |
| Demonstrated recovery | Measured restore time and recoverable data point, including keys, policy, authorization, and audit continuity | Compares actual recovery exercises with recovery objectives, rather than treating backup completion as proof |
Treat numerator and denominator definitions as part of the evidence. For example, reconciling only successful exports would miss denied export attempts that policy also requires auditing. Checking disabled-account behavior with a fresh login would miss an existing session that remains valid. Reconcile both expected events and enforcement behavior.
Review distributions and worst observed cases where they matter, rather than relying only on averages. Keep the test version, deployment configuration, and known gaps beside the scorecard. A configuration change can invalidate earlier assurance even when the dashboard’s numbers remain green.
These are proposed engineering measures, not a certification scheme or a claim that every compromise can be detected. Their purpose is to expose unsupported assumptions and drive a specific corrective action.
Prove controls with negative tests and recovery exercises
Security acceptance should verify behavior at the boundary. A configuration screenshot is useful context, but it does not demonstrate that the intended denial occurs. Test with representative identities, actual enforcement components, and the same deployment mode used in production.
- State the threat
Identify the actor, protected asset, prohibited action, and trust assumptions.
- Exercise the boundary
Attempt the prohibited action through realistic interfaces and failure conditions.
- Inspect the evidence
Confirm denial, absence of unintended side effects, and a useful protected audit record.
- Retest after change
Repeat relevant checks when software, hardware, policy, identity, or connectivity changes.
I would require at least these exercises before accepting a deployment:
- Tenant boundary: use a valid identity from one tenant to access another tenant’s source, job, checkpoint, output, and status endpoint. Check object storage and direct API paths as well as the UI.
- Outbound denial: run installation and representative jobs without external connectivity; inspect attempted DNS and network access and verify allowed internal paths still work.
- Import tampering: modify a signed bundle, supply an unauthorized signer, and mismatch artifact provenance. Confirm rejection before production promotion.
- Identity lifecycle: disable an account, rotate signing keys, expire a certificate, and exercise revocation with fresh and stale internal evidence. Verify the documented access outcome.
- Evidence failure: interrupt collection and approach storage limits in a controlled environment. Confirm alerts, retention behavior, and the defined restriction on sensitive actions.
- Backup restore: restore data, metadata, policy, and necessary key access in isolation. Check authorization after recovery and confirm that expired or revoked privileges do not silently return.
- Export denial: attempt a release with a mismatched artifact, missing approval, or unauthorized recipient. Inspect the full package for unintended accompanying data.
- Cleanup and reuse: cancel and crash jobs, then examine retained artifacts and supported device reuse behavior against the documented retention and isolation requirements.
Keep the test’s version context and limitations with the result. A tenancy check against one storage adapter does not prove a newly installed adapter behaves identically. A recovery exercise without realistic key controls does not prove that recovery will succeed under those controls.
Prepare incident response without an emergency internet dependency
An incident plan needs to function inside the same boundary as the incident. NIST SP 800-61 Revision 3 frames incident response within cybersecurity risk management. For a disconnected platform, I translate that into locally available playbooks, designated decision makers, protected evidence collection, and rehearsed recovery procedures. 10
Containment should account for both computation and data movement. Depending on the incident, suspend imports, exports, a specific tenant, a credential, or an affected worker pool. Preserve the evidence needed to decide scope. Avoid reflexively deleting the pod, reimaging the host, or clearing scratch storage before the incident team determines what must be retained.
An urgent patch still needs controlled import and verification. Prepare the expedited process before an incident, with explicit authority and compensating controls when normal testing cannot be completed. Break-glass access needs attribution, bounded use, and a review of changes and retained privileges afterward.
Vendor support is another potential export path. Build a diagnostic package that excludes raw records, secrets, sensitive model artifacts, and unnecessary identifiers. Review its content and obtain the required export approval. “Support needs the logs” should start a scoping conversation, not override the transfer policy.
After recovery, verify trust material, identities, registry contents, policy, and monitoring configuration as well as service availability. A restored dashboard does not demonstrate that the environment has returned to a known trustworthy state.
Assign ownership and make acceptance explicit
I recommend a small responsibility matrix agreed by the enterprise and platform team. The following is illustrative; A means accountable, R responsible for execution, and C consulted. Assign one accountable owner for each decision in the actual deployment.
| Decision or activity | Platform team | Enterprise security / infrastructure | Data governance / privacy |
|---|---|---|---|
| Release packaging and supported runtime | A/R | C | C |
| Enclave boundary, identity, and key infrastructure | C | A/R | C |
| Workload policy and control acceptance | R | A | C |
| Dataset export decision | R | C | A |
| Incident coordination and evidence handling | R | A | C |
Acceptance should require a signed-off boundary and threat model, verified installation material, working internal identity and key lifecycle, tested authorization and isolation, protected local evidence, demonstrated recovery, and an operational export process. Record open exceptions with owners and review dates. An unresolved limitation should not disappear into a green deployment checklist.
The strongest air-gapped platform is one whose operators can explain every approved crossing, every important privilege, and every claim made about released data—and can produce evidence that those explanations match the running system. The absence of an internet route is one useful control. Engineering that makes the remaining responsibilities explicit is what turns it into a defensible enterprise deployment.
References and further reading
- NIST. Zero Trust Architecture, SP 800-207. 2020. Reference for avoiding implicit trust based on network location.
- NIST. Security and Privacy Controls for Information Systems and Organizations, SP 800-53 Revision 5. 2020. A control catalog for access, auditing, media, contingency planning, and related responsibilities.
- Kubernetes. Multi-tenancy. Guidance on tenancy and the limits and requirements of isolation in shared clusters.
- Kubernetes. Security Checklist. Practical hardening considerations; applicability depends on the installation.
- NIST. Secure Software Development Framework (SSDF) Version 1.1, SP 800-218. 2022. Practices relevant to secure software development and acquisition.
- SLSA. Verifying artifacts, version 1.0. Guidance for evaluating artifacts against expected provenance and policy; cited as a versioned reference, not a claim about the latest specification.
- OWASP. Machine Learning Security Top Ten. A candidate reference for ML-specific threats; format and loader choices still require product-specific assessment.
- NIST. Guidelines for Evaluating Differential Privacy Guarantees, SP 800-226. 2025. Reference for assessing formal differential privacy claims and assumptions.
- Kubernetes. Auditing. Kubernetes API audit policy, backends, and operational considerations.
- NIST. Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile, SP 800-61 Revision 3. 2025. Incident response guidance within cybersecurity risk management.