· · 14 min read
Building AI Platforms at Scale
A practical architecture for synthetic data platforms: reliable execution, tenant isolation, measurable utility, privacy evaluation, and evidence-backed releases.
By Shamiul Shifat
Lead AI Platform Engineer · Betterdata AI
In this article
A model that generates convincing rows is a research result. A platform that can produce an authorized, useful, traceable dataset repeatedly is an engineering capability. The distance between those two outcomes is where most of the difficult work lives.
As a Lead AI Platform Engineer at Betterdata AI, I approach platform design through the demands of synthetic data and privacy technology. My first question is not how many accelerators we can provision. It is whether a team can explain what entered a run, what happened to it, what evidence supports the output, and who is allowed to use that output.
This article sets out the architecture and operating practices I recommend. The examples are illustrative design scenarios, not descriptions of Betterdata’s internal infrastructure or customer deployments.
Start with the product promise
“Scale” needs a unit. It might mean concurrent tenants, training jobs per day, dataset size, release throughput, geographical coverage, or the number of engineers who can ship safely. Each creates a different bottleneck. A platform that scales inference requests may still fail when a single large training job exhausts shared storage.
For a synthetic data product, I would define a successful request as a dataset delivered within its agreed service window, meeting its intended utility requirements, passing the applicable release policy, and retaining enough evidence to reproduce the decision. A job ending with exit code zero is only one part of that outcome.
Write this promise before selecting infrastructure. Specify the workload classes, data sensitivity, expected consumers, residency constraints, recovery objectives, and the cost envelope. A developer test fixture and a dataset released outside an organization deserve different acceptance criteria. So do an interactive preview and an overnight training run.
This is also where platform ownership becomes concrete. The platform team owns the supported execution path and its reliability. Research teams own model-specific behavior and evaluation assumptions. Security and privacy specialists help define acceptable exposure. Product owners specify what useful means. None of these responsibilities disappears because a workflow is automated.
Design three contracts before adding services
Google’s research on hidden technical debt in machine learning describes how dependencies, configuration, feedback loops, and surrounding infrastructure can dominate the maintenance burden of an ML system.1 I use that observation as a reason to make boundaries explicit early.
The first boundary is an input contract. Record the dataset version, schema, source authorization, intended purpose, sensitivity classification, retention policy, and permitted processing location. Include relational constraints and temporal meaning. A timestamp without a timezone or a foreign key without its parent table is not a minor data-quality inconvenience; it changes what the model can learn.
The second is an execution contract. A run should identify its code revision, model configuration, dependency or image digest, seed where relevant, resource requirements, and policy version. A seed helps debugging, but it does not guarantee bitwise reproducibility across different hardware and nondeterministic kernels. State the level of reproducibility you actually support.
The third is an output contract. Specify schema and constraint checks, task-based utility evaluation, privacy evaluation, authorized recipients, retention, and the evidence required for release. Version these expectations. Otherwise, “passed evaluation” becomes an ambiguous statement after the evaluation code changes.
These contracts do not require three new microservices. They can begin as versioned records, a durable workflow, and a small policy module. The objective is a stable agreement between components, not a larger service diagram.
- Authorize inputs
Resolve purpose, dataset version, access, residency, and retention before scheduling compute.
- Execute a pinned run
Record code, configuration, dependencies, resources, and model artifacts inside the approved boundary.
- Evaluate independently
Check validity, task utility, privacy risks, and the limits of each measurement.
- Release with evidence
Publish only approved artifacts to authorized consumers, with a manifest and a revocation path.
Separate orchestration from sensitive computation
I recommend a logical separation between the control plane and the data plane. The control plane authenticates requests, authorizes actions, schedules work, records state, and resolves policy. The data plane reads sensitive inputs and performs training, generation, and evaluation. They may share infrastructure initially; the responsibilities and permissions should still be distinct.
That separation makes a useful security question possible: can the component that schedules a job read the underlying personal data? If it does not need that access, do not grant it. Likewise, a release service may need an evaluation decision and an artifact identifier without needing the raw training dataset.
Use short-lived, workload-specific credentials. Separate service identities for training, evaluation, and release where the risk justifies it. Encrypt storage and transport, but do not treat encryption as a substitute for authorization: a broadly privileged process can still read decrypted data through legitimate APIs.
Multi-tenancy requires an explicit threat model. Kubernetes documentation distinguishes soft and hard multi-tenancy and describes controls such as namespaces, role-based access, quotas, and network policies.2 A namespace is an organizational boundary, not a complete answer to hostile tenant isolation. For untrusted workloads or sensitive contractual boundaries, separate clusters, accounts, or stronger workload isolation may be appropriate.
Apply the same reasoning outside compute. Check object-store prefixes and policies, temporary volumes, cached features, model registries, logs, backups, and export links. A tenant-isolated training job is of limited value if its output appears in a shared bucket with permissive access. Test negative cases: tenant A must not enumerate, fetch, cancel, or obtain metadata about tenant B’s work.
Make long-running work recoverable
Synthetic data workflows often outlive an HTTP request by several orders of magnitude. Design them as durable jobs with explicit states: accepted, queued, running, evaluating, awaiting approval, released, failed, or cancelled. Keep “generation finished” separate from “dataset available to consumers.”
A client retry should not silently create a second expensive training run. Accept an idempotency key scoped to the tenant and operation, record it durably, and return the existing operation when the same request is repeated. Reject reuse with a different payload. Within the workflow, make side effects such as publishing artifacts idempotent too; request-level deduplication does not make every internal step safe.
Use bounded retries with backoff and jitter for transient failures. Do not retry deterministic schema errors, denied access, or evaluation failures as if they were network interruptions. Preserve a failure reason that tells the developer whether to change the request, retry later, or contact an owner. Cancellation should release compute and clean up temporary artifacts under a defined retention policy.
Google’s SRE guidance on overload emphasizes controlling demand and shedding work before overload turns into cascading failure.3 For a training platform, that means admission control, bounded queues, per-tenant quotas, and separate workload classes. Interactive previews should not wait behind an unrestricted queue of large batch jobs. Nor should a premium queue starve every other tenant indefinitely.
Track queue age as well as queue length. Ten tiny jobs and ten multi-hour jobs create different experiences. Reserve enough capacity for recovery and critical evaluation; a fleet running at maximum utilization has little room to absorb a failed node or an urgent release. Autoscaling helps only when the dependency that constrains progress can scale with it.
Treat synthetic data utility as a task-specific claim
“Looks realistic” is not an acceptance test. A generator can reproduce marginal distributions while breaking relationships that matter to a downstream model. It can also preserve average performance while weakening results for a small but important population.
Start with structural validity: schema, types, nullability, ranges, primary and foreign keys, temporal ordering, and domain constraints. Then inspect statistical behavior relevant to the task, including conditional relationships, tails, missingness patterns, and class balance. A single similarity score compresses too many different failure modes into one number.
For a predictive use case, compare a downstream model trained on synthetic data and tested on an appropriately held-out real dataset with a comparable baseline trained on real data. Keep the evaluation protocol, preprocessing, model family, and tuning budget consistent enough to make the comparison interpretable. Report absolute performance and the difference from baseline, including uncertainty across runs or samples when feasible.
Protect the holdout from generator training and model-selection leakage. In longitudinal or relational data, splitting random rows can put the same person’s history or linked entities on both sides. Choose a split that reflects the intended deployment: by person, organization, time, or another meaningful unit. An apparently excellent evaluation can be an artifact of a poor split.
Segment the results where the use case requires it. For fraud detection, a rare event may matter more than the dominant class. For software testing, referential integrity and coverage of edge cases may matter more than predictive accuracy. Define these priorities with consumers before selecting thresholds. There is no universally acceptable percentage of “utility retained.”
Privacy is a property to establish, not a label to inherit
Synthetic data is not automatically anonymous. Stadler, Oprisanu, and Troncoso’s study, Synthetic Data — Anonymisation Groundhog Day, evaluates privacy and utility trade-offs and demonstrates why generating synthetic records does not by itself establish reliable privacy protection.4 The implication is practical: a release decision needs a threat model and evidence beyond the absence of obvious names or identifiers.
Ask what an adversary might know and what they might learn. Membership inference asks whether a person’s record was in a training dataset. Attribute inference asks whether hidden information can be inferred using other available information. Singling out and linkage raise additional questions about distinctive records and auxiliary data. These are related risks, not interchangeable metrics.
Exact duplicate checks and nearest-neighbor analyses are useful diagnostics. They are not proofs of privacy. Distance depends on representation, scaling, feature types, and population density. A low match rate can coexist with other disclosures. Likewise, a failed attack only shows that a particular attack, with particular assumptions and resources, did not succeed in that evaluation.
Differential privacy offers a different kind of assurance: a mathematical bound on how much a mechanism’s output distribution can change when the protected input changes. NIST SP 800-226 explains the guarantee and the implementation hazards that can undermine it.5 In practice, the definition of neighboring datasets is crucial. A record-level guarantee may not provide the intended person-level protection when one person contributes many records.
For a differentially private workflow, record the privacy unit, contribution bounds, mechanism, clipping or sensitivity assumptions, accountant, and the resulting parameters. Epsilon and delta are not a privacy percentage or a universal pass/fail score. Explain them in the context of the threat model and release policy, with specialist review where needed.
Account for composition across accesses to private data. Repeated training and private-data-dependent tuning can consume additional privacy budget. Generating more samples solely from an already differentially private model is post-processing and does not by itself add privacy loss; consulting the private data again changes the analysis. A release pipeline that publishes non-private evaluation summaries can also undermine the intended protection even if training was private.
Finally, formal privacy guarantees do not replace access control, secure execution, or purpose restrictions. They also do not guarantee that no sensitive fact about a population can be learned. Decide whether an output is appropriate for its recipients and use, rather than treating one technical property as a universal permission to distribute it.
Build a release gate that can explain its decision
The release gate should combine independent checks rather than average them into a reassuring overall score. A strong utility result cannot compensate for unauthorized source data. A privacy evaluation does not repair broken foreign keys. A fast run does not make an unreviewed export acceptable.
I would retain an immutable release manifest containing the input and model identifiers, code and configuration versions, evaluation results, privacy assumptions, policy version, approval identity when applicable, output digest, and permitted destination. Keep sensitive evaluation artifacts under appropriate access controls. “Audit evidence” is still data that may need protection.
Google’s ML Test Score work argues for testing and monitoring across data, models, infrastructure, and production behavior.6 That broader view is especially useful here: evaluate the workflow that delivers the artifact, not just the algorithm that generates it. Exercise failed approvals, stale reports, missing artifacts, partial writes, and a revoked source authorization.
A failed gate should preserve enough diagnostic evidence for an authorized engineer to understand the problem while withholding the output from consumers. An exception needs an owner, scope, reason, expiry, and review path. Avoid a permanent “skip validation” switch becoming the unofficial production workflow.
- Is it authorized?
Source permission, destination, processing location, and intended use match the active policy.
- Is it valid and useful?
Structural checks and task-specific evaluation meet pre-agreed criteria, including important subgroups.
- Is the privacy case sufficient?
The threat model, evaluation limitations, and any formal guarantee support this release context.
- Can we explain and operate it?
Lineage, approval evidence, ownership, retention, and the response to a withdrawn artifact are recorded.
Observe the user journey without leaking the dataset
Infrastructure metrics answer whether the machines are busy. Platform metrics should also answer whether users are making progress. Track time to an accepted job, queue wait, completion time by workload class, evaluation turnaround, release success, and failure categories. Define an SLO around an eligible user journey with explicit exclusions, not an ambiguous percentage of jobs that eventually finish.
Connect API requests, jobs, evaluation runs, and releases through opaque correlation identifiers. OpenTelemetry’s semantic conventions offer shared names for telemetry across components.7 Consistent instrumentation helps, but it does not decide which data is safe to record. Do not put raw samples, prompts containing personal data, access tokens, or sensitive feature values into general-purpose traces and logs.
Keep low-cardinality operational dimensions in metrics. Put per-run detail into controlled logs or trace attributes where appropriate, with retention and access limits. Review error paths as carefully as successful paths: libraries sometimes include sample records or connection details in exception messages.
Alert on symptoms that demand action: sustained queue delay, repeated tenant-isolation denials, missing evaluation evidence, abnormal release failures, or a breached service objective. A GPU utilization alert without a user impact hypothesis often creates noise. Every page should have an owner and a response that the on-call engineer can execute.
Put economics alongside reliability
Compute cost is easy to see; the cost of failed experiments, repeated preprocessing, idle reservations, storage growth, and human investigation is easier to miss. Track cost by workload class and tenant internally, under the same access discipline as other operational data.
For batch generation, I prefer cost per useful, approved workload over cost per generated row. Row counts reward volume even when outputs fail evaluation. Include failed and retried runs in the numerator. Define the denominator with product owners so it reflects a useful outcome rather than a convenient technical event.
Use smaller representative development datasets, checkpointing where supported, and interruptible capacity for workloads that can tolerate interruption. Cache only when the cache key captures the relevant dataset, code, policy, and configuration versions. A cheap reuse of stale evaluation evidence is not an optimization.
Publish resource classes with predictable defaults and limits. Show an estimate before launching unusually expensive work, then compare estimates with actuals. Budget controls should prevent surprise without forcing every routine run through a manual approval queue. The goal is understandable trade-offs, not maximum utilization at any cost.
Deliver a narrow path, then widen it deliberately
Google Cloud’s MLOps architecture guidance describes increasing automation across training, validation, and delivery.8 I would apply that progression to a synthetic data platform by making one complete workflow reliable before generalizing it into a large internal framework.
Begin with one dataset class, one supported model family, one evaluation protocol, and one consumer journey. Make authorization, lineage, failure recovery, and release evidence part of that first path. A thin end-to-end platform teaches more than a broad collection of disconnected tools.
Then improve the bottleneck that users can demonstrate. If researchers wait days for environments, standardize environment provisioning. If consumers cannot interpret evaluation reports, improve the report and contract. If approvals stall, make the evidence and decision ownership clear. More orchestration is not the default answer to every delay.
Before broadening adoption, run failure exercises: interrupt a worker, exhaust a quota, deny a storage permission, submit malformed relational data, withdraw a release, and restore required metadata from backup. Measure recovery against the objectives you promised. A backup that has never been restored is an assumption, not an established recovery capability.
The platform is ready to scale when the supported path remains understandable under pressure. Engineers should know how to start work, why it failed, what the output means, and what evidence makes it suitable for release. In synthetic data, trust is not an additional feature attached to a fast pipeline. It is part of the product the pipeline has to deliver.
References and further reading
- D. Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS, 2015. A foundational account of system-level maintenance risks around ML models.
- Kubernetes documentation, Multi-tenancy. Isolation models, namespace controls, and the limits of shared clusters.
- Google, Handling Overload, Site Reliability Engineering. Admission control, load shedding, and preventing overload from spreading.
- Theresa Stadler, Bristena Oprisanu, and Carmela Troncoso, Synthetic Data — Anonymisation Groundhog Day, USENIX Security, 2022. Empirical analysis of synthetic data privacy and utility trade-offs; findings should be interpreted within the evaluated methods and threat models.
- NIST, Guidelines for Evaluating Differential Privacy Guarantees, SP 800-226, 2025. Guidance on interpreting privacy guarantees and identifying implementation hazards.
- Eric Breck et al., The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction, 2017. Testing and monitoring recommendations across the ML production system.
- OpenTelemetry documentation, Semantic Conventions. Common conventions for interoperable telemetry; organizations must still define their own data-minimization policy.
- Google Cloud, MLOps: Continuous delivery and automation pipelines in machine learning. A reference for staged ML workflow automation, rather than a requirement to adopt a particular cloud.