· 17 min read
Operating Synthetic Data Platforms with Less Compute and Less Toil
A practical guide to simpler deployments, recoverable upgrades, efficient scheduling, and lower cost per approved synthetic dataset—without weakening privacy or release checks.
By Shamiul Shifat
Lead AI Platform Engineer · Betterdata AI
In this article
The most expensive synthetic data platform is not necessarily the one with the largest GPU bill. It may be the one that requires an engineer to repair every installation, rerun ambiguous failures, interpret undocumented configuration, and manually assemble the evidence needed to release a dataset.
As a Lead AI Platform Engineer at Betterdata AI, I think about efficiency as an end-to-end property: how much compute, elapsed time, and human attention does it take to deliver a useful, authorized dataset? An optimization that reduces training time but doubles operational work is not automatically a good trade.
The approach below is the operating model I recommend for synthetic data systems. The configurations, capacity calculations, and costs are illustrative; they do not describe Betterdata’s internal systems, performance, or commercial pricing. The broader architecture is covered in Building AI Platforms at Scale. Here, the focus is making that architecture inexpensive to deploy, predictable to operate, and easier to recover.
Define efficiency around an approved outcome
Start with three measures: time to the first approved dataset, cost per approved workload, and operator intervention per release. These make deployment friction, wasted execution, and recurring manual work visible alongside infrastructure consumption.
Define the unit carefully. A tiny test fixture and a large relational dataset are not equivalent outputs. Group workloads by meaningful classes—data volume, model family, evaluation requirements, or service tier—and compare like with like. Do not improve the dashboard by dividing the bill by more generated rows.
Minimum cost is a constrained objective: required utility, privacy protection, latency, and recovery capability must still hold. There is no universally cheapest architecture or lowest-compute model that satisfies every workload. Choose between candidates using representative benchmarks that you measure under those constraints, rather than transferring a performance claim from an unrelated dataset or environment.
The FinOps Framework’s unit-economics capability connects technology spending to business-relevant units.1 For synthetic data, I would extend that view to the work needed to produce an approved artifact. Include failed runs, idle reservations, storage, evaluation, and support effort. Report machine cost and fully loaded operating cost separately so changes in labor allocation do not obscure actual infrastructure consumption.
Google’s SRE definition of toil emphasizes work that is manual, repetitive, automatable, tactical, and grows with service scale.2 Installation recovery, routine credential repair, and repeated manual job cleanup often fit that pattern. The objective is not to eliminate human judgment. It is to preserve that judgment for decisions that deserve it.
Choose the smallest deployment model that meets the requirements
An orchestration platform earns its complexity when it solves a demonstrated coordination problem. It should not be the admission ticket for generating a dataset.
| Operating situation | Reasonable starting point | Evidence that justifies moving further |
|---|---|---|
| One trusted operator, infrequent batch runs, accepted maintenance windows | A versioned command-line runner or single service on a managed host, with durable artifacts and backups | Concurrent users, queue contention, or recovery requirements exceed what one host can support |
| Several trusted teams, recurring work, modest concurrency | API plus durable queue, worker pool, and a supported state store | Independent scaling, deployment coordination, or isolation needs become difficult to manage |
| Mixed workload classes, multiple tenants, strong availability requirements | An orchestrator such as Kubernetes, if the team can operate it or use a suitable managed service | Scheduling, placement, autoscaling, and lifecycle controls offset the platform’s operating cost |
| Restricted or disconnected environment | A supported offline bundle with an explicit infrastructure contract | Requirements determine the scheduler; lack of internet alone does not imply Kubernetes |
These are starting points, not security certifications. A single host may be unsuitable for hostile tenants or strict availability objectives. A Kubernetes namespace does not independently establish a strong tenant boundary. Conversely, placing a small trusted batch workload into a cluster does not make its data contract, backups, or release process reliable.
Preserve a stable job interface across deployment modes. Research code should receive a validated specification, read authorized inputs, write versioned artifacts, and return structured status. It should not need to know whether a worker was started by a service manager or a cluster scheduler. That boundary makes later migration much less invasive.
Ship a complete release bundle
A deployment is not reproducible if the installer downloads an unspecified model revision, resolves the newest compatible dependency, or discovers a missing package halfway through setup. Build the release bundle before it reaches the target environment.
I would include pinned images or binaries, model and tokenizer artifacts where licensed for redistribution, configuration schemas, database migration files, a bill of materials, a compatibility matrix, and a manifest of cryptographic digests. Record build provenance and verify that the artifacts match the expected producer and release identity; a digest detects a mismatch only when the expected digest comes from a trusted source. SLSA’s verification guidance explains why provenance must be checked against expectations, rather than merely attached to an artifact.3
Pin the application and its transitive runtime dependencies. Document the host assumptions that remain outside the bundle: architecture, operating system support, accelerator driver compatibility, available storage, and required services. A pinned container does not make a host driver compatible or guarantee deterministic numerical results across hardware.
Use offline-compatible defaults. Startup should not require an external model registry, license discovery endpoint, package repository, or telemetry collector unless that dependency is an explicit supported requirement. Disable accidental update checks and fail clearly when a configured dependency is unavailable. For stronger disconnected-environment controls, see Securing Air-Gapped Synthetic Data Platforms.
Make installation preflighted and repeatable
The installer should diagnose an unsuitable environment before pulling large files or changing state. Check manifest integrity, supported configuration, driver and runtime compatibility, free disk space, filesystem permissions, service reachability, clock assumptions, and the intended storage location. Show the effective configuration with secrets redacted, including defaults the operator did not explicitly set.
Run a small smoke job against bundled, non-sensitive fixture data. It should exercise artifact loading, worker execution, storage access, and the release-report path. This catches integration failures that a collection of individual health checks can miss.
The following is conceptual installer logic, not a command to run unchanged:
verify_bundle_against_trusted_release_identity()
validate_configuration_and_host_requirements()
inspect_existing_installation_and_state_version()
show_redacted_change_plan()
if existing_release_matches_requested_release:
verify_services_and_report_current_state()
else:
acquire_exclusive_installation_lock()
recheck_state_after_lock()
checkpoint_recoverable_state()
apply_supported_migrations_with_a_durable_journal()
activate_pinned_release()
run_fixture_smoke_test()
record_completed_installation()
Repeatability means a rerun converges on a known state without duplicating accounts, regenerating keys, or destroying data. It does not mean rerunning destructive steps until they stop failing. Journal the transitions and distinguish “not started,” “in progress,” and “completed.” If a non-transactional migration is interrupted, resume or recover according to its documented procedure.
Do not hide uncertainty behind an installer that always exits successfully. Report the failing stage, what changed, what remains safe to retry, and where the diagnostic bundle is stored. That one error contract can remove hours of support work.
- Verify the bundle
Check trusted release identity, digests, signatures or provenance, and the compatibility contract.
- Preflight the environment
Validate configuration, resources, permissions, state version, and required local dependencies.
- Apply a journaled change
Lock the installation, checkpoint recoverable state, and execute supported migrations once.
- Prove and record readiness
Run a non-sensitive fixture workflow and retain the active version plus a clear recovery procedure.
Design upgrades before the first customer installation
The first install is usually the easiest lifecycle event. Upgrades must account for existing datasets, partially completed jobs, old workers, new API contracts, and state that an older release cannot read.
Use an expand-and-contract migration strategy where practical. First introduce a compatible schema change. Deploy code that can operate during the transition. Backfill or transform existing state in bounded, observable batches. Remove the old representation only after the compatibility window closes and recovery requirements are satisfied.
Define which versions can coexist. If old workers cannot interpret a new job specification, keep producing the old schema until they drain, route jobs to compatible workers, or explicitly reject the unsupported combination. Version serialized messages and artifact manifests as well as the database.
A binary rollback does not undo a database migration or reverse an output already delivered to a consumer. Document whether recovery means switching application versions, restoring state, rolling forward with a repair, or withdrawing an artifact. Test that procedure using a representative backup. Specify the recovery point objective so the operator knows what work could be lost.
For long-running jobs, decide whether upgrades drain, pause at checkpoints, or terminate work. Enforce an upper bound on draining so one stuck job cannot block maintenance indefinitely. A canary release should first prove compatibility and operational behavior on controlled workloads; it does not authorize publishing production outputs that have not passed their release checks.
Measure the bottleneck before buying capacity
Split the workload into preprocessing, training, generation, evaluation, and artifact transfer. Measure elapsed time, CPU time, accelerator activity, memory pressure, storage throughput, and queue wait for each stage. Whole-job GPU utilization is a poor diagnostic when the GPU spends half the run waiting for CPU preprocessing or remote storage.
For a simple stage with equivalent workers, approximate service capacity as:
stage capacity = worker count / average service time
pipeline capacity <= the smallest stage capacity
This is a capacity sketch, not a queueing model. It assumes steady input, available data, independent stages, and sufficient buffering. Variance, shared resources, failures, startup delay, and synchronized bursts reduce usable capacity. Near saturation, queueing delay can grow sharply even when average arrival rate is below nominal capacity.
Consider a hypothetical workload:
| Stage | Average time per attempt | Parallel workers | Nominal capacity |
|---|---|---|---|
| Preprocessing | 10 minutes | 2 CPU workers | 12 attempts/hour |
| Training and generation | 20 minutes | 2 GPU workers | 6 attempts/hour |
| Evaluation | 30 minutes | 1 CPU worker | 2 attempts/hour |
Evaluation limits the pipeline to at most two completed attempts per hour. If 80% of fully evaluated attempts produce an approved output, the corresponding upper bound is 1.6 approved datasets per hour. Adding two more GPUs does not change that bound. Adding a second independent evaluation worker raises evaluation capacity to four attempts per hour and the illustrative approved-output bound to 3.2, provided CPU, memory, and storage are available.
The no-queue latency of an individual attempt is still 60 minutes in this example. More parallel workers improve throughput; they do not necessarily shorten a single job. Measure both because different users care about different promises.
Match resources to the actual stage
Keep CPU-only work off expensive accelerator reservations when the transfer and orchestration overhead permit it. Parse, validate, and prepare data before acquiring a GPU. Release the accelerator when generation finishes if the evaluator does not use it. For very short runs, however, loading the model and transferring data can outweigh the savings from splitting every stage.
Profile representative workloads before changing batch size, precision, compilation, or model implementation. Compare peak memory, elapsed time, stability, and the complete utility and privacy evaluation. Faster execution is not a valid optimization if it changes the mechanism behind a claimed privacy guarantee or causes unacceptable task degradation.
Include a simple CPU-based baseline when it can represent the data and required constraints. Compare it with accelerator-based candidates using the same evaluation protocol, including rare categories and the largest supported input shape. A GPU should earn its reservation through measured benefit. Likewise, a successful tiny benchmark does not establish the memory ceiling for a production workload.
On Kubernetes, set resource requests from measured requirements and choose limits with an understanding of their enforcement. CPU limits can throttle execution; memory limits can lead to termination when a container exceeds its allocation. The official resource-management documentation describes these semantics.4 Oversized requests strand capacity, while undersized requests encourage unsafe packing and noisy-neighbor behavior.
Start with a small number of supported resource classes. Let users choose a class with a clear memory ceiling and expected workload envelope. Add exceptions when measurements justify them. An unrestricted catalogue of arbitrary CPU, memory, and GPU combinations is expensive to support and difficult to schedule efficiently.
Stop invalid work early without weakening evaluation
Run cheap, deterministic checks before expensive model execution: authorization, schema compatibility, column counts, input presence, relational integrity where feasible, supported size limits, and configuration validity. Return all actionable validation errors together rather than making a user resubmit for each missing field.
Evaluation can also be staged. Structural failures can stop a run before costly downstream benchmarks. Small diagnostic samples can help identify a broken pipeline. But a sample check must not silently replace the full release protocol, particularly for rare events, subgroup utility, or privacy assessment.
Make the distinction explicit in state and reporting: “diagnostic passed” is not “release approved.” Retain which checks ran, against which artifact, under which policy. If evaluation is interrupted, resume a valid stage or rerun the required check; do not infer success from a partially written report.
The same discipline applies to caching. Cache public model artifacts by trusted digest. Cache tenant data or derived features only within authorized boundaries, with encryption, access controls, retention, and deletion behavior. A cache key should include source version, transformation version, relevant parameters, and privacy scope—not merely a filename.
Evaluation reuse requires an exact match of the evaluated artifact and all relevant evaluator, policy, and reference-data versions. Invalidate results when those inputs change. Do not return a cached approval for an artifact whose release permission has been revoked. A cache hit improves performance; it does not create authorization.
Make retries safe and charge them honestly
Failures are inevitable; duplicate expensive work is often avoidable. AWS’s Builders’ Library recommends bounded retries with backoff and jitter and explains how idempotent APIs reduce the ambiguity of retrying operations.56
Record a tenant-scoped idempotency key and request fingerprint before scheduling a logical operation. A repeated request with the same key should return the existing operation; the same key with a different payload should fail clearly. Retain the record for a documented interval consistent with client retry behavior. Inside the job, publish artifacts through an idempotent transition so duplicate workers cannot release two conflicting outputs.
Classify failure before retrying. Temporary storage unavailability may justify a bounded retry. Invalid input, insufficient permission, an incompatible model, or a failed release criterion requires a different response. Kubernetes Jobs can restart work or create replacement Pods, and the application must account for repeated execution.7 A scheduler’s retry policy is not an exactly-once guarantee.
For differentially private processing, a retry is not automatically free. Repeating private-data access, training, or tuning can affect composition, depending on the mechanism and what is retained or disclosed. NIST’s guidance on evaluating differential privacy guarantees provides the relevant framework.8 Reserve and reconcile privacy accounting through a durable ledger; do not decrement a counter and then forget the run when a worker dies.
Generating additional samples solely from an already differentially private model is post-processing. Retraining against private data is a different operation. Distinguish the two in both privacy accounting and billing. If interruption leaves the privacy consequences uncertain, fail conservatively and investigate rather than refunding budget by assumption.
Bound demand and decide when warmth is worth paying for
A queue is a buffer, not infinite capacity. Set limits on pending work, input sizes, concurrent execution, and aggregate resources per tenant. Reject or defer excess demand with an explicit reason and retry guidance. Google SRE’s overload guidance explains why admission control and load shedding are necessary to prevent cascading failure.9
Use weighted fair sharing or another documented scheduling policy so a large tenant cannot monopolize the fleet. Separate interactive previews from large batch runs where the product requires different latency. Bound the resources reserved for priority work and use aging or another anti-starvation mechanism for lower-priority jobs. Otherwise, “best effort” can become “never runs.”
Scale workers using a signal that reflects useful backlog: queue age, estimated remaining work, and available capacity. CPU utilization alone may miss a queue waiting for GPU placement. Kubernetes HPA supports custom and external metrics, but signal quality, polling delays, stabilization, and resource startup still constrain the response.10
Compare warm pools with scale-to-zero by workload class. A warm worker reduces image loading, model initialization, and driver warm-up delay, but incurs idle cost. Scale-to-zero saves that reservation while increasing cold-start latency and dependence on capacity availability. Keep a small warm pool only where the service objective or workload frequency justifies it; infrequent batch workloads may tolerate colder starts.
- Measure a workload class
Capture queue time, stage time, resource use, retries, approval yield, and operator minutes.
- Find the real constraint
Identify whether progress waits on compute, data movement, evaluation, deployment, or human diagnosis.
- Change one useful boundary
Right-size a stage, eliminate invalid work, make a retry safe, or automate a repeated recovery step.
- Recheck the outcome
Compare approved-output cost and latency while preserving required validity, utility, privacy, and release evidence.
Work through the economics, including human time
Here is a deliberately simplified, hypothetical accounting example. Assume every billed attempt costs an average of $1: 20 GPU minutes at $2.40/hour equals $0.80, 40 aggregate CPU-worker minutes at $0.12/hour equals $0.08, and per-attempt transfer and temporary storage add $0.12. These are invented rates for arithmetic, not provider quotes. Actual failed attempts may consume different amounts and should be metered accordingly.
| Cost for 100 approved datasets | Initial operation | After targeted improvements |
|---|---|---|
| Billed attempts at $1 average | 125 attempts: $125 | 115 attempts: $115 |
| Idle reserved capacity | $20 | $10 |
| Allocated shared infrastructure, retained storage, and logs | $30 | $30 |
| Operator effort at $75/hour | 6 hours: $450 | 3 hours: $225 |
| Total modeled operating cost | $625 | $380 |
| Cost per approved dataset | $6.25 | $3.80 |
The improved column assumes preflight and recovery changes reduce wasted billed attempts, while better diagnostics reduce operator time. Acceptance criteria are unchanged. Machine cost falls from $175 to $155; most of the modeled saving comes from less intervention. Include license fees, support contracts, and other material costs in a real allocation; this example assumes no additional charges beyond the listed categories.
These are modeled operating savings, not automatically cash savings. Freed employee time becomes capacity for other work unless staffing expense actually changes. Likewise, reducing active GPU minutes saves money only when the billing or reservation model allows that capacity to be released or productively reused.
Keep implementation cost visible. If an automation saves three operator hours per month but takes months to build and maintain, its payback may be poor. Prioritize frequent, high-friction tasks with bounded implementation effort before building a general automation framework.
Give operators a small, useful control surface
Begin with queue age by workload class, running and failed jobs, stage durations, approval yield, resource saturation, retry counts, storage growth, and cost allocation. Add the active release version and migration state to the same operational view. These signals explain more than a wall of infrastructure charts without a user journey.
Every failed job should expose a stable error category, current stage, correlation identifier, and safe next action. Offer a redacted diagnostic bundle that contains configuration versions and system facts without raw datasets, credentials, or sensitive model inputs. Test redaction on failure paths, where libraries are most likely to include unexpected details.
Keep a short set of practiced runbooks: installation failure, stuck queue, exhausted storage, failed upgrade, worker loss, and recovery from backup. Give each alert an owner and a specific response. A dashboard is helpful; a recovery procedure that an engineer can execute under pressure is more valuable.
A practical 30/60/90-day sequence
Days 1–30: establish the baseline. Choose representative workload classes. Measure stage time, cost, failure reasons, and operator effort. Inventory deployment dependencies and pin the release bundle. Implement deterministic preflight and a fixture smoke job. Agree on the approved-output definition and preserve the existing release requirements.
Days 31–60: remove the largest repeated waste. Fix the most frequent invalid submission and the most expensive unsafe retry. Separate CPU and GPU stages where measurement supports it. Introduce bounded queues, tenant quotas, and resource classes. Automate the highest-frequency recovery task, then compare the same workload classes against the baseline.
Days 61–90: prove lifecycle reliability. Rehearse an upgrade, interrupted migration, rollback or roll-forward recovery, and backup restoration. Test cold starts and capacity exhaustion. Review cache invalidation and privacy accounting during retries. Expand adoption only when operators can explain the active state and recover the supported workflow without improvised repair.
This sequence is a planning template, not a promised delivery schedule. The right pace depends on the starting system and risk. The engineering test remains consistent: can more people obtain useful, authorized synthetic data with fewer resources and fewer interventions, while retaining the evidence that makes the output trustworthy? That is the efficiency worth pursuing.
For the developer-facing side of that operating model, continue with Designing for Developer Experience.
References and further reading
- FinOps Foundation, Unit Economics. Connecting technology costs to meaningful units of value; the approved-dataset accounting model here is an editorial application.
- Google, Eliminating Toil, Site Reliability Engineering. Characteristics of toil and the need to preserve engineering capacity.
- SLSA, Verifying artifacts, specification v1.0. Verification of provenance against expected artifact and producer properties; this is a versioned reference, not a claim about the latest specification.
- Kubernetes documentation, Resource Management for Pods and Containers. Resource requests, limits, and their scheduling and enforcement behavior.
- AWS Builders’ Library, Timeouts, retries, and backoff with jitter. Guidance for limiting retry amplification and responding to transient failures.
- AWS Builders’ Library, Making retries safe with idempotent APIs. Idempotency tokens, request intent, and safe handling of repeated operations.
- Kubernetes documentation, Jobs. Job execution, retries, replacement Pods, and application responsibilities.
- NIST, Guidelines for Evaluating Differential Privacy Guarantees, SP 800-226, 2025. Interpreting guarantees, composition, and implementation hazards; the retry-ledger design is an engineering recommendation.
- Google, Handling Overload, Site Reliability Engineering. Admission control and protecting a system under excess demand.
- Kubernetes documentation, Horizontal Pod Autoscaling. Metric-driven scaling and controller behavior; use a supported queue-worker scaling design rather than assuming HPA alone provides scale-to-zero.