← Back to writing

· · 12 min read

Designing for Developer Experience

A practical approach to AI platform design: shorten feedback loops, make privacy decisions legible, and help engineers deliver with confidence.

By Shamiul Shifat
Lead AI Platform Engineer · Betterdata AI

In this article

A developer submits a synthetic-data generation job. The API returns 202 Accepted, the dashboard shows a spinner, and twenty minutes later the job fails with “validation error.” The platform has accepted work, consumed compute, and produced no usable explanation. Every service might be technically available, yet the developer’s experience is a failure.

That hypothetical workflow captures how I think about platform quality. As a Lead AI Platform Engineer at Betterdata AI, I approach developer experience through the responsibilities of working with synthetic data and privacy technology: sensitive inputs, expensive computation, asynchronous workflows, and outputs whose usefulness and disclosure risk require careful evaluation. A convenient interface matters. A trustworthy journey matters more.

The design recommendations below are my professional perspective. The examples are illustrative, rather than descriptions of Betterdata’s internal architecture or measured results.

Start with the developer’s uncertainty

In DevEx: What Actually Drives Productivity, Abi Noda, Margaret-Anne Storey, Nicole Forsgren, and Michaela Greiler describe three dimensions of developer experience: feedback loops, cognitive load, and flow state. Their framework moves the discussion beyond individual tools toward the conditions in which people do their work. 1

I use those dimensions as design questions:

  • Feedback loops: How quickly can someone discover whether their next step is valid, and whether their previous step worked?
  • Cognitive load: How many concepts, dependencies, and exceptions must they keep in working memory?
  • Flow state: Can they complete a meaningful unit of work without repeated interruptions, permission chases, and context switches?

A synthetic-data platform makes these questions concrete. An engineer may need to select a source, understand permitted uses, define constraints, choose a generator, configure evaluation, inspect failures, and obtain permission to release an artifact. These are different decisions with different owners. Putting all of them on one attractive screen does not make them one simple task.

My first platform discovery exercise would be to follow a developer through one real journey. I would record when they leave the product to ask someone a question, copy an identifier into another system, search for an undocumented prerequisite, or wait for a response. Those interruptions reveal where the platform’s conceptual model ends and the developer’s improvisation begins.

Treat the platform as a product with a defined customer

The CNCF Platforms White Paper describes platforms in terms of the capabilities and experiences they provide to their users, including the importance of treating an internal platform as a product. That framing makes adoption and usefulness part of the platform team’s responsibility. 2

“Our customer is engineering” is too broad to guide a useful roadmap. Consider three users:

UserImmediate jobWhat confidence requires
ML researcherCompare approaches on a controlled experimentReproducible inputs, visible configuration, and interpretable evaluation
Application engineerObtain suitable data for a test environmentStable contracts, representative scenarios, and clear usage restrictions
Privacy or security reviewerDecide whether a proposed release meets its requirementsTraceable evidence, explicit assumptions, and an accountable release decision

The shared platform should connect these journeys without pretending their needs are identical. A researcher may need configuration freedom; an application engineer may prefer a small set of supported presets. A reviewer should see evidence without requiring broad access to the original sensitive records.

I would begin with one frequent, valuable journey and make it complete. For example: request a test dataset from an approved source, generate it, evaluate it, and publish it to an authorized development environment. Completeness means handling invalid inputs, cancellation, denied access, evaluation failure, and cleanup—not just the successful demonstration.

Design a supported path that carries the hard decisions

A golden path is useful when it packages sound decisions into a maintained workflow. It becomes a liability when it hides decisions that users still need to understand.

For synthetic data, a supported path should make the purpose of the output explicit before work begins. A dataset for UI testing has different utility requirements from one intended for model development. An internal evaluation artifact has a different release context from a downloadable external sample.

Field guide A developer journey with evidence at every step An illustrative workflow. Each stage produces something the next stage can inspect; generation success alone does not authorize release.
  1. Declare intent

    Identify the intended task, approved source, destination, and accountable owner.

  2. Validate early

    Check permissions, schema, constraints, quotas, and configuration before expensive work.

  3. Run visibly

    Expose lifecycle state, progress, cancellation, and actionable failure details.

  4. Evaluate separately

    Produce task-specific utility results and privacy evidence with their assumptions.

  5. Release deliberately

    Apply the relevant release decision, access scope, expiry, and audit record.

The supported path needs an escape route. Advanced users should be able to request a capability the default does not provide. That exception should have an owner, an explicit support boundary, and a route back into the standard product when demand becomes recurring.

Otherwise, the platform accumulates invisible forks: notebooks with private credentials, unreviewed scripts, and informal operating instructions. The architectural issue is not that developers dislike standards. It is that the standard stopped short of their actual work.

Make asynchronous APIs understandable

Long-running AI workloads need a contract that survives interruptions. I would expose a job resource with a stable identifier, a documented state model, timestamps, an authorized way to inspect results, and a machine-readable failure structure.

A possible state model is queued → running → evaluating → awaiting_release → released, with explicit terminal outcomes such as failed, cancelled, and rejected. These names are an example, not a universal standard. Their value lies in eliminating ambiguous states. In particular, “completed” should not blur the difference between a finished computation and an artifact approved for its intended destination.

Submission should support idempotency with documented scope and lifetime. If a client times out after submission, retrying the same operation should not silently create another expensive job. The service must define what happens when the same key is supplied with a different payload, when the key expires, and when an earlier attempt is still executing.

AWS’s Builders’ Library discusses why retries can amplify load and why side effects require care. Its guidance on timeouts, backoff, and jitter is especially relevant to clients polling or resubmitting work during a platform incident. 3

I would document a retry policy alongside the API: which failures are retryable, how to honor server guidance, where retries stop, and how callers can discover the outcome of an uncertain submission. Cancellation also needs a precise meaning. “Cancellation requested” is different from “all compute stopped and temporary artifacts removed.”

An illustrative validation response might look like this:

{
  "code": "CONSTRAINT_REFERENCE_INVALID",
  "stage": "preflight",
  "message": "The configured constraint refers to an unavailable column.",
  "requestId": "req_example_001",
  "retryable": false,
  "resolution": "Check the constraint against the selected schema version."
}

The response identifies the problem and next action without echoing a sensitive row. Detailed diagnostics can live behind appropriate authorization. A raw stack trace or a sample of the offending record is not a substitute for a designed error contract.

Move useful feedback before expensive computation

Feedback speed is not simply job duration. It includes how soon the developer discovers a mistake and how much work must be repeated afterward.

I would split feedback into three layers. Local feedback validates configuration structure and demonstrates the API against deliberately fabricated fixtures. Preflight feedback checks authorized metadata, compatible constraints, budget limits, and required permissions. Execution feedback reports actual progress, resource problems, and evaluation outcomes.

Each layer has limits. A local mock cannot demonstrate privacy properties. A small sample can reveal a pipeline error without proving that full-scale output will preserve rare patterns. Preflight validation cannot guarantee that runtime dependencies will remain available. The interface should communicate those boundaries instead of displaying a single reassuring green tick.

The practical opportunity is to fail cheap mistakes early. Discovering an invalid column name should not require a GPU allocation. Discovering missing output permissions should not require finishing a generation run. The platform can make both decisions before the developer pays the cost of waiting.

Progress reporting deserves the same discipline. If the system cannot estimate time remaining reliably, it should show a factual stage, queue position where meaningful, and last progress timestamp. A fabricated percentage creates confidence the system has not earned.

Put privacy into the contract and the defaults

In privacy technology, developer experience includes understanding what the system does and does not establish. “Synthetic” describes how data was produced; it does not, by itself, establish a formal privacy guarantee. NIST SP 800-226 provides guidance for evaluating differential privacy guarantees and their implementation assumptions. 4

I would design the product around several distinctions:

Utility evidence is separate from privacy evidence. A dataset can preserve useful statistical patterns and still require further disclosure-risk assessment. A privacy assessment can support a particular release decision without establishing that every downstream model will perform well.

Empirical tests are separate from formal guarantees. A low measured attack success rate is evidence about that evaluation setup. It is not automatically differential privacy. If a workflow claims differential privacy, its documentation needs the mechanism, protected unit, parameters, accounting assumptions, and relevant composition context—not merely a “private” badge.

Permission to generate is separate from permission to distribute. A developer authorized to run an experiment is not necessarily authorized to publish its outputs externally. The release workflow should preserve this distinction even when the same person performs both tasks.

These distinctions should lead to practical defaults: restricted output access, explicit destinations, retention controls, tenant-scoped artifacts, and logs that avoid raw sensitive values. Default sample datasets and documentation examples should use deliberately fabricated records. An example copied from a production incident can become an accidental distribution channel.

The goal is to make the responsible workflow easy to follow and understandable. A hidden security rule that produces a generic 403 after a long job is poor developer experience and poor communication of policy.

Build documentation around decisions and recovery

A reference manual explains what each field means. A usable platform also explains which fields matter for the task in front of the developer.

I would maintain four connected documentation layers: a short first-success guide, task-oriented recipes, a complete reference, and operational recovery guidance. A synthetic-data recipe should explain its intended use, prerequisites, approximate resource considerations, evaluation expectations, and what the example does not establish.

Every example should have a version context. If an SDK example works only with a particular API version or feature flag, that dependency belongs beside the example. Successful documentation is a maintained interface; abandoned snippets are a source of production defects.

Recovery guidance needs more than “contact support.” For each common failure, explain what the developer can inspect, what can be safely retried, which artifacts remain, and what evidence to include in a support request. Correlation identifiers are useful precisely because a person can share them without copying sensitive payloads into a chat channel.

I would also document ownership. Developers should be able to determine whether a problem belongs to their configuration, the platform runtime, a source system, or a policy decision. The interface should help them route the problem without asking them to reverse-engineer the organization’s structure.

Measure the journey without ranking people

The SPACE framework, developed by Nicole Forsgren and colleagues, argues that developer productivity cannot be reduced to one metric. Its dimensions include satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. 5

That is an important constraint on platform measurement. Counting deployments, generated datasets, or API calls can describe activity. It does not show whether developers obtained a useful result, understood the evidence, or spent the afternoon recovering from an unclear error.

My initial measurement set would be deliberately small:

MeasureQuestion it helps answerInterpretation risk
Time to first useful artifactCan a new user complete a meaningful task?Different tasks and approval requirements are not directly comparable
Time from actionable failure to recoveryDoes the product help users fix problems?Some failures need external owners or policy decisions
Support requests per completed workflowWhere does self-service break down?A drop may mean abandonment rather than improvement
Completion and abandonment by stageWhere do users lose progress?Instrumentation must avoid sensitive payloads and misleading session assumptions
Short experience surveys and interviewsWhat uncertainty does telemetry miss?Small samples and response bias require careful interpretation

I would segment by journey and experience level, examine distributions, and pair telemetry with interviews. A median can hide severe delays for new users or a particular workload class. These measures should improve the system rather than become individual engineer rankings.

Service health still matters. Google’s SRE guidance recommends monitoring latency, traffic, errors, and saturation. 6 I would connect those operational signals to user journeys: queue wait, submission failures, evaluation delays, and inaccessible results. A healthy submission endpoint does not prove that developers can obtain usable output.

Design the improvement loop as carefully as the interface

Field guide A platform improvement loop A qualitative operating model, not a claimed productivity result. Start with a real workflow and use evidence to decide whether a change helped.
  1. Observe

    Watch users complete one important journey; record uncertainty and interruptions.

  2. Prioritize

    Choose a recurring source of friction with meaningful impact and an accountable owner.

  3. Change

    Improve the contract, default, feedback, or documentation at the point of failure.

  4. Evaluate

    Compare journey outcomes and user feedback; investigate unintended effects.

For a first month, I would choose one journey, interview several users with different levels of familiarity, and map the support tickets against its stages. I would establish a baseline before promising a numerical improvement. The most valuable first change may be a precise preflight error or a reproducible example, rather than a new portal.

Next, I would close a small number of repeated gaps and watch users attempt the journey again. Did they reach a useful outcome? Did they understand the release constraints? Could they recover independently when something failed? These questions keep improvement tied to actual work.

Finally, I would make maintenance explicit. Supported SDK versions, ownership of templates, deprecation windows, and documentation reviews need responsible people. Without that operating commitment, yesterday’s golden path becomes tomorrow’s migration burden.

For an AI platform, developer experience is the confidence to act with an accurate understanding of consequences. The platform earns that confidence by making states visible, feedback actionable, privacy claims precise, and responsibilities clear. That is the standard I would use to judge whether a platform is ready to become part of someone else’s daily work.

References and further reading

  1. Abi Noda, Margaret-Anne Storey, Nicole Forsgren, and Michaela Greiler. DevEx: What Actually Drives Productivity. ACM Queue, 2023. The source for feedback loops, cognitive load, and flow state.
  1. CNCF TAG App Delivery. Platforms White Paper. Practitioner guidance on platform capabilities, users, and treating a platform as a product.
  1. Marc Brooker. Timeouts, retries, and backoff with jitter. Amazon Builders’ Library. Recommendations on managing retry behavior and distributed-system failure.
  1. NIST. Guidelines for Evaluating Differential Privacy Guarantees, SP 800-226. 2025. Guidance for evaluating differential privacy claims and the assumptions behind them.
  1. Nicole Forsgren and colleagues. The SPACE of Developer Productivity. ACM Queue, 2021. A multidimensional framework for developer productivity.
  1. Google. Monitoring Distributed Systems. Site Reliability Engineering. The source for the four golden signals of monitoring.

Department of small adventures

Pip’s pocket passport

Six little souvenirs. One very well-travelled duck.

Kept in this browser, just for you. Visit each section and collect its stamp here—or catch one from Pip.