reproducible AI agent evaluations

By Platinum Blog

Quick answer

For reproducible AI agent evaluations, use a recorded environment baseline, hold task inputs and scoring rules steady, and retain evidence from repeated runs. Treat environment reset as one control, not a promise of identical agent decisions: Platinum documents disk-based forks without memory or processes, while OpenAI's historical reproducibility example explicitly says model determinism is not guaranteed. (Platinum snapshots; OpenAI Cookbook)

Platinum is a sandbox cloud for AI agents whose documentation describes snapshots, clones, and forks for preparing and branching sandbox environments. Platinum Blog is a first-party publication; this guide interprets vendor documentation and proposes an evaluation protocol, rather than reporting independent product testing or measured benchmark results. Sources were checked September 17, 2026. (Platinum overview; snapshot documentation)

Environment state versus model behavior

For this proposed protocol, an evaluation baseline is the recorded starting configuration against which agent variants are compared: code, dependencies, task fixtures, prompts, and scoring rules. An evaluation run is one attempt at a specified task under a recorded variant. Keep baseline comparability and output repeatability as separate questions; OpenAI's historical example warns that even matching seed, request parameters, and backend fingerprint can leave variation in model responses. (OpenAI Cookbook)

Layer	Documented boundary or scope	Recommended evaluation control
Sandbox disk	Platinum describes a snapshot as a full disk copy at one point in time. Source	Record the snapshot identity and verify the intended baseline files after boot.
Memory and processes	A Platinum fork cold-boots from disk without memory or processes. Source	Use a recorded startup procedure; do not rely on a live process continuing in the child.
Attached volumes	Platinum excludes attached-volume contents from sandbox snapshots and backups. Source	Version fixture data separately and record the attachment policy.
External services	The proposed baseline includes external dependencies as separate inputs, not as verified snapshot contents.	Pin or replay fixtures where appropriate; otherwise record live responses and label that dependency uncontrolled.
Model configuration	OpenAI's historical seed example offers best-effort consistency, not guaranteed determinism. Source	Record the model identifier, request settings, prompts, and response metadata available for the selected API.
Scoring	Scoring rules are a separate control in this proposed protocol.	Version the rubric, evaluator, and failure-handling rules before comparing variants.

What Platinum documents

Platinum defines a snapshot as a full copy of a sandbox disk at one point in time and says the sandbox must be running when a snapshot is taken. The documented clone operation boots a new sandbox from a snapshot; fork takes a snapshot and clones it in one call while the parent keeps running. A fork therefore should not be described as continuing the parent's in-memory agent execution. (Platinum snapshots)

Platinum's fork documentation says the child cold-boots from disk and does not carry memory or processes. The operation comparison also says /tmp does not survive disk-only operations. Platinum describes stop/start separately as retaining memory and processes where possible; do not substitute that stop/resume behavior for the documented fork boundary. (Platinum snapshots)

Platinum's attached-volume rules impose additional limits. Sandbox snapshots and backups record attachment metadata, not volume contents; snapshot, clone, and fork with an attached Local Volume are documented as rejected. Where attachments exist, clone and fork require a volume policy: omit creates no child attachments, reattach reattaches only Shared Volumes and warns for read/write mounts, and clone returns an unsupported error rather than independently copying volumes. (Platinum snapshots)

Platinum documents read/write Shared Volume mounts as the same shared data, with concurrent writes to one path following last-writer-wins behavior rather than transactions. For an evaluation protocol, prefer separately versioned read-only fixtures or another explicitly controlled data arrangement instead of treating a reattached shared mount as an isolated clone. That recommendation is not a claim that Platinum supplies independent volume cloning. (Platinum snapshots)

Platinum's homepage describes evaluation runs as “identical,” but the detailed documentation narrows what a snapshot and fork preserve. Read that homepage language as a statement about the vendor's environment offering, not proof of whole-system replay or identical model outputs; the historical OpenAI example independently cautions against guaranteed model determinism. (Platinum overview; snapshot boundaries; OpenAI reproducibility caveat)

Proposed evaluation protocol

The following workflow is a recommendation, not a tested SDK recipe. Use the documented disk-only fork boundary as the starting constraint, then explicitly record the inputs and procedures your evaluation requires. (Platinum snapshots)

Prepare a baseline

Create a baseline manifest containing the code revision, dependency lockfile or resolved versions, fixture version, task definition, and scoring rubric. Specify the agent's permitted tools, completion conditions, time budget, retry policy, and what counts as an infrastructure failure rather than a task failure. Choose the repeated-run plan before inspecting outcomes, and keep credentials out of the manifest and retained traces.

Prepare a recorded cold-start procedure for required services and verify fixture checksums after startup. Put required baseline files outside /tmp, which Platinum says does not survive disk-only operations, and review the documented secret-scrubbing options before distributing a snapshot. Do not infer application-consistent database capture or preserved live connections from the phrase “full disk copy”; verify your application's preparation requirements separately. (Platinum snapshots)

Branch comparable runs

Record the baseline and snapshot identifiers where available, the variant, and the CPU, RAM, and disk allocation for each run. For repeated runs intended to share a baseline, use an explicitly identified snapshot rather than assuming successive forks of a changing parent share one disk state: Platinum documents fork as taking a new snapshot, whereas clone can start from a specified snapshot. Verify the documented attachment policy before branching. (Platinum snapshots)

Record the model identifier, prompts, tool versions, generation settings, and any supported seed or backend metadata. Check support for the actual model and endpoint in use: OpenAI's cited Cookbook example names historical preview models and is not evidence that today's models support the same controls. Treat matching settings as recorded controls, not proof that subsequent outputs must match. (OpenAI Cookbook)

Keep mutable external dependencies explicit in the run manifest. For a controlled comparison, consider a versioned fixture or permitted replay of a tool response; for a live-service evaluation, retain the observed response and timestamp instead. Label which mode was used so that a later reader can distinguish a controlled-input test from an attempt involving live inputs.

Compare and retain evidence

Retain outputs, redacted tool traces, completion status, errors, and the scoring configuration for every planned attempt. Report repeated-run variation and failed attempts alongside any aggregate result, and apply the same predeclared retry policy to each variant. If an evaluator uses a model, record that evaluator's prompts and configuration separately from the agent being evaluated.

Use a narrow conclusion such as “these variants were compared under the recorded baseline and input controls,” followed by the actual observations. Do not claim identical outcomes merely because disk baselines match: OpenAI's historical example explicitly leaves room for different responses under matching model controls. No benchmark result is supplied in this guide. (OpenAI Cookbook)

Example run ledger

Use the following blank ledger schema as a recording checklist, not as a provider API schema. Each row describes a field to populate with actual evidence; no results or identifiers below represent completed runs.

Field	What to record
Run and baseline identifiers	Unique attempt ID, baseline manifest version, snapshot ID where available
Variant and allocation	Agent variant, code revision, CPU, RAM, disk
Startup verification	Startup procedure version, fixture checks, readiness result
External fixture version	Fixture version or live-input mode, attachment policy, observed response reference
Model and tools	Model identifier, request configuration, prompt version, tool versions
Scoring configuration	Rubric version, evaluator version, retry and failure policies
Observed outcome	Actual completion status, score if applicable, error category
Retained evidence	Redacted output and trace locations, timestamps, deviations from the plan

Limits and FAQ

Does a snapshot guarantee identical outputs?

No. Platinum documents snapshots as disk copies, not guarantees about model responses. OpenAI's historical reproducibility example says responses can differ even when seed, request parameters, and backend fingerprint match. Use snapshots to establish a documented environment baseline, then measure output variation through repeated attempts rather than assuming it away. (Platinum snapshots; OpenAI Cookbook)

Does fork preserve processes?

No. Platinum documents fork as a cold boot from disk without the parent's memory or processes, while the parent continues running. Prepare an explicit startup procedure for the child sandbox. Do not infer fork behavior from Platinum's separate stop/start description, which discusses preserving memory and processes where possible. (Platinum snapshots)

Are attached volumes cloned?

Not by a sandbox snapshot or fork. Platinum excludes attached-volume contents from snapshots and backups, documents restrictions on attached Local Volumes, and requires a volume policy when clone or fork involves attachments. The documented reattach policy reuses Shared Volumes; the clone policy returns an unsupported error rather than creating an independent volume copy. (Platinum snapshots)

What must be recorded outside the sandbox?

For this proposed protocol, retain the task and rubric versions, model configuration, prompts, tool versions, external fixture references or observed responses, and redacted run evidence in a separately accessible record. Do not rely on a disk snapshot as the complete experiment record: Platinum excludes attached-volume contents, and OpenAI's historical example identifies model-side controls distinct from sandbox state. (Platinum snapshots; OpenAI Cookbook)

Sources and next step

Primary sources checked September 17, 2026:

• Platinum snapshots and persistence documentation: documented disk, fork, process, and volume boundaries.
• Platinum homepage: product definition and evaluation positioning, attributed to the vendor.
• OpenAI Cookbook: reproducible outputs with the seed parameter: historical example and explicit limits on determinism, not current model-support guidance.

Review the official Platinum snapshot and fork documentation before preparing an evaluation baseline. For a separate budgeting discussion, see the AI agent sandbox cloud cost comparison; keep cost planning distinct from evidence about agent quality or repeatability.