valve_

Read benchmark results

Interpret safety coverage, throughput, latency, paired confidence intervals, A/A controls, and retained evidence.

A benchmark result consists of the measurement, its correctness coverage, and its execution conditions. Read the manifest and safety findings before using timings to judge a change. Exit 0 alone does not establish nonregression.

Safety and performance are independent

Safety statusInterpretation
PASSApplicable checks were observed and passed for this trial's coverage.
FAILAn invariant or execution check failed. Inspect violation counts and samples.
UNVERIFIEDRequired evidence or coverage was missing. This cannot be treated as a pass.

Safety checks include rate/window/scoped admission, leases and concurrency, callbacks after cancellation or maxWait, and lost-job accounting. Every deadline violation is fatal to the run. Safety from recorded invalid attempts is retained; resource noise cannot turn a violation into a confidence interval.

Distributed performance trials use unmodified Lua. Exact rate/window checks need server-side logs, so those rows can remain UNVERIFIED even when callback checks pass. A separate instrumented audit or library valve-audit can pass without changing the performance row's coverage.

Performance verdictInterpretation
DESCRIPTIVEFewer than five independent paired repetitions; no confidence interval. Diagnostic probes also remain descriptive.
EQUIVALENTThe interval lies strictly inside both practical margins.
NONINFERIORThe interval excludes regression beyond the negative margin.
IMPROVEMENTThe interval is entirely beyond the positive practical margin.
REGRESSIONThe interval is entirely beyond the negative practical margin.
INCONCLUSIVEThe interval or available evidence cannot resolve the comparison.
UNSTABLEThe A/A control cannot meet repeatability requirements at its final look; candidate conclusions are withheld.

These are per-metric, per-cell conclusions. The margin represents a practical difference, not simply whether the estimated change is above or below zero.

Throughput and admission metrics

The measurement interval is [start, end). An admission, callback start, and completion are different events and can fall in different windows. Weighted workloads also distinguish jobs from cost units.

MetricMeaning
offeredPerSecCost scheduled to arrive per second during the window; offeredJobsPerSec counts jobs instead.
ruleCapacityPerSecLong-run rule capacity in cost units/second, excluding the initial burst bank.
idealBoundPerSecModelled zero-latency greedy supply for compatible offered demand and known starting state; unavailable when assumptions cannot be established.
idealBoundNonPrefixPerSecCorresponding skip-policy bound for supported weighted workloads; not a globally optimal weighted schedule.
admittedPerSecAdmitted cost/second, using complete server logs in correctness arms or observable client grant costs in performance arms.
clientGrantsPerSecClient-observed number of granted jobs/second.
usefulStartsPerSecCallbacks that actually began in the window, in jobs/second.
usefulStartCostPerSecCost of those started callbacks/second.
completionsPerSecSuccessful completions in the window, in jobs/second.
boundEfficiencyAdmitted cost divided by the ideal bound.
capacityUtilizationAdmitted cost divided by long-run rule capacity over the window.
goodputStarted cost divided by offered cost for open-loop demand.

The ideal bound assumes zero RTT, timer error, and CPU cost. It is a reference bound, not attainable throughput. Current measured quota IDs start fresh because warmup uses separate IDs. Faulted/lifecycle cells, finite-client demand, and unsupported saturated weighted/scoped/lease streams withhold the bound instead of inventing a denominator. Independent memory-store workers have independent budgets; the analyzer accounts for their copies.

Capacity utilization can exceed 1 in a short window because the long-run capacity excludes initial burst credit and fixed-window boundary effects. Inspect admission safety before interpreting that ratio.

Demand and generator validity

Open-loop arrivals are generated from a seed independently of the system's responses. Submission lag is measured against intended arrival time. A maximum lag over 5 ms marks generator saturation and excludes the affected trial from performance inference while retaining safety evidence.

Maintained-backlog cells track minimum backlog depth to show demand remained available. Finite-client closed loops generate new work after responses and think time, so demand depends on system performance. They have no exogenous ideal-bound ratio.

Latency and outcomes

Always read the counts alongside latency. Started callbacks can complete or fail; non-starting jobs can expire, abort, reject, or remain unserved at the observation end. Unserved jobs are right-censored at their observed waiting time. A missing outcome after close is a lost-job safety issue, with declared worker-kill cases handled separately.

Reports expose three time-to-start views for each applicable cohort:

  • Conditional wait (waitP50, waitP99, and related fields): percentiles among callbacks that started. Fast values can coexist with many failures.
  • Cumulative incidence (cifP50, cifP99): Aalen-Johansen estimates for starting, with expiry/rejection as competing risks and abort/unserved work as censoring. A quantile is not reached if the estimated fraction starting never reaches it. Treating aborts as non-informative censoring is an assumption.
  • Failure-ranked wait (failureRankedP99): non-starting outcomes rank worse than every observed start. This is deliberately conservative.

Waits use intended arrival times, so generator delay is visible. Deadline success instead checks whether a callback started within maxWait of actual submission. deadlineSuccess is the fraction of deadline-bearing jobs that met that condition, not the fraction that eventually completed.

Cost, fairness, and resource diagnostics

Metric familyInterpretation
callsPerGrant, zeroGrantPerGrantAcquisition calls or zero-grant calls per client grant.
scriptsPerUsefulJobRecorded store-script calls per callback start.
earlyZeroPerGrant, lostRacePerGrantAttribution from sufficient server-log evidence; otherwise withheld.
redisCpuUsPerGrant, evalshaUsPerCall, redisCoresServer CPU and script cost from Redis/Valkey INFO deltas.
commandsPerSecServer command-count deltas, excluding the sampler's INFO commands; commands inside scripts remain server work.
appCpuUsPerGrant, appCoresWorker CPU cost; despite its name, appCpuUsPerGrant divides by callback starts.
eldP99Ms, maxRssMBWorst worker event-loop p99 and maximum worker RSS; not aggregate fleet RSS.
jain, minShare, maxNoStartGapMsEqual-demand fairness and starvation diagnostics when applicable. Single-worker cells exclude Jain from decision metrics.
recoveryMs, faultDipRatioRecovery toward 90% of pre-fault throughput and the observed dip. Missing recovery within the window remains unresolved.
Heap/RSS/server memory slopesSoak trend diagnostics. Correlated time samples and GC behavior limit interpretation; a fitted slope alone does not prove a leak.

Correctness instrumentation adds work to Lua, including log appends. Its CPU, command counts, and timings must not be compared as ordinary performance arms.

Paired comparisons and A/A

One replication is an independent cold trial for each arm, with fresh worker processes and fresh store state. Arms share a workload seed within a replication. The analyzer pairs trial-level metric values, not individual requests. Thousands of requests in one trial do not supply thousands of independent reps.

Matrix arms are base, aa (the same baseline artifact run independently), and head (candidate). Trial order rotates across arms and cell order is shuffled deterministically per replication to reduce ordering bias.

For each paired metric, the analyzer computes candidate - baseline and a Student-t interval on those differences. It orients lower-is-better metrics so improvement is positive when choosing the verdict; report differences remain in original units. Relative margins resolve against the baseline mean, with any declared absolute minimum margin.

The manifest fixes the metric set, practical margins, alpha, and replication looks before execution. Ordinary targeted studies use a fixed look at five reps; decision studies use 5/10/20. Confidence statements require at least five complete pairs and a preregistered look. Missing, invalid, or nonfinite pairs withhold survivor-only inference.

For matrix inference, the interval alpha is divided by the number of planned cell/metric comparisons in the suite and by the number of looks. This Bonferroni control includes the full declared family, including cells resolved earlier. Library comparisons allocate their own planned category/arm/metric families. Sign-flip permutation tests provide robustness diagnostics against zero difference: exact enumeration through 16 pairs, seeded Monte Carlo above that, with Holm-adjusted p-values across applicable families. These p-values do not replace the practical-margin interval decision. Bootstrap intervals are not reported in v1.

Why the A/A control can block a result

The baseline's independent A/A measurement tests whether this experiment can reproduce its own baseline. At the final look, an A/A interval that excludes zero or extends outside the practical margins is UNSTABLE. A/A equivalence requires compatibility with zero and an interval strictly inside both margins.

If any preregistered A/A metric is unresolved or unstable, the cell's inferential candidate verdicts become INCONCLUSIVE. Failed/aborted execution and incomplete evidence also withhold conclusions. A faster point estimate does not override these gates.

Plan a new run with better isolation, longer windows, or more repetitions when precision is insufficient. Keep the original margins and evidence. Changing margins or deleting valid observations to force equivalence changes the claim.

Retained files

For real-time runs, .perf/runs/<run-id>/ contains:

FileContents
manifest.jsonHarness/build identities, fingerprint, arms, cells, seeds, decision plan, resource controls, and plan hash.
rows.jsonlAppend-only trial records, including failed/resource-invalid attempts and safety findings.
trials/<trial-id>/trial.jsonDetailed trial record when produced. Early setup/resource failures can exist only in the retained row.
trials/<trial-id>/payload.json.gzPacked jobs, calls, worker summaries, server events when available, INFO snapshots, faults, and samples when produced.
summary.jsonAggregated metrics, safety, comparisons, exclusions, and warnings.
metrics.csvArm-level absolute metrics.
comparisons.csvPaired differences, intervals, margins, and verdicts.
report.md, optional report.htmlHuman-readable presentation of the saved summary.

Portable/diagnostic runs retain command-specific JSON and a manifest; diagnostic probes also write a short report. report <run-id> --html can render these local results. Nonfinite numeric values use tagged objects such as {"$valveNumber":"Infinity"}. Consumers should preserve this representation rather than coercing missing or unreached values to zero. See src/core/codec.ts for the encoding.

When sharing a claim, include the complete run directory, source revision, build/patch identities, harness and plan hashes, environment fingerprint, server images, authority classification, safety status, performance verdicts, and excluded or invalid attempts. Do not pool runs across fingerprints or protocols. The same harness must measure both artifacts within a comparison; separately produced summaries are not a substitute for paired evidence.

On this page