Public API

These entries are generated from the docstrings of PerfChecker's exported Julia bindings. The Full API adds documented internal hooks from Core; the six companion APIs are documented separately. Julia's help mode uses these same source docstrings: type ?run_scenarios or ?AdvisorConfig after loading PerfChecker.

Use a workflow guide when setting up your first experiment:

Task	Entry points	Guide
Measure an existing test	discover_testitems, run_testitems	First check
Define and compare workloads	FeatureSpec, PackageSuite, SoftwareSuite, plan_suite, run_suite	Software suites
Share a correctness protocol across collectors	ScenarioSpec, run_scenarios, diagnose, compare_scenarios	Shared scenarios
Read portable evidence	read_run_bundle, verify_run_bundle, query_bundle	Run bundles
Discuss evidence and review an implementation	AdvisorConfig, chat_advice, implement_advice	MCP guide
Use a notebook, web view or custom figure	Interface-package entry points	Interface ownership
Select suite runs in a terminal UI	PerfCheckerTachikoma.tui_model, tui, refresh!, close!	Optional package APIs
Collect optional hardware counters	Companion-owned measure_counters, counter_executor, run_counter_suite	Counter APIs

Optional interface methods require their documented package to be installed and loaded. An exported generic function can exist before its optional methods are available. Loading an interface and constructing a configuration are separate from starting a measurement.

The generated entries below cover Core exports only. The six companion Public and Full APIs render their actual Julia docstrings in isolated projects and share this site's navigation and search. Companion bindings and methods retain their owning modules and pinned source links; they are not silently imported into the Core documentation environment. The ownership index and Julia help examples explain how to load the selected owner.

Exported docstrings

PerfChecker.PerfChecker Module

Performance measurement, comparison and qualification for Julia workloads. Start with discover_testitems for existing tests or plan_suite for explicit feature workloads. Interface packages consume the same plans and saved evidence; measurement and diagnostic processes run in separately prepared environments.

source

PerfChecker.AdvisorConfig Type

AdvisorConfig(; endpoint="http://127.0.0.1:8081/v1/chat/completions",
              model="local", timeout=90, max_tokens=512,
              max_evidence_chars=12000, api_key_env="", allow_remote=false,
              protocol=:chat_completions, provider_package="", instructions="",
              mcp_tool="", mcp_prompt_argument="prompt", mcp_arguments=Dict(),
              mcp_version="2026-07-28", mcp_response=:text)

Validate and store an optional advisor connection. Construction does not start a server, install a provider package or send evidence. The worker loads provider_package when supplied, otherwise HTTP; that package must already be available in the advisor's project environment.

Connection and limits

• endpoint is an HTTP(S) URL without credentials, a query or a fragment. Paths accept RFC 3986 path characters and valid percent escapes; raw whitespace, backslashes and encoded control characters are rejected. HTTP is accepted for localhost, 127.0.0.1 and [::1]; other hosts require HTTPS and allow_remote=true.

• model selects the model for Chat Completions and Ollama. PerfChecker does not bundle a model; the MCP transport does not send this field.

• timeout bounds the isolated request, including worker startup, in seconds (0 < timeout <= 3600). max_tokens is in 1:4096 and is sent to Chat Completions/Ollama, but not to MCP tools.

• max_evidence_chars bounds the complete serialized measurement/recommendation array in Unicode characters (1000:100000), including delimiters and JSON escaping. Whole records are retained or omitted; conversation limits are separate.

• api_key_env names a credential environment variable, not its secret value. instructions adds up to 5000 characters before the fixed response contract.

MCP tool selection

For protocol=:mcp_http, choose a real server tool explicitly with mcp_tool. mcp_prompt_argument is the tool argument receiving the bounded evidence request; mcp_arguments supplies its other JSON-compatible arguments with string keys. The prompt argument is reserved and cannot also appear in mcp_arguments. The serialized mcp_arguments object is limited to 12,000 UTF-8 bytes. MCP servers choose their own model and generation limits. To request those settings, provide only arguments admitted by the selected tool's input schema in mcp_arguments; model and max_tokens do not configure MCP generation. Supported mcp_version values are "2025-11-25" and "2026-07-28". mcp_response=:text supports conversation; :structured validates narrative cards against supplied evidence IDs. Protocol shape validation does not prove that an agent's prose or proposed changes are correct.

Example

This prepares a configuration without making a network request:

julia> config = AdvisorConfig(protocol=:mcp_http, endpoint="http://127.0.0.1:8081/mcp", mcp_tool="ask");

julia> (config.protocol, config.mcp_tool, config.mcp_response)
(:mcp_http, "ask", :text)

source

PerfChecker.BundleComparison Type

BundleComparison

Evidence returned by compare_bundles. baseline_run_id and candidate_run_id identify the inputs; inputs_passed records their execution validity. environment_status is :identical, :compatible (with warnings), or :incomparable. records retain each matched metric's definitions, sample counts, selected statistic, deltas, policy and status.

Use comparison_verdict for the overall decision: a successfully executed bundle or a small delta alone does not qualify the comparison.

source

PerfChecker.CancellationToken Type

CancellationToken()

Create a thread-safe, initially unrequested cancellation token for scenario, diagnostic or advisor operations. Pass the same token to the operation and call cancel! from another task to request interruption. A requested token is not reset automatically; use a new token for an independent operation.

Requesting cancellation does not wait for worker termination. Wait for the operation's final status before accessing resources owned by its worker.

julia> token = CancellationToken(); cancel!(token);

julia> token.requested[]
true

source

PerfChecker.ComparisonPolicy Type

ComparisonPolicy(id; package="", feature="", comparison_key="",
                 baselines, candidates, aggregation=:median)

Select named targets for within-suite version comparison. baselines and candidates are nonempty string-label vectors; duplicates within each are removed. With several baselines, aggregate their per-target statistic using :median, :mean, :minimum or :maximum before comparing each candidate. This reference aggregation is separate from the sample statistic within a target.

At least feature or comparison_key is required. A nonempty comparison key takes precedence over the feature selector; empty package matches any package. IDs must be nonempty and unique in a suite. This declares selections only; metric budgets are supplied to compare_suite_versions or write_suite_reports, not stored in the policy.

ComparisonPolicy("recent-reference"; feature="sum_squares",
    baselines=["0.5.4", "0.5.5"], candidates=["dev@0.5.6"],
    aggregation=:median)

source

PerfChecker.CompatibilityReport Type

CompatibilityReport(suite, profile, checked_at, diagnostics)

Store suite/profile identity, UTC check time and per-run preparation diagnostics. Produced by preflight_suite, with dependency-resolution and optional functional-probe outcomes. This does not contain measured workload performance. Use preflight_passed for blocking status and compatibility_report_dict for portable transport.

source

PerfChecker.ExternalCommandSpec Type

ExternalCommandSpec(id::Symbol, language, command; directory=pwd(),
                    environment=Dict{String,String}(), timeout_seconds=300)

Describe an external provider invoked without a shell by run_external_command. command is a nonempty vector of executable and arguments; directory must exist and is normalized to an absolute path. Environment keys/values are converted to strings and added to inherited values. The positive timeout is in seconds. Invalid commands, directories and nonpositive timeouts raise ArgumentError; construction does not start a process. The provider must write perfchecker-provider-result/1 JSON to the runtime's PERFCHECKER_OUTPUT path. Language is descriptive metadata, not executable selection.

source

PerfChecker.ExternalMemorySnapshot Type

ExternalMemorySnapshot

Normalized workload-owned external-memory result with schema, timestamp, optional live/reserved byte gauges, cumulative allocation/free totals, provider, status and message. Use external_memory_snapshot to validate a dictionary or named tuple. Workload allocator counters are separate from Julia allocation counters and process RSS.

source

PerfChecker.FeatureRun Type

FeatureRun
FeatureRun(planned, status, elapsed_seconds::Float64, result, message::String)

Result of one planned feature execution. planned identifies the target; status distinguishes :pass, :unavailable, :blocked, :invalid and :error. elapsed_seconds is orchestration time, not a replacement for measured samples. result holds backend data when available; qualification retains probe, correctness and provenance evidence even when execution fails. The five-argument convenience constructor initializes unqualified evidence; it does not infer correctness from a :pass status. Normal suite execution produces these records automatically.

source

PerfChecker.FeatureSpec Type

FeatureSpec(id::Symbol; description="", backend=:benchmark,
    variants=nothing, entrypoint=nothing, workload=nothing,
    since=nothing, until=nothing, excluded=VersionNumber[],
    julia_since=nothing, julia_until=nothing, julia_excluded=VersionNumber[],
    comparison_key=string(id), options=Dict{Symbol,Any}(),
    probes=ProbeSpec[], oracle=nothing, state_policy=:fresh)

Declare one collector-specific feature contract. Supply either an existing entrypoint (creating one FeatureVariant with the package window) or a nonempty variants collection. The julia_* bounds restrict the controller runtime separately from package versions. workload defaults to id; sharing it across collectors lets reports and interfaces identify one business workload.

backend selects the checker, for example :benchmark, :chairmark, :profile, :wall_profile, :profile_alloc or :alloc. Availability and required backend packages are checked when planning/executing, not installed by this constructor. options has symbol keys. Execution merges feature options, variant options, then call-site overrides in that order.

state_policy=:fresh prepares separate state for measured invocations on the fresh-capable backends and requires evals=1; :reuse intentionally reuses state. An explicit options[:state_policy] takes precedence over this keyword. probes declare capability checks and oracle is an OracleSpec or nothing; without an oracle, successful execution is not correctness validation. No workload is run or environment prepared during construction.

source

PerfChecker.FeatureVariant Type

FeatureVariant(entrypoint; since=nothing, until=nothing,
               excluded=VersionNumber[], comparison_key="",
               options=Dict{Symbol,Any}())

Declare an existing Julia workload file for an inclusive package-version window. entrypoint becomes an absolute path relative to the current working directory; it must already be a file. The worker includes this file, which defines perf_workload(state) and may define setup, cleanup, synchronization and qualification hooks.

comparison_key groups compatible implementations for comparison; an empty key falls back to the feature ID during planning. options has symbol keys and overrides the parent feature's options at execution. This constructor checks the path and declarations without including or measuring the workload. Planning rejects overlapping variants that both match a selected package version.

source

PerfChecker.InvestigationJob Type

InvestigationJob

Handle returned by launch_investigation. It owns an asynchronous task, its cancellation token and synchronized status, result/advice, error and timing fields. Use investigation_status instead of reading mutable fields while the task is active. cancel! requests interruption; only wait_investigation establishes that the task has finished.

source

PerfChecker.InvestigationView Type

InvestigationView(payload::Dict{String, Any})

Display wrapper for recorded discovery, diagnosis, advice, narrative and bounded investigation evidence. Plain-text display summarizes the records; HTML display escapes their text for notebook and report presentation. Construct through investigation_view to validate the payload's supported schema. Displaying a view does not execute its target workload or verify model prose.

source

PerfChecker.IsolatedNetworkCommandResult Type

Result of an external command executed in a requested network namespace. Carries the command outcome, captured output, provider and network sample. Inspect isolated_network_result_dict before attributing traffic to the target; unavailable isolation is not a zero-traffic measurement.

source

PerfChecker.JuliaRegressionInvestigation Type

JuliaRegressionInvestigation

Source-attribution evidence returned by investigate_julia_regressions. campaign_id links it to its runtime campaign, generated_at is a UTC timestamp, and comparisons holds ranked source-line deltas or explicit missing-evidence records for each baseline/candidate pair.

This is an attribution aid, not a causal diagnosis or a generated minimal reproduction. A dominant sampled source frame alone cannot establish a Julia bug.

source

PerfChecker.JuliaRuntimeCampaign Type

JuliaRuntimeCampaign

Result of run_julia_runtime_campaign: a campaign ID, suite path, profile, baseline runtime ID, UTC start/finish times, per-runtime runs, and baseline-to-runtime comparisons. Each run retains the requested spec, observed runtime probe when available, execution/failure details and report or bundle paths.

This object summarizes finished orchestration; some runs may have failed or timed out. Use runtime_campaign_passed, rather than the presence of report files, to assess execution and comparison policy together.

source

PerfChecker.JuliaRuntimeSpec Type

JuliaRuntimeSpec(id::Symbol, selector::AbstractString;
                 role=:candidate, source=:juliaup,
                 executable="julia", arguments=String[])

Describe one Julia executable in a runtime campaign, independently of the package versions selected by its suite. role is :baseline, :candidate or :control; a campaign requires exactly one baseline.

With source=:juliaup, the command is executable +selector (for example, julia +1.10). With source=:executable, executable must name an existing file and the nonempty selector is its descriptive label. arguments are additional command-prefix arguments, copied as strings. Construction validates these fields but neither installs a Julia channel nor launches a process.

baseline = JuliaRuntimeSpec(:lts, "1.10"; role=:baseline)
candidate = JuliaRuntimeSpec(:current, "release")

source

PerfChecker.NativeDependencyEvidence Type

NativeDependencyEvidence(phase, packages, artifact_files, loaded_libraries, warnings)

Group a named capture phase with resolved package records, declared artifact binding files, observed current-process libraries and warnings. Use dependency_evidence to collect it and dependency_evidence_dict for transport. Declared artifacts and loaded libraries are distinct evidence; this type does not assert that every declared artifact was installed or loaded, nor cover child processes or services.

source

PerfChecker.NetworkInterfaceSnapshot Type

NetworkInterfaceSnapshot

Store interface identity, monotonic timestamp, unsigned byte/packet/discard counters in both directions, and provider. Obtain observations with network_interface_snapshot and differences with network_interface_delta. Counters cover all traffic on that interface; they are not automatically attributable to one workload or process.

source

PerfChecker.NetworkIsolationSpec Type

NetworkIsolationSpec(; provider=:auto, distribution=nothing, interface=nothing,
                     external_connectivity=false, dns_servers=nothing)

Describe a Linux user/network namespace or WSL2 namespace request. :auto selects :wsl2_netns on Windows and :linux_netns elsewhere; construction does not establish that this host can run it. distribution optionally selects a WSL distribution. The interface defaults to lo, or tap0 for external connectivity. External mode uses a dedicated TAP interface and requires slirp4netns; DNS defaults to ["1.1.1.1"] only in that mode.

Unsupported providers, empty interfaces, external mode with lo, non-IPv4 DNS addresses or DNS without external connectivity raise ArgumentError. This stores a request without installing tools or starting a namespace. Use network_isolation_capabilities to inspect the prepared host.

source

PerfChecker.OracleSpec Type

OracleSpec(; function_name=:perf_oracle, required=true)

Declare a correctness check for a FeatureSpec, outside the measured region. Its entrypoint may define function_name(state), function_name(), or function_name(state, result). For the last form, qualification executes the workload and its optional synchronization hook before calling the oracle.

The result uses the same status normalization as ProbeSpec. A false result or exception invalidates correctness. required=false permits an absent oracle function, recorded as not_checked; it does not ignore an oracle that exists and fails. This declaration alone does not run the workload.

OracleSpec(function_name=:verify_result)

source

PerfChecker.PackageSuite Type

PackageSuite(package; id=Symbol(package), environment=nothing,
    worker_environment=nothing, source=nothing, versions=:all,
    dev_sources=String[], release_pins=Dict{VersionNumber,Vector{Any}}(),
    features, include_dev=true, candidates=SuiteCandidate[])

Declare package targets and their FeatureSpec contracts. versions is :all, :latest or a vector of VersionNumber; explicit versions are deduplicated and sorted. Registry lookup occurs during planning when needed. include_dev adds the development target outside the :release profile; candidates adds named Git targets outside that profile.

worker_environment is an existing seed directory copied for isolated workers, not the controller project. environment is a compatibility alias; differing paths supplied to both keywords are rejected. The default is pwd(). Keep PerfChecker in the controller, and measurement backends/workload dependencies in the worker seed: normal execution rejects a seed that directly depends on PerfChecker unless PerfChecker itself is the measured package.

source is an existing development-source directory; dev_sources lists existing local dependency directories. release_pins maps versions to extra package specifications. Construction validates declarations but performs no package resolution, checkout or measurement. A development target requires a source path at execution and a readable project version at planning.

source

PerfChecker.PerfConfig Type

PerfConfig(backend::Symbol; path=pwd(), kwargs...)
PerfConfig(backend::Symbol, options)

Julia-native public configuration object for @check.

PerfConfig keeps the existing dictionary-based API available while giving REPL, scripts, and Pluto notebooks a clearer object to pass around. Keyword arguments are stored with symbolic keys and validated by normalize_config just before a check runs. Explicit options override check_preferences. Construction does not read or write preferences or start a worker.

Example:

config = PerfConfig(:benchmark; path=pwd(), tags=[:local], samples=10)
result = @check config begin
    nothing
end begin
    sum(1:100)
end

source

PerfChecker.PerformanceDocumentBlock Type

PerformanceDocumentBlock(id, title, query; views=[:summary, :comparison, :plots],
                         interactive_url=nothing)

Describe a report section backed by a PerformanceQuery. Supported views are :summary, :comparison, :observations, :diagnostics, :plots and :artifacts; duplicate views are removed and unsupported views raise ArgumentError. interactive_url is optional presentation metadata, converted to String without fetching or validating the destination. Use performance_document_block(bundle, block) to attach saved evidence and provenance. This constructor writes no report.

source

PerfChecker.PerformanceQuery Type

PerformanceQuery(; id="query", resources=[:observations, :diagnostics,
                 :artifacts, :plots, :comparison], predicates=QueryPredicate[],
                 order_by=Pair{String,Symbol}[], limit=0)

Describe a selection of saved evidence for query_bundle. All predicates must match. order_by contains dotted-field pairs with :asc or :desc; earlier pairs take precedence. limit=0 leaves each resource unbounded; a positive limit caps each resource independently. Duplicate resources are removed. Construction does not inspect a bundle or run measurements.

Throw ArgumentError for unsupported resources, empty sort fields, unsupported directions or a negative limit.

julia> query = PerformanceQuery(resources=[:observations], limit=5);

julia> performance_query_dict(query)["resources"]
1-element Vector{String}:
 "observations"

source

PerfChecker.PlannedFeatureRun Type

PlannedFeatureRun

One resolved package, feature and target produced by plan_suite. suite, package_suite, feature and target retain their declarations; variant is the matching implementation or nothing, and comparison_key identifies comparable measurements. planned_status is :ready or :unavailable; reason explains unavailable package/Julia compatibility. Unavailable rows are preserved for reports and are not executed.

Use planned_run_id for UI selection and workload_id for the business workload. A planned row contains no completed measurement evidence.

source

PerfChecker.ProbeSpec Type

ProbeSpec(id::Symbol; function_name=Symbol("perf_probe_", id),
          blocking=true, category=:native)

Declare a capability check for a FeatureSpec. The feature entrypoint must define function_name; qualification calls it with the prepared state when applicable, otherwise with no arguments, outside the measured region.

category is one of :native, :runtime, :service, :device or :custom. A missing function is recorded as unavailable. false, an exception, or an explicit failed result records a failure. A blocking failed/unavailable probe prevents a valid measurement; blocking=false retains a warning instead. Results may be Bool, nothing, a message string, or a dictionary/named tuple with status, message and optional evidence. Constructing the specification does not execute the check.

ProbeSpec(:database; function_name=:check_database,
          blocking=true, category=:service)

source

PerfChecker.ProcessMemorySnapshot Type

ProcessMemorySnapshot(pid, timestamp_ns; rss_bytes=nothing,
                      peak_rss_bytes=nothing, private_bytes=nothing,
                      provider="fixture", status=:observed, message="")

Store byte counters for one process at a monotonic timestamp. Counters are nonnegative integers or nothing, normalized to UInt64. status is :observed or :unavailable; observed snapshots need at least one counter. Invalid status or counters raise ArgumentError. Construction does not inspect the operating system; use process_memory_snapshot for observations. Peak RSS is a process-lifetime high-water mark, and missing counters are unknown.

source

PerfChecker.QueryPredicate Type

QueryPredicate(field, operator, value=nothing)

Select records by a dotted field path. Fields resolve in the record first, then its attributes; manifest. explicitly selects bundle metadata. Supported operators are :equals, :not_equals, :one_of, :contains, :prefix, :greater_or_equal, :less_or_equal and :exists. Equality also accepts equal string representations; text matching ignores case. Numeric comparisons require numeric operands. Missing fields match only :exists with value=false. With :exists, nothing means require presence.

Throw ArgumentError for an empty field, unsupported operator, noncollection :one_of value, or an :exists value other than nothing or Bool.

julia> predicate = QueryPredicate("metric", :equals, "julia.wall.time");

julia> performance_query_dict(predicate)["operator"]
"equals"

source

PerfChecker.ResourceEnvelope Type

ResourceEnvelope(elapsed_seconds, process_before, process_after,
                 external_before, external_after; external_requested=false)

Combine before/after snapshots and collector wall time. Process snapshots must have the same PID and elapsed time must be nonnegative, otherwise throw ArgumentError. external_requested records intent, not availability. Construction stores evidence without sampling or evaluating limits. Collector time includes collection overhead and differs from benchmark sample durations.

source

PerfChecker.RunBundle Type

RunBundle(manifest, measurement_definitions, observations, diagnostics, artifacts)

In-memory portable evidence: a string-keyed manifest and vectors of dictionaries for measurement definitions, observations, diagnostics and artifact descriptors. Construction stores supplied records without validating their schema or loading artifact bytes. Use read_run_bundle for saved bundles or read_provider_result for validated external results. bundle_passed checks execution; comparison APIs decide regressions.

source

PerfChecker.ScenarioCatalog Type

ScenarioCatalog(root::AbstractString, scenarios::AbstractVector{ScenarioSpec})

Store an absolute package root and a copied vector of explicitly declared scenarios. Reject duplicate (id, implementation) identities. Construction does not discover tests, evaluate source or prepare dependencies. Proposals returned by discover are not executable catalog members until the caller defines their factory and correctness protocol.

julia> catalog = ScenarioCatalog(pwd(), ScenarioSpec[]);

julia> isempty(catalog.scenarios)
true

source

PerfChecker.ScenarioSpec Type

ScenarioSpec(id; source, factory="make_case", implementation="default",
             parameters=Dict(), fixtures=String[], collectors=[:benchmark],
             repeatable=false, requirements=String[])

Declare a Julia workload and correctness protocol shared by measurement and diagnostic workers. id and implementation form the scenario identity; factory is a possibly dotted Julia function name in the existing source file. The constructor resolves source and fixture paths, checks their existence and validates TOML-compatible parameters. It does not include the source or invoke the factory. requirements lists packages that must already be available to the worker; packages are not installed implicitly.

The factory receives the parameter dictionary and returns a named tuple with callbacks prepare(), operation(state) and verify(state, result). Verification must return true. Optional synchronize(state, result) belongs to the operation boundary; cleanup(state) releases state when Julia unwinds normally, including operation or verification errors. Forced worker termination cannot guarantee an arbitrary user cleanup callback, so externally owned files must remain recoverable. An optional availability() callback returns (available=Bool, reason=String) and can mark the workload unavailable before measurement.

collectors is a nonempty selection of :benchmark, :chairmark, :profile and :profile_alloc. Each run prepares a fresh worker and preserves its collector identity. repeatable records the workload's declared repeatability; it does not prove it or bypass the correctness oracle. Empty or unsupported collector selections are rejected; repeated collector values are normalized to one entry.

spec = ScenarioSpec("sum_squares"; source="perf/cases.jl",
    factory="make_sum_case", collectors=[:benchmark, :profile_alloc],
    parameters=Dict("length" => 1000), requirements=["Example"])
# The source factory has not run; select a prepared worker project before execution.

source

PerfChecker.SoftwareSuite Type

SoftwareSuite(id::Symbol, packages::AbstractVector{PackageSuite};
              description="", comparisons=ComparisonPolicy[])

Combine a nonempty collection of package suites into one named performance suite. Package-suite IDs and comparison-policy IDs must each be unique. description is report/UI text; comparisons declares within-suite references. The constructor collects declarations but does not resolve targets or measure them. Use plan_suite, then run_suite or launch_suite.

source

PerfChecker.SoftwareSuiteResult Type

SoftwareSuiteResult

Completed suite evidence: the original plan, UTC start/finish timestamps and a vector of FeatureRun records. Use suite_verdict for qualification and write_suite_reports to export it without repeating measurements. Individual records may be unavailable, blocked, invalid or erroneous. This object is not proof that every selected row ran or that a baseline policy passed.

source

PerfChecker.SuiteCandidate Type

SuiteCandidate(label, revision; source=nothing,
               compatibility_version=nothing, dependencies=Any[])

Declare a named Git branch, tag or commit to measure alongside registry releases and the development source. label identifies the target in plans and reports; revision is the Git ref to resolve. Both must be nonempty.

source supplies a repository URL or path; otherwise planning uses the package suite's source. compatibility_version selects the FeatureVariant window; when omitted, planning uses the current project version. dependencies contains additional package specifications for this candidate's worker. Construction neither checks out Git nor installs dependencies. Candidate labels must be unique within a package and across all of its selected target labels.

SuiteCandidate("proposed", "feature/faster-sum";
    source="https://github.com/example/Example.jl.git",
    compatibility_version=v"0.5.5")

source

PerfChecker.SuiteJob Type

SuiteJob

Handle returned by launch_suite for asynchronous controller execution. Use suite_job_status, suite_job_progress, cancel_suite! and wait_suite instead of mutating its task and reference fields. The handle belongs to its controller process; callers must request cancellation and await its cleanup before shutting that process down. A :complete job means orchestration finished and may still contain failing feature records.

source

PerfChecker.SuitePlan Type

SuitePlan

Resolved suite, execution profile, ordered runs and comparison policies. Obtain it with plan_suite, inspect suite_plan_dict or select rows with select_suite_plan, then execute it. The struct retains vectors and their declarations; it is not a frozen snapshot of source files or registry state. Creating or holding a plan does not start measurement workers.

source

PerfChecker.VersionComparison Type

VersionComparison

Within-bundle version evidence returned by compare_suite_versions. run_id identifies the bundle and input_passed records its execution validity. availability preserves observed, failed and unavailable targets; series contains plottable metric points, and records contains baseline/candidate decisions. warnings explain missing version metadata or definitions.

Use version_comparison_verdict to distinguish qualified, regressed, invalid and inconclusive evidence. An available plot is not itself a passing performance policy.

source

PerfChecker.VersionWindow Type

VersionWindow(; since=nothing, until=nothing, excluded=VersionNumber[])

Declare an inclusive version interval with explicit excluded versions. nothing leaves the corresponding end unbounded; since and until are converted to VersionNumber. A reversed interval raises ArgumentError. excluded must be a vector of VersionNumber values and is stored as a set.

Used for both package-version variants and a feature's Julia compatibility. Constructing the window does not query registries or install versions.

VersionWindow(since=v"1.0", until=v"2.0", excluded=[v"1.4.0"])

source

PerfChecker.@check Macro

@check backend config begin
    # preparation code
end begin
    # measured code
end

Run a performance check using backend and return a CheckerResult. Alternatively, pass a nonempty vector of backend symbols to return a vector of CheckerResults in that order. All collectors and configurations are validated before any worker starts. Repeated symbols perform separate checks; each check uses the same preparation and measured blocks and retains its own cleanup. Load optional collector packages before requesting their symbols (for example, BenchmarkTools for :benchmark and Chairmarks for :chairmark). An error in a check stops the sequence after that check's cleanup.

The public config argument is usually a Dict. PerfChecker merges it with backend defaults, validates it with normalize_config, copies the environment at config[:path], launches isolated Julia workers, installs the requested package versions, runs the two code blocks, and stores result tables plus metadata.

Example:

using PerfChecker, BenchmarkTools, Chairmarks

config = Dict(:path => @__DIR__, :samples => 10, :evals => 1)

result = @check :benchmark config begin
    using Random
end begin
    sum(rand(Random.MersenneTwister(1), 1_000))
end

results = @check [:alloc, :benchmark, :chairmark] config begin
    using Random
end begin
    sum(rand(Random.MersenneTwister(1), 1_000))
end

source

PerfChecker.@check Macro

@check config begin
    # preparation code
end begin
    # measured code
end

Run a performance check from a PerfConfig.

This is equivalent to @check config.backend Dict(config) ..., but keeps the backend and options bundled in one Julia object for scripts, REPL sessions, and Pluto notebooks.

source

PerfChecker.advise Method

advise(bundle::RunBundle; min_samples=10)
advise(diagnosis::AbstractDict; bundles=RunBundle[])

Return perfchecker-advice/1 with evidence-linked recommendations, authority="advisory_only" and rules version. The bundle form first checks execution/correctness evidence, then considers sample sufficiency, empty CPU or allocation profiles and dominant allocation sites. Positive min_samples controls a sample-count recommendation; it is not a guarantee of tail accuracy.

measurement_summaries contains canonical measured quantities from only the supplied bundles, independently of whether a recommendation exists. Each record preserves its definition, unit, collector, bundle execution status, correctness status and its scope, and aggregation. record_count counts saved records: operation_measurement denotes operation measurements, while allocation sites, allocation-profile records, profile frames and interface captures are explicitly distinguished. Their minimum/median/maximum describe those records, not an operation cost or a count of independent repetitions. Independent operation totals remain separate. Compatibility aliases with the same definition and sample/table index count once. Distinct comparison keys remain separate via a fingerprint; their text is not copied. Conflicting correctness statuses at the same case/target are represented as not_checked with correctness_scope="ambiguous_case_target", rather than attributed to an arbitrary variant. Bundle execution status is not a verdict on an individual case. Direct manifest correctness has scope bundle; matching suite qualifications have scope case_target, or not_recorded if absent.

Only known definitions/scopes and bounded identifiers (letters, numbers and _.:@+-, at most 256 characters per identifier/segment) are projected. Canonical suite case IDs also allow exactly three such slash-separated segments (512 characters total). Absolute paths, Windows drives, backslashes, parent traversal and unknown quantities are omitted; caller-chosen identifiers can still contain sensitive text. Source, attributes, parameters, diagnostics and artifacts are not copied or read. Review identifiers before sharing the resulting report.

The diagnosis form requires schema perfchecker-diagnosis/1, interprets analyzer records and optionally incorporates saved bundle advice. Unsupported schemas or nonpositive sample policies raise ArgumentError. Neither method executes target code, contacts a provider or mutates files. Recommendations carry an unmeasured predicted gain; an empty list does not establish qualification.

source

PerfChecker.advisor_setup Method

advisor_setup(configuration; action=:probe, model="", confirmed=false,
              project=dirname(Base.active_project()), cancellation=CancellationToken())

Check a connection or discover available models/MCP tools without sending project evidence. Ollama :pull, :delete, and :unload require explicit confirmation and a loopback endpoint. Operations run in a cancellable worker. Model files are managed by the existing Ollama server, never bundled with PerfChecker. Accept an AdvisorConfig or draft dictionary. :validate returns validated configuration immediately without contacting a provider; :probe/:models allow an MCP discovery draft with no selected tool. Return a status dictionary, with worker results labelled perfchecker-advisor-setup/1, evidence_sent=false and generation_tested=false. MCP probes use the explicitly configured version: 2026 discovery reads server/discover before listing tools; 2025 uses initialize. The server result contains the selected protocol, advertised versions, bounded server identity and known capabilities. These are self-reported display metadata, not trusted authorization or a guarantee that a tool can generate advice. Missing provider packages can be unavailable; connection errors become diagnostic results. Invalid actions, model names or unconfirmed/nonlocal management raise ArgumentError before execution. Cancellation stops the worker; partial provider-owned downloads may remain.

source

PerfChecker.advisor_setup_transport Method

advisor_setup_transport(config, action, model)

Provider extension hook called in a setup worker by advisor_setup. The HTTP advisor extension implements connection probes, model/tool listings and authorized local Ollama management, returning a status dictionary. The base fallback throws ArgumentError when no provider method is available. Call the public worker API for timeout/cancellation and confirmation validation; direct invocation does not provide that lifecycle boundary.

source

PerfChecker.advisor_transport Method

advisor_transport(::Val{protocol}, config::AdvisorConfig, body::AbstractDict)

Provider hook called inside an isolated advisor worker. Implement a method for the selected protocol and return its Chat Completions-shaped response, preserving usage when available. The MCP text adapter instead supplies external_review. Transport methods own their network and response-shape handling; core validates the returned narrative references separately. The fallback throws ArgumentError when the protocol extension is unavailable. Call narrate_advice or chat_advice to get worker lifetime and cancellation handling rather than invoking this hook directly.

source

PerfChecker.agent_evidence Method

agent_evidence(diagnosis::AbstractDict; max_records=100)

Return a perfchecker-investigation-evidence/1 dictionary containing bounded records and recommendations, with advisory authority and a truncated flag. Accept diagnosis, advice, narrative and investigation schemas. Narrative cards retain unverified_narrative authority and deterministic fallback advice; investigation results retain status, experiments and external-review metadata. Positive max_records caps each list independently, not the total payload size. Invalid input schemas propagate an advice validation error. No target code, provider call, report write or code modification is performed.

source

PerfChecker.agent_evidence Function

agent_evidence(bundle::RunBundle, query=PerformanceQuery(); max_records=100)

Return a perfchecker-agent-evidence/1 dictionary with run identity, bundle pass status, selected diagnostic severity counts, deterministic advice and a query result. Require positive max_records; cap each query resource and the recommendation list at that value (or the query's smaller positive limit). The cap is per collection, not a total-record or byte budget.

Advice is derived from the complete saved bundle. No provider is contacted and no code is changed, workload rerun, report published or issue created. The envelope records this mutation policy for downstream consumers.

source

PerfChecker.bundle_dict Method

bundle_dict(bundle::RunBundle; include_records=true)

Return a dictionary containing schema, manifest and execution pass flag. By default also include definitions, observations, diagnostics and artifact descriptors; set include_records=false for metadata alone. Nested collections are retained, not deep-copied. No integrity verification, measurement or file export is performed.

source

PerfChecker.bundle_passed Method

bundle_passed(bundle::RunBundle) -> Bool

Return whether the manifest is complete and no diagnostic has error severity. This is an execution check, not a comparison against a performance budget. Use compare_bundles and comparison_verdict for a regression decision.

source

PerfChecker.cancel! Method

cancel!(token::CancellationToken) -> nothing

Atomically record a cancellation request. Repeated calls are harmless. Worker owners observe the request and perform their own shutdown and cleanup; this function does not itself kill a process or wait for cleanup. A remote advisor server may have its own interruption contract. Inspect the operation's returned status and diagnostics rather than treating this request as completed shutdown.

source

PerfChecker.cancel! Method

cancel!(job::InvestigationJob) -> nothing

Request interruption through this job's cancellation token. The call is idempotent and does not wait for its worker. Follow it with wait_investigation, then inspect status and error before restarting or removing workload-owned resources. Force-stopping a worker cannot guarantee an arbitrary user cleanup callback.

source

PerfChecker.cancel! Method

cancel!(job::SuiteJob) -> Bool

Request suite cancellation through the shared job API. Equivalent to cancel_suite!; wait for the task or wait_suite before closing its controller.

source

PerfChecker.cancel_suite! Method

cancel_suite!(job::SuiteJob) -> Bool

Request interruption of an active suite task and mark it as cancelling. Return false if the task already finished or cancellation was already requested. A true return acknowledges the request; use wait_suite or the job status to observe completed cancellation and cleanup.

source

PerfChecker.chat_advice Method

chat_advice(messages; config::AdvisorConfig, advice=nothing,
            project=dirname(Base.active_project()),
            cancellation=CancellationToken()) -> Dict

Ask a configured MCP advice tool a follow-up question without executing target code. Messages alternate user/assistant and end with a user question (at most 21 messages, 16000 characters each and 32000 total). Optional saved advice uses the same bounded evidence projection as narrate_advice. Replies remain unverified; no conversation is persisted by this API. MCP text mode is required.

advice=nothing attaches no report and supplies no measurement context. An explicit report with no findings is still an attachment; old reports with no usable summaries are identified as such rather than treated as missing data.

Each message is a dictionary with "role" and "content" strings. Supply the earlier assistant replies yourself for subsequent turns. advice defaults to an absent attachment, so configuration questions can be asked without saved measurements. The returned fallback then contains an empty perfchecker-advice/1 report. project needs an existing provider installation.

Return a perfchecker-narrative/1 dictionary with advisor_mode="advice", message_count, evidence IDs and the deterministic fallback. A complete MCP text reply appears in external_review with reference_status="unstructured_not_verified". Check status for complete, unavailable, error, timeout or cancelled; an interrupted call is not a verified response. This client prompts the advice tool to avoid edits, but its server's permissions and behavior remain under the caller's control.

Example request shape (requires your configured, running MCP server):

messages = [Dict("role" => "user", "content" => "How should I validate this change?")]
reply = chat_advice(messages; config, project="perf/controller", advice=saved_advice)
if get(reply, "status", "error") == "complete"
    push!(messages, Dict("role" => "assistant", "content" => reply["external_review"]))
    push!(messages, Dict("role" => "user", "content" => "Which empty-input case matters?"))
end

source

PerfChecker.check_preferences Method

check_preferences() -> Dict{Symbol,Any}

Read PerfChecker's persistent threads, repeat and quiet defaults through Preferences.jl in the current Julia environment stack. Return only settings that have a preference, as an independent dictionary; this does not write files or start workers. Malformed settings raise ArgumentError.

Checks read these settings at runtime, below explicit Dict/PerfConfig options and suite feature, variant and run overrides. A suite freezes them before its first worker is prepared. They are not compile-time switches, so changing them does not require restarting Julia. Preferences belong to the controller's active environment stack, not the workload directory in PerfConfig.path.

repeat controls the optional warmup used by allocation, profile and network collectors; BenchmarkTools and Chairmarks use their own sampling controls and do not use repeat. quiet suppresses PerfChecker's package-management logs, not arbitrary output printed by the workload.

source

PerfChecker.check_to_metadata_csv Method

check_to_metadata_csv(backend, pkg, version, tags; metadata="")

Compatibility wrapper for the legacy metadata format. New runs use structured metadata with result UUIDs and config hashes. Return (flattened_parameters, uuid_seed) for backend, package/version and tags. With a nonempty metadata path, append the parameters/UUID row unless it is already present, creating parents when needed. With the default empty path, no file is written. This is legacy cache metadata, not a measurement table or modern run bundle.

source

PerfChecker.checkres_to_boxplots Function

checkres_to_boxplots(result::CheckerResult, ::Val{backend}; kwarg=:times)

Load PerfCheckerMakie to return a figure of version distributions from a CheckerResult for Val(:benchmark) or Val(:chairmark). kwarg is the symbolic table column (default :times); values retain collector units. Missing columns or misaligned package/table data raise normal access errors. This reads saved samples without measuring, normalizing units or saving a file.

source

PerfChecker.checkres_to_pie Function

checkres_to_pie(result::CheckerResult, ::Val{backend}; kwargs...)

Load PerfCheckerMakie to create pie charts from a CheckerResult. For allocation checks this returns pairs mapping version labels to Makie figures. Forwards min_percentage and top to table_to_pie. Values can be nothing for empty tables. This reads saved allocation tables and package labels without running workloads or exporting figures; load a Makie backend for rendering.

source

PerfChecker.checkres_to_scatterlines Function

checkres_to_scatterlines(result::CheckerResult, ::Val{backend}; kwargs...)

Load PerfCheckerMakie and a backend to create an evolution figure from saved CheckerResult tables. Dispatch is explicit: use Val(:benchmark), Val(:chairmark) or Val(:alloc). Benchmark/Chairmarks views overlay version minima normalized per metric (minimum = 1); allocation views aggregate by file. Collector-specific keywords apply only to methods accepting them. Result tables must align with package specs; no workload is run or image saved.

source

PerfChecker.compare_bundles Method

compare_bundles(baseline::RunBundle, candidate::RunBundle;
    relative_limits=Dict{String,Float64}(), min_samples=1,
    sample_statistics=Dict{String,Symbol}(), default_sample_statistic=:median)
    -> BundleComparison

Compare existing observations with matching case, target, comparison key, metric and measurement definition. Definitions and units must match exactly; no unit conversion or new measurement is performed. Missing observations, incompatible definitions and too few numeric samples remain explicit records.

sample_statistics maps metric names (string or symbol keys) to :median, :mean, :minimum, :maximum, :p95 or :p99; the default applies to other metrics. Percentiles use linear R7 interpolation. min_samples must be positive. relative_limits maps metric strings to nonnegative fractions: for example, 0.05 permits a 5% regression in the definition's preference direction. Metrics without a limit are diagnostic, not passing CI assertions. A zero baseline is checked by direction rather than by division.

Different runtime languages, operating systems, architectures, thread counts or recorded hardware identities make environments incomparable. Runtime version and dependency fingerprints can differ with warnings. Input bundle failure still makes the overall verdict :invalid_inputs.

baseline = read_run_bundle("reports/baseline")
candidate = read_run_bundle("reports/candidate")
comparison = compare_bundles(baseline, candidate;
    relative_limits=Dict("julia.wall.time" => 0.05), min_samples=3)
comparison_verdict(comparison)

source

PerfChecker.compare_scenarios Method

compare_scenarios(baselines::AbstractVector{RunBundle},
                  candidates::AbstractVector{RunBundle}; kwargs...) -> Dict

Join saved bundles by (scenario id, implementation, collector) and compare each matched pair with compare_bundles. Forward comparison keywords such as relative_limits, min_samples and sample statistics; units and measurement definitions must remain compatible. Duplicate identities on either side are errors. Unmatched configurations stay "not_tested", rather than being counted as improvements or silently dropped.

Return perfchecker-scenario-comparison/1 with a configurations array containing each identity, status and matched comparison evidence. This only compares existing evidence; it does not run workloads, install versions or change source.

source

PerfChecker.compare_suite_versions Method

compare_suite_versions(bundle::RunBundle;
    relative_limits=Dict{String,Float64}(), min_samples=1,
    sample_statistics=Dict{String,Symbol}(), default_sample_statistic=:median)
    -> VersionComparison

Reduce a saved suite bundle into version series and performance comparisons. By default, compare adjacent releases and compare development/Git targets to the latest release. Explicit ComparisonPolicy declarations carried by the bundle instead select named or grouped reference targets. Intentionally unavailable targets remain in availability evidence but are excluded from the comparison domain; failed or missing measurements remain visible.

relative_limits uses metric-string keys and nonnegative fractional budgets. min_samples must be positive. sample_statistics chooses a statistic per metric (string or symbol key), with :median, :mean, :minimum, :maximum, :p95 and :p99 supported. Reference aggregation combines the chosen statistic across baseline targets; it is distinct from reducing samples within one target. Without a budget, records are diagnostic and the verdict is inconclusive.

The returned object retains availability, warnings, plot series and all comparison records. This performs no measurements and does not write reports.

bundle = read_run_bundle("reports/bundles/run-example")
comparison = compare_suite_versions(bundle;
    relative_limits=Dict("julia.wall.time" => 0.05), min_samples=3)
write_version_comparison_json(comparison, "reports/version-comparison.json")

source

PerfChecker.comparison_dict Method

comparison_dict(comparison::BundleComparison) -> Dict{String,Any}

Return schema perfchecker-comparison/1, input run IDs and validity, environment status, warnings, passed/verdict, and per-metric records. This serializes the decision already present in memory; it does not reread bundles, compare new observations or write a report.

source

PerfChecker.comparison_passed Method

comparison_passed(comparison::BundleComparison) -> Bool

Return true only when comparison_verdict is :qualified. Diagnostic, missing, incomparable or insufficient-sample records cannot satisfy this predicate.

source

PerfChecker.comparison_policy_dict Method

comparison_policy_dict(policy::ComparisonPolicy) -> Dict{String,Any}

Return the policy ID, package/feature/comparison-key selectors, baseline and candidate labels, and string aggregation. It declares a reference selection, not metric limits or a measured verdict, and performs no comparison or I/O.

source

PerfChecker.comparison_verdict Method

comparison_verdict(comparison::BundleComparison) -> Symbol

Return :invalid_inputs, :incomparable, :regressed, :qualified or :inconclusive. Qualification requires valid inputs, comparable environments, at least one comparison record and a passing status for every record. The checks follow that priority order; warnings alone do not fail an otherwise qualified comparison.

source

PerfChecker.compatibility_report_dict Method

compatibility_report_dict(report::CompatibilityReport)

Return perfchecker-compatibility-report/1 with suite/profile, check time, preflight pass flag, counts for known/present statuses and diagnostic records. The records are retained in memory; no resolution, probe or file write is repeated.

source

PerfChecker.configure_suite_repl Method

configure_suite_repl(suite; profile=:historical, input=stdin, output=stdout)

Prompt for package, workload, collector, version bounds, search and sort order, then print the selected plan. Read positive sample, duration and worker-thread settings and return (plan, overrides) after confirmation. This prepares a run; it does not execute workloads or write reports. Pass the returned values to run_suite_repl.

Empty selectors include all matching runs; an empty numeric answer uses the displayed default. Reject an empty selection or invalid settings with ArgumentError; declining confirmation throws InterruptException. input and output may be redirected for a reproducible scripted dialogue.

source

PerfChecker.csv_to_table Method

csv_to_table(path::String) -> Table

Read a CSV file into a TypedTables.Table, using CSV's default column type inference. Return the table; file/parser errors propagate. CSV does not retain run-bundle provenance, qualification or measurement definitions, so loading a table alone does not establish comparison compatibility.

source

PerfChecker.dependency_evidence Method

dependency_evidence(; phase="snapshot", hash_libraries=false)

Return NativeDependencyEvidence combining active Pkg inventory, declared artifact bindings and currently loaded libraries. phase is an arbitrary caller label; hash_libraries=true opts into reading library bytes. Artifact binding files are always read and hashed when present; platform selection and runtime overrides prevent interpreting declarations as loads. Warnings explicitly record that child processes and services are outside scope. No dependency is installed and no report is written.

source

PerfChecker.dependency_evidence_dict Method

dependency_evidence_dict(evidence::NativeDependencyEvidence)

Return perfchecker-dependency-evidence/1 with the capture phase, package, artifact/library records, warnings and explicit coverage flags. captured_at is the UTC conversion time; it is not a retained timestamp of every observation. Nested records are retained and no new inventory or file export is performed.

source

PerfChecker.diagnose Method

diagnose(catalog::ScenarioCatalog; project=catalog.root,
         tools=[:jet, :aqua, :alloccheck, :snoopcompile, :latency],
         timeout=120, threads=1, cancellation=CancellationToken(),
         options=Dict(), reports=nothing) -> Dict{String, Any}

Run the selected optional analyzers in separate bounded Julia workers. Each non-Aqua analyzer operates on each declared scenario; Aqua runs once for the package identified by the catalog root's Project.toml. Reject unknown tool names. Required packages must already exist in project; a missing dependency produces unavailable evidence rather than installing it or passing the check.

timeout bounds each worker including startup and threads sets its Julia thread count. options supplies analyzer-specific data; reports supplies artifact paths, notably for the redacted heap snapshot. Source/fixture and environment fingerprints invalidate results if those inputs change during analysis.

Return perfchecker-diagnosis/1 with source_provenance and a records array. Each record carries its tool, scenario/implementation, configuration, input fingerprints, summary and outcome. Execution status, correctness, quality and performance answer different questions: a completed Aqua analysis can report failed package quality without verifying a scenario's correctness. Cancellation stops further jobs after the current worker's final cleanup.

catalog = load_scenario_catalog("perf/scenarios.toml")
report = diagnose(catalog; project=".", tools=[:jet, :latency], timeout=120)
[(r["tool"], r["status"]) for r in report["records"]]

source

PerfChecker.diagnostic_capabilities Method

diagnostic_capabilities() -> Vector{Dict}

Describe each supported scenario analyzer's name, package, evidence scope, compatible package versions and installation scope. The catalogue includes JET, Aqua, AllocCheck, SnoopCompile and built-in latency, GC, memory, heap and lock diagnostics. Reading it neither imports those packages nor executes target code.

Catalogue presence is not availability or a passing diagnosis. Analyzer packages must exist in the selected worker environment; heap snapshots need a report directory and lock contention needs Julia 1.11 or newer. Consult each record's scope, then inspect the actual diagnose result for that configuration.

source

PerfChecker.discover Function

discover(root=pwd(); previous=nothing)

Statically inspect a directory's supported test files, scenario catalogues, fixtures and literal CI configurations. Return perfchecker-discovery/1 with test-derived candidates, declared scenarios, CI candidates, fixture/corpus metadata, fingerprints, warnings and file changes. Proposals require explicit adoption before execution; parsed CI environments are not qualification.

previous may be a prior discovery dictionary or JSON path, used to report added/removed/changed fingerprints and affected scenarios. A missing root or invalid baseline schema raises ArgumentError; unsupported constructs and catalogue parsing problems are recorded as warnings where possible. Files are read and hashed, but target Julia code and CI commands are not run. No scenarios, workflows or reports are written by discovery.

source

PerfChecker.discover_testitems Function

discover_testitems(root=pwd(); mode=:performance, tags=[], exclude_tags=[])

List existing @testitem declarations without executing their bodies, setup modules, or surrounding source code. Load TestItemRunner to enable this method.

mode=:performance includes ordinary items and :check_only/:perf_only, but excludes :test_only. mode=:test does the converse; :all keeps both categories. tags matches any requested tag; exclude_tags always removes matching items. An item carrying both reserved tags is rejected.

Return a dictionary with schema perfchecker-testitems/1, normalized root, selection mode, executed=false, and an items array. Each item contains its stable ID, name, relative file, tags and source SHA-256. IDs identify the file/name pair; renaming either changes the ID. Duplicate names in one file are rejected.

Discovery writes no report and launches no measured workload. Missing directories, invalid declarations and ambiguous identities raise an error rather than silently producing a partial listing. See run_testitems to execute a selection.

source

PerfChecker.documenter_makedocs Function

documenter_makedocs(bundle; root=pwd(), source="src", build="build",
                    page="performance.md", sitename="Performance", kwargs...)

Load Documenter. Generate a performance page from saved evidence and invoke Documenter.makedocs, forwarding additional options. This writes documentation outputs but does not deploy them. For VitePress, use the corresponding extension.

source

PerfChecker.documenter_page Function

documenter_page(bundle_or_result, destination; title="Performance",
                blocks=PerformanceDocumentBlock[], config=nothing)

Load Documenter and write a Markdown report page from existing evidence. config selects blocks from a configuration file; without blocks, include the version comparison summary. Create parent directories, replace the destination and return its absolute path. This renders saved data without running workloads.

source

PerfChecker.documenter_vitepress_makedocs Function

documenter_vitepress_makedocs(bundle; repo, devbranch="main",
                              devurl="dev", deploy_url=nothing, kwargs...)

Load Documenter and DocumenterVitepress. Configure the VitePress formatter and build a report site through documenter_makedocs. repo is required. This is a report-publishing integration for user projects, separate from PerfChecker's own website and its collection qualification gate.

source

PerfChecker.drwatson_parameters Function

drwatson_parameters(plan::SuitePlan) -> Vector{Dict{String,Any}}
drwatson_parameters(suite::SoftwareSuite; profile=:quick, kwargs...)

Load DrWatson to enable these methods. Return one parameter dictionary per planned row with suite, package, feature, version, comparison key and planned status. The suite overload calls plan_suite, forwarding keywords; registry/project inspection can occur but no measurement is started.

source

PerfChecker.drwatson_produce_or_load Function

drwatson_produce_or_load(producer, parameters; directory="",
                        force=false, tag=true, kwargs...)

Load DrWatson to enable its disk-cache workflow. Delegate cache lookup and writing to DrWatson.produce_or_load; wrap a non-dictionary producer result under "result". Return DrWatson's result. The producer executes only when its cache policy requires it; cached data is not fresh measurement evidence.

source

PerfChecker.drwatson_run_suite Function

drwatson_run_suite(suite; profile=:quick, directory="", force=false,
                  tag=true, kwargs...)

Load DrWatson. Plan a suite, use its plan revision as a cache parameter and run it with reports when production is required. Return the DrWatson cache result. Existing artifacts may be reused: request force=true when fresh execution is required, and inspect the saved provenance before comparison.

source

PerfChecker.drwatson_savename Function

drwatson_savename(run::PlannedFeatureRun; suffix="jld2") -> String

Load DrWatson to enable this method. Delegate filename generation to DrWatson.savename using suite, package, feature and target-version parameters. suffix chooses the extension. No directory/file is created and no source or dependency fingerprint is included in this filename.

source

PerfChecker.estimate_performance Method

estimate_performance(target, references, workload; neighbours=3,
                     min_calibrations=3, max_log_error=0.25)

Experimental calibrated nearest-neighbour transfer. Each record contains machine, context, calibration (workload => positive seconds) and measurements. Context must explicitly identify suite_revision, environment, measurement, unit, and resource_policy; only equal contexts and OS/architecture are compared. Fit a log time ratio on shared calibration workloads. Leave-one-workload-out error rejects unstable transfers. Return a donor spread/error envelope, NOT a confidence interval or a measured result. At least two independent machine labels are required. Use only to prioritize actual measurements, never to qualify a CI performance gate.

source

PerfChecker.evaluate_advisors Method

evaluate_advisors(cases; config=nothing, project=dirname(Base.active_project()),
                  cancellation=CancellationToken(), include_investigator=false,
                  max_experiments=4, budget_seconds=300, samples=10,
                  tools=[:jet, :alloccheck, :latency])

Evaluate each case's supplied id, advice and expected_rules. Always score deterministic evidence selection. With config, also ask the model writer to select narrative evidence; include_investigator=true runs declared experiments from each case's required catalog path, forwarding the count, time, sample and analyzer limits to investigate.

Return perfchecker-advisor-evaluation/1 with per-case/mode selected rules, true/false positives, false negatives, elapsed time and optional narrative/ investigation payloads. Cancellation stops later cases/modes. Malformed cases and execution errors propagate. Prose truth still requires human review and monetary cost is not measured; this is rule selection evaluation, not automatic qualification of recommendations. Reports are not written by this function.

source

PerfChecker.evaluate_resource_envelope Method

evaluate_resource_envelope(envelope; upper_limits=Dict{Symbol,Float64}(),
                          require_process=false, require_external=false,
                          require_external_balance=false)

Evaluate saved metrics. upper_limits maps metric symbols/strings to finite nonnegative limits; equality passes. Unknown metrics or invalid limits raise ArgumentError. Required missing process/external snapshots are unavailable. Balance requires known allocation/free deltas to be equal and live bytes not to grow; reserved pools need not shrink.

Return perfchecker-resource-policy-evaluation/1 with metrics, normalized policy, violations, unavailable reasons and status. failed takes precedence over unavailable, otherwise the status is passed. Missing required metrics cannot pass. Use resource_policy_passed to decide a gate.

source

PerfChecker.external_command_dict Method

external_command_dict(spec::ExternalCommandSpec)

Return provider identity, language, command arguments, directory, sorted environment names and timeout. Environment values are deliberately omitted; command arguments can still contain caller-supplied sensitive text. This representation describes configuration without executing or writing it.

source

PerfChecker.external_memory_snapshot Method

external_memory_snapshot(raw; provider="workload")

Validate a dictionary or named tuple declaring schema_version="perfchecker-external-memory/1". Return an ExternalMemorySnapshot; existing snapshots are returned unchanged, and nothing means unavailable. Observed results need one nonnegative integer counter among live_bytes, reserved_bytes, allocated_bytes_total and freed_bytes_total. Missing counters remain unknown.

Statuses observed, available, pass, passed, ok normalize to :observed; unavailable, unsupported, missing normalize to :unavailable. timestamp_ns defaults to time_ns(); an explicit provider overrides the keyword. Invalid schema, status, counters or timestamp raise ArgumentError. This does not invoke the probe or free memory.

julia> external = external_memory_snapshot((schema_version="perfchecker-external-memory/1", live_bytes=128));

julia> external.live_bytes == 128
true

source

PerfChecker.external_memory_snapshot_dict Method

external_memory_snapshot_dict(snapshot::ExternalMemorySnapshot)

Return the schema, timestamp, optional byte gauges/cumulative totals, provider, string status and message as a dictionary. Unknown counters remain nothing; the workload's probe is not invoked again.

source

PerfChecker.filter_suite_plan Method

filter_suite_plan(plan; packages=nothing, features=nothing, backends=nothing,
                  statuses=nothing, from_version=nothing, to_version=nothing,
                  search="", sort=:package_version, include_unavailable=true)

Return a new SuitePlan containing selected runs and the original suite, profile and comparison policies. Selectors accept one value or a collection; nothing leaves that dimension unrestricted. Package matching and free-text search ignore case; features match either feature ID or workload ID. Version bounds are inclusive compatibility versions, accepting VersionNumber or version strings (including a dev@ prefix). include_unavailable=false keeps only :ready runs.

Sort modes are :package_version, :version_package, :feature_version, :version_desc and :plan (retain plan order). Reversed bounds and unknown sort modes raise ArgumentError when evaluated. Filtering can return an empty plan and does not replan targets, start workers or write evidence.

source

PerfChecker.find_by_tags Method

find_by_tags(tags::Vector{Symbol}, results; exact_match=true)

Find results whose tags match tags.

With exact_match=true, tags must match exactly. With exact_match=false, any overlap is accepted. Exact matching compares vectors in order, including duplicates; it is not set equality. Missing result tags are treated as empty. Return matching integer indices; a single CheckerResult returns [1] or Int[], while a result vector returns all matching positions. No result is modified and no saved cache is searched.

source

PerfChecker.freeze_propcheck_corpus Function

freeze_propcheck_corpus(path, generator; count=100, seed=0,
                        encode=identity, metadata=Dict(), force=false)

Load PropCheck to enable this extension. Generate cases with a seeded Xoshiro RNG, extract tree roots, apply encode and save a frozen JSON corpus. Return its absolute path. count must be positive; overwriting requires force=true. Generation happens now, outside later measurement workers.

source

PerfChecker.freeze_supposition_corpus Method

freeze_supposition_corpus(path, possibility; count=100, tries=100_000,
                          encode=identity, metadata=Dict(), force=false)

Load Supposition to enable this extension. Sample the possibility, encode the cases as JSON-compatible values and persist them for later reproducible replay. Return the corpus path; a positive count is required and existing files need force=true. Replay uses saved inputs rather than resampling the generator.

Generation with the published Supposition backend requires 64-bit Julia. On 32-bit Julia, use freeze_propcheck_corpus or replay a saved corpus with read_property_corpus. Loading Supposition itself on 32-bit Julia can fail before this API is reached because of an upstream dependency defect.

source

PerfChecker.get_versions Function

get_versions((name, selector, versions, prefer_latest), regname=nothing)
get_versions(spec::PackageVersionSpec, regname=nothing)

Return (name, selected_versions) using installed registry metadata and legacy selectors :custom, :patches, :minor, :major or :breaking. regname is an optional vector of registry names restricting lookup. Custom selection returns requested versions; grouped selection applies its grouping to each requested version, optionally choosing earliest/latest representatives. The flattened result can contain duplicates from overlapping requested groups. Unknown selectors raise an error. Registry lookup does not install or measure selected targets; prefer suite planning for explicit availability evidence.

source

PerfChecker.implement_advice Method

implement_advice(messages; config::AdvisorConfig, workspace::AbstractString,
                 workspace_argument="workspace", advice=nothing,
                 project=dirname(Base.active_project()),
                 cancellation=CancellationToken()) -> Dict

Invoke an explicitly selected MCP implementation tool on a caller-owned isolated checkout. The caller must checkpoint the original code and review/apply the resulting diff, including after cancellation or failure: the server might already have edited the checkout. This function does not apply changes or create a backup. The tool receives an absolute workspace path in addition to bounded conversation and advice. MCP and the prompt do not enforce a filesystem sandbox; configure a trusted tool that confines its operations to this checkout and can access its filesystem. Only the workspace path is transmitted, not its files. A remote server needs separately configured access to that isolated copy; HTTP does not provide a mount or upload the checkout.

messages follows chat_advice's alternating conversation contract. workspace must exist; its resolved absolute path is sent under workspace_argument. That argument must differ from the prompt argument and must not already be supplied in config.mcp_arguments. Configure a distinct implementation tool and explicitly authorize its access to the isolated copy.

Return the same narrative/status fields as chat_advice, plus advisor_mode="implementation" and implementation_status="requires_diff_review". The latter is a review obligation, not a success verdict: inspect status, the actual diff and correctness results even when the provider returned a complete reply. Cancellation stops the local worker; an external server may continue work. The caller owns cleanup and recovery of its isolated checkout and original code.

# isolated_checkout was prepared and checkpointed by the caller.
proposal = implement_advice(messages; config=implementation_config,
    workspace=isolated_checkout, project="perf/controller", advice=saved_advice)
# Review the on-disk diff and run the agreed oracle before applying anything.

source

PerfChecker.investigate Method

investigate(catalog; project=catalog.root, samples=10,
            tools=[:jet, :alloccheck, :latency], max_experiments=4,
            budget_seconds=300, timeout=120, threads=1, advisor=nothing,
            cancellation=CancellationToken(), reports=nothing)

Run a bounded menu of declared scenario collectors and known analyzers. Prefer timing collectors, then JET, before secondary experiments. Require a nonempty catalogue, 1–100 experiments, finite positive budget up to 86,400 seconds, known tools and at most 128 proposed experiments. Invalid selections raise ArgumentError; runner errors can propagate.

Return perfchecker-investigation/1 with attempted/unexecuted experiments, runs, diagnosis records, deterministic advice, decisions and elapsed time. Status may be complete, cancelled, budget_exhausted or advisor_stopped. Timeouts are capped by remaining budget; count/budget exhaustion can leave experiments unexecuted. Cancellation follows worker termination contracts.

advisor optionally supplies an AdvisorConfig for bounded menu selection; its configured instructions and provider settings are preserved while its timeout is capped by the remaining budget. Provider output remains advisory. reports writes per-experiment evidence and a final report. The controller environment must already contain needed tools. This explicitly executes declared code but never modifies it, compares a performance baseline or certifies the advice as an improvement.

source

PerfChecker.investigate_julia_regressions Method

investigate_julia_regressions(campaign::JuliaRuntimeCampaign;
                             max_frames=50, obvious_share=0.6)
    -> JuliaRegressionInvestigation

Read the campaign's referenced bundles and rank positive source-line deltas. Only numeric observations with source_file and source_line attributes contribute. Values are summed for matching comparison key, metric, source location and definition, then candidate-minus-baseline increases are ranked. max_frames must be positive and limits returned rows per comparison.

obvious_share must lie in (0, 1]. obvious_candidate=true means the largest row belongs to a Julia Base/stdlib/compiler path and accounts for at least that fraction of all positive deltas. It is a heuristic, not proof of causation; ranked rows can contain different metrics and units, so their combined shares are not a unit-normalized performance budget.

Missing bundles produce "unavailable" records with failure details; no source deltas produce "no_source_evidence"; other records are "ranked". Existing bundle read/integrity errors propagate. No campaign is rerun, file modified, minimal reproduction generated or report written.

source

PerfChecker.investigation_status Method

investigation_status(job::InvestigationJob; include_result=true) -> Dict

Read a lock-protected perfchecker-investigation-job/1 snapshot with the job ID, action, status, elapsed seconds and captured error. Include available result and advice unless include_result=false. This operation neither waits nor executes new work; "running" is not a final outcome.

source

PerfChecker.investigation_view Method

investigation_view(payload::AbstractDict) -> InvestigationView

Validate an investigation/report schema and wrap a string-keyed copy for REPL or HTML display. Supported inputs include discovery, scenario runs/comparisons, diagnosis, deterministic advice, tool catalogues, narratives, bounded investigations, advisor evaluations and scenario sync reports. An unsupported schema is an error. The wrapper renders saved data and retains unavailable, unverified and incomplete statuses; it does not start new work.

source

PerfChecker.isolated_network_result_dict Method

isolated_network_result_dict(result; include_output=false, max_output_chars=16_384)

Return perfchecker-isolated-network-result/1 with provider, command, exit code, lifecycle phase and sample. Stdout/stderr default to empty strings; opt in with include_output=true to retain at most max_output_chars characters per stream. A negative bound raises ArgumentError. Output can contain workload-sensitive text; this conversion does not redact it or write a file.

source

PerfChecker.julia_investigation_dict Method

julia_investigation_dict(investigation::JuliaRegressionInvestigation)
    -> Dict{String,Any}

Return schema perfchecker-julia-investigation/1, campaign ID, generation timestamp and comparison attribution records. This preserves unavailable and no-source-evidence classifications and does not reread or execute a campaign.

source

PerfChecker.julia_runtime_campaign_dict Method

julia_runtime_campaign_dict(campaign::JuliaRuntimeCampaign;
                            include_output=false) -> Dict{String,Any}

Return schema perfchecker-julia-runtime-campaign/1, campaign identity, timestamps, baseline ID, passing predicate, runs and comparisons. By default, captured stdout and stderr are replaced by empty strings in copied run records; set include_output=true to retain them. The campaign is not mutated or rerun and no report is written.

source

PerfChecker.julia_runtime_command Method

julia_runtime_command(spec::JuliaRuntimeSpec; project=nothing,
                      startup_file=false, history_file=false,
                      extra_arguments=String[]) -> Vector{String}

Build command arguments without executing them. Append the Juliaup selector when required, spec.arguments, startup/history flags, an absolute --project path when supplied, then extra_arguments. Startup and REPL history files are disabled by default; the command still uses the caller's environment and Julia depot.

spec = JuliaRuntimeSpec(:lts, "1.10"; role=:baseline)
command = julia_runtime_command(spec; project=pwd(),
    extra_arguments=["-e", "println(VERSION)"])
# Execute explicitly when desired: run(Cmd(command))

source

PerfChecker.julia_runtime_matrix Method

julia_runtime_matrix(; baseline="release", candidates=["rc", "nightly"])
    -> Vector{JuliaRuntimeSpec}

Create a Juliaup runtime axis with baseline ID :baseline and candidate IDs :candidate_1, :candidate_2, and so on, preserving candidate order. The selectors must already be usable when the campaign executes; this function does not install or probe channels. Override candidates to use explicit installed versions instead of moving aliases.

julia_runtime_matrix(baseline="1.10", candidates=["1.12", "1.13"])

source

PerfChecker.julia_runtime_spec_dict Method

julia_runtime_spec_dict(spec::JuliaRuntimeSpec) -> Dict{String,Any}

Return a perfchecker-julia-runtime-spec/1 declaration containing id, selector, role, source, executable and arguments. Symbols become strings. This does not resolve the selector to an installed runtime; use probe_julia_runtime for an observed version and commit.

source

PerfChecker.julia_runtime_suite_command Method

julia_runtime_suite_command(spec::JuliaRuntimeSpec; suite, reports,
    profile=:ci, factory=:build_suite,
    perfchecker_project=dirname(@__DIR__),
    controller_project=dirname(abspath(suite)), backend_packages=String[])
    -> Vector{String}

Build a child-controller invocation of bin/perfchecker-runtime-suite.jl for spec. suite is a Julia suite definition; reports is its output directory. perfchecker_project must contain that CLI, and controller_project must contain Project.toml. Paths become absolute. The factory/profile and each requested backend package are passed to the CLI.

This validates local input files but does not start Julia, prepare dependencies or create reports. The runtime campaign constructs separate controller projects for its runtimes; callers using this lower-level function supply their own.

source

PerfChecker.launch_advisor_setup Method

launch_advisor_setup(input; kwargs...)

Start advisor_setup asynchronously and return an InvestigationJob with action :advisor_setup. Forward setup keywords and use the job's own cancellation token. Inspect with investigation_status, wait with wait_investigation, or request cancellation with cancel!. Setup errors are retained in the job as status=:error; successful worker status/result is copied to the job. Calling this explicitly starts setup; constructing or displaying an investigation view does not.

source

PerfChecker.launch_investigation Method

launch_investigation(action::Symbol; root=pwd(), catalog=nothing, project=root,
    tools=[:jet, :aqua, :alloccheck, :snoopcompile, :latency], samples=10,
    timeout=120, threads=1, reports=nothing, previous=nothing, advisor=nothing,
    evidence=nothing, max_experiments=4, budget_seconds=300) -> InvestigationJob

Explicitly start one asynchronous :discover, :run, :diagnose, :sync, :tools, :narrate or :investigate action. :run, :diagnose and :investigate require a nonempty declared catalog; :narrate requires saved deterministic evidence. Other actions inspect declarations or the tool catalogue without executing a workload. Forward the selected project, tools and limits to the underlying operation; construction of a view alone does not launch this job.

Return immediately with a running handle. The task records :complete, :cancelled or :error, retains its result and advice when available, and captures exceptions in error. With reports, write JSON/Markdown investigation reports and the action's run artifacts. Use wait_investigation to inspect the final outcome; the asynchronous API does not rethrow a recorded workload error.

job = launch_investigation(:diagnose; catalog, project=".", tools=[:latency])
snapshot = investigation_status(job; include_result=false)
final = wait_investigation(job)

source

PerfChecker.launch_pluto_dashboard Function

launch_pluto_dashboard(path; kwargs...)

Load PerfCheckerPluto and open an existing generated notebook with Pluto.run. Reject a missing file and forward server options to Pluto. This starts a notebook server; generating a notebook alone uses prepare_pluto_dashboard instead.

source

PerfChecker.launch_suite Method

launch_suite(plan::SuitePlan; overrides=Dict{Symbol,Any}(),
             executor=_default_suite_executor,
             progress_callback=payload -> nothing) -> SuiteJob
launch_suite(suite::SoftwareSuite; profile=:quick, kwargs...) -> SuiteJob

Start an asynchronous controller task which executes the selected plan. The normal executor prepares isolated environments and starts measurement workers. Return immediately with a job handle. Poll suite_job_progress for progress, use cancel_suite! to request interruption and wait_suite to obtain the result or surface a failure. overrides has symbol keys and takes precedence over feature and variant options. Progress callbacks receive counts and active-run identity; exceptions in those callbacks are logged.

The normal executor requires the relevant backend packages in the controller, copies/prepares worker environments, can resolve/download target dependencies, and includes the workload only in measurement workers. A custom executor(planned, config, setup, workload) replaces that measurement path and is responsible for its own behavior. The suite overload first calls plan_suite with profile and version_provider; additional keywords go to the plan executor.

source

PerfChecker.list_run_bundles Method

list_run_bundles(root; recursive=false)

Read manifests in child bundle directories and return them with bundle_path. A missing root returns an empty vector. Unreadable or foreign manifests are ignored. This is discovery only: call read_run_bundle to verify bundle integrity before using its observations.

source

PerfChecker.load_advisor_config Method

load_advisor_config(path::AbstractString) -> AdvisorConfig

Read a JSON object whose keys are the keyword arguments of AdvisorConfig and validate it with that constructor. Unknown keywords or invalid connection and size limits are rejected. Reading the file does not contact the provider. Store only the name of a credential variable in api_key_env; the configuration must not contain the credential itself.

source

PerfChecker.load_scenario_catalog Method

load_scenario_catalog(path::AbstractString) -> ScenarioCatalog

Parse a TOML file with schema perfchecker-scenario-catalog/1. Resolve root, scenario source and fixture paths against the catalog file's directory, then validate each ScenarioSpec. Missing files, unsupported collectors and duplicate identities are errors. Parsing does not include package or test source, run the factory, or install its dependencies. Prepare the selected worker project before calling run_scenarios or diagnose.

source

PerfChecker.load_software_suite Method

load_software_suite(path; factory=:build_suite) -> SoftwareSuite

Load an ordinary Julia suite definition in an isolated controller module. The file must define the selected zero-argument factory or bind suite to a SoftwareSuite. Only the controller evaluates this file; measured Malt workers still load just their backend, package, and feature entrypoint. The path becomes absolute and must exist. Including a Julia definition executes its top-level code, so only load trusted files. A defined factory takes precedence over the suite binding; wrong return types and load/factory errors are raised.

source

PerfChecker.loaded_library_inventory Method

loaded_library_inventory(; digest=false)

Return (libraries, warnings) for unique Libdl.dllist() entries, sorted by path. Each record includes absolute path, filename, existence, optional size, architecture and optional SHA-256. With digest=true, read existing library files to hash them; failed hashes remain nothing. Inspection failures become bounded warnings. Paths may reveal local installation information. This does not load libraries or inspect symbols, child processes or native heaps.

source

PerfChecker.machine_profile Method

machine_profile(; label="", limits=Dict())

Capture a specification fingerprint, independent of host name and transient CPU load. This identifies a specification class, not a unique physical machine. Declare CI CPU quotas, affinity and memory limits in limits; unknown limits remain unknown. Return perfchecker-machine-profile/1 with specs, SHA-256 spec_id, label, Julia version and limits_verified=false. Limits are caller declarations, not measured enforcement. This takes a local hardware snapshot without running a calibration workload or writing a file. The native snapshot records Hwloc's physical/logical topology, falling back to Julia's OS logical CPU count if topology is unavailable. Zero physical cores or SIMD bytes mean unknown; similarity excludes unknown capabilities. Fresh hardware identities and cache keys can differ from older CPUID-based snapshots, including on x86; saved historical snapshots are not rewritten.

source

PerfChecker.measure_isolated_network_command Method

measure_isolated_network_command(command; spec=NetworkIsolationSpec(),
                                directory=pwd(), environment=Dict(),
                                timeout_seconds=300, strict=false)

Execute an argument vector through PerfChecker's Linux/WSL2 namespace wrapper and return an IsolatedNetworkCommandResult with sample, exit code, stdout, stderr and counter provider. The selected environment must already supply the namespace tools. Supplied environment names must be portable identifiers; WSL2 names are added to WSLENV. Capture covers command process lifecycle and its owned descendants, including startup/shutdown overhead.

Provider launch/capture/exit-mismatch errors and excessive shutdown grace raise errors. With strict=true, a nonzero captured exit also raises; otherwise return it for inspection. Owned processes and temporary captures are cleaned on exit, while command-created project files remain caller-owned. No report is written and command success alone does not prove correctness.

source

PerfChecker.measure_network_interface Method

measure_network_interface(workload::Function; interface="auto", repetitions=1)

Execute the zero-argument workload once per positive integer repetition and return a vector of network sample named tuples. Select the interface before the first call and reuse its explicit name for later samples. Each sample includes counter deltas and workload wall time; workload return values are discarded. State preparation, correctness and cleanup remain caller-owned. Workload and provider errors propagate, with no retry or rollback.

All other traffic on the interface is included. Use an isolated network collector for process-tree attribution. This function writes no reports.

source

PerfChecker.measure_network_isolated Method

measure_network_isolated(workload::Function; interface="lo", repetitions=1)

Run the zero-argument workload inside an already launched PerfChecker Linux namespace and return sample named tuples. Require its isolation marker, Linux, and positive repetitions, otherwise throw ArgumentError. Capture namespace counters around each call with workload wall time and isolated-worker-group attribution. Return values are discarded, and workload/provider errors propagate. This function does not itself create a namespace, prepare fresh state, check correctness or write reports; the launcher and workload own those steps.

source

PerfChecker.migrate_run_bundle Method

migrate_run_bundle(source, destination)

Read source with integrity checking disabled and save its records to a new destination using write_run_bundle. Return the absolute destination; source files are unchanged and existing destinations are rejected. This adds digest protection to the records now read; it cannot retroactively prove a legacy bundle's authenticity. Artifact payloads are not copied. Parsing and writing errors propagate.

source

PerfChecker.narrate_advice Method

narrate_advice(advice::AbstractDict; config=AdvisorConfig(),
               project=dirname(Base.active_project()),
               cancellation=CancellationToken(), experiments=Dict[]) -> Dict

Request optional prose about recorded evidence in a separate Julia worker. advice must be a perfchecker-advice/1 report. Canonical measurement summaries and unique recommendation IDs share one exact serialized-array limit, config.max_evidence_chars; measurements are considered first, with whole records omitted when they do not fit. Measurements retain explicit units, collector, record semantics, aggregation and qualification status, without becoming findings. No source, credentials or raw logs are read automatically. Caller-supplied identifiers and recommendation hypothesis/action/validation strings are retained; review them for sensitive text before sending. project must already contain the chosen provider package. Old reports without summaries remain supported.

The returned perfchecker-narrative/1 dictionary records status, cards, evidence_ids, evidence_truncated, timing and the deterministic fallback. It labels prose authority="unverified_narrative": reference validation does not establish semantic truth. Empty evidence and no experiments produce status="not_needed" without launching a worker. Failed, unavailable, cancelled or timed-out requests remain explicit outcomes rather than successful advice.

experiments is an allowlist of at most 128 unique id/purpose objects. Structured replies may select one of those IDs or "stop"; no experiment runs merely because it was selected. MCP text mode cannot select experiments. cancel! requests interruption through cancellation; inspect the final status and diagnostics before using any response.

source

PerfChecker.native_tool_plan Method

native_tool_plan(tool, script; project=dirname(Base.active_project()),
                 output="native-profile", julia=first(Base.julia_cmd().exec),
                 suppressions=nothing)

Prepare shell-free arguments for optional native profilers. No process is started and no software is installed. Linux tools can be planned on Windows for execution inside Linux/WSL with Linux paths. The plan reports local availability separately from qualification. Capture the workload's oracle and dependency inventory alongside the resulting artifact; a profiler exit code does not prove workload correctness.

Supported tools are :memcheck, :callgrind, :massif, :cachegrind, :heaptrack, :perf and :vtune; unknown tools or empty script/output strings raise ArgumentError. Return perfchecker-native-tool-plan/1 with executable, argument vector, intended platform/artifact paths, availability and limitations. suppressions adds a Valgrind suppressions-file argument only. Paths are passed as supplied, so callers planning a WSL/Linux run must supply paths valid there. No output directories, suppression files or profiler artifacts are created.

source

PerfChecker.network_interface_capabilities Method

network_interface_capabilities()

Describe platform support and available interfaces for network-counter sampling. This reports collection capability; interface counters include other traffic using the same interface. Use network isolation for process-tree attribution.

source

PerfChecker.network_interface_delta Method

network_interface_delta(before, after; workload_seconds=<timestamp interval>,
                        capture_layer="interface", attribution_scope="host_interface",
                        latency_context="end_to_end_informative")

Return a named tuple of Float64 byte/packet/discard deltas, counter interval seconds, separate workload_seconds, and capture/attribution metadata. Snapshots must represent the same interface in chronological order. Different interfaces, decreased counters or negative workload durations raise ArgumentError. Counter resets/wraps are not repaired. Metadata keywords label the observation; they do not establish isolation or measure network latency.

source

PerfChecker.network_interface_snapshot Function

network_interface_snapshot(interface="auto")

Return a NetworkInterfaceSnapshot using the Linux or Windows provider. An explicit name is matched exactly; auto (case insensitive) selects the nonloopback interface with the largest cumulative sent-plus-received bytes, falling back to loopback when necessary. It does not infer the route used by a particular destination. Unknown names or unsupported platforms raise ArgumentError; no available counters raise ErrorException. Provider filesystem/process errors may propagate.

source

PerfChecker.network_isolation_capabilities Function

network_isolation_capabilities(spec=NetworkIsolationSpec(); probe=false)

Return provider support/reason, attribution, interface, connectivity, counters and prepared-environment requirements. By default inspect platform and required executables only. probe=true starts a short namespace command to test user/ network namespace and required-tool access; it does not run a workload. Linux requires unshare, ip, timeout and nft, plus slirp4netns in external mode; WSL2 must be prepared inside the selected distribution. Unsupported capability is reported as supported=false; process-launch failures can propagate. The result describes availability, not benchmark qualification.

source

PerfChecker.network_isolation_spec_dict Method

network_isolation_spec_dict(spec::NetworkIsolationSpec)

Return perfchecker-network-isolation-spec/1 with string provider, optional distribution, interface, external-connectivity flag and DNS addresses. This is configuration transport; it neither probes the host nor starts isolation.

source

PerfChecker.oracle_spec_dict Method

oracle_spec_dict(oracle::OracleSpec) -> Dict{String,Any}

Return the declared function name as "function" and its Boolean "required" flag. This in-memory representation contains no correctness evidence and does not invoke the oracle.

source

PerfChecker.package_dependency_inventory Method

package_dependency_inventory()

Return (records, warnings) from the active environment's Pkg.dependencies(). Records are sorted by package name and UUID and retain version/tree identity, tracking origin, source path, direct-dependency status and an existing Artifacts.toml path. JLL wrappers are labelled separately from Julia packages. If Pkg inventory fails, return an empty collection and a bounded error warning. No package is loaded, installed or resolved by this function.

source

PerfChecker.perf_setup Method

perf_setup(; dir="perf", kinds=(:benchmark, :chairmark, :alloc, :pluto), force=false)

Create a small Julia-native performance workspace.

This writes benchmark, chairmark, allocation, and Pluto dashboard starter files by default. It intentionally does not create a Perf.toml; options stay in Julia code through PerfConfig. The Pluto dashboard activates the surrounding Julia project so it can be used as a controlled project-local view over stored or newly run checks. Return a vector of generated paths in kinds order. Existing destination files require force=true; if a later kind fails, earlier files may already have been written. This creates source starters, not measurements or a prepared package environment.

source

PerfChecker.perfchecker_main Function

perfchecker_main(args=ARGS; stdout=Base.stdout, stderr=Base.stderr)

Dispatch the perfchecker command line and return an integer exit code. With no arguments or a help command, print usage. Commands cover suite planning, preflight/runs, saved-evidence export/comparison/integrity, test items, scenarios, diagnostics, advisor setup/conversation/implementation, Julia campaigns, network isolation, native plans and machine transfer. Options use the documented CLI grammar, and external command arguments follow --.

Return 0 for successful commands, 1 for command-specific failed evidence/gates, 2 for handled input/runtime errors, or 130 for InterruptException. Errors handled by the dispatcher are printed to stderr; option splitting before dispatch can still propagate errors. stdout/stderr accept redirected streams. File writes, worker execution, provider calls and explicit implementation operations occur only for their selected command; version and help are read-only.

julia> output = IOBuffer();

julia> perfchecker_main(["version"]; stdout=output)
0

source

PerfChecker.performance_document_block Method

performance_document_block(payload::AbstractDict)
performance_document_block(bundle::RunBundle, block::PerformanceDocumentBlock)

Parse a block dictionary with string or symbol keys, or materialize a block against a saved bundle. Parsing accepts the perfchecker-document-block/1 schema (also the default when omitted), requires a query dictionary and validates its views and query. Missing id and title default to empty strings; missing views in this dictionary form defaults to an empty list.

The bundle form returns a dictionary containing the block's presentation fields, run/attempt/environment provenance, and query_bundle result. Neither form runs workloads or writes files. Malformed schemas or fields raise ArgumentError or conversion errors.

source

PerfChecker.performance_figure Function

performance_figure(plot::PerformancePlot)
performance_figure(bundle::RunBundle, [id]; kwargs...)

Load PerfCheckerMakie and a Makie backend to render a saved-evidence plot as a Makie figure. Bundle overloads first call performance_plot, forwarding its selection/grouping keywords. The companion accepts separate figure_kwargs, axis_kwargs and plot_kwargs attribute bundles plus presentation tags. Unsupported plot kinds raise ArgumentError. No workload is rerun, and rendering does not save an image; use the active backend's save API for an explicit export. Without the companion's methods, calling this generic function raises MethodError.

source

PerfChecker.performance_plot Method

performance_plot(bundle, [id]; reference_version=nothing, version=nothing,
                 statistic=:minimum, top=40, min_percentage=5)

Build a backend-neutral plot. With no identifier, prefer overlaid BenchmarkTools or Chairmarks metrics. By default, use each version's minimum sample, divided by the minimum across the compared versions separately for each metric (best = 1). Pass statistic=:median to compare medians instead. Release targets use semantic version order; Git labels follow the series order. Use reference_version=:latest or a version label to select a target instead of the minimum. Raw measurements and units remain in every record. Equal zeros appear at 1 (unchanged), by convention; a nonzero value over a zero reference, or a missing reference, has no finite ratio. Separate absolute plots remain available through plot_catalog. Allocation pies combine sites contributing strictly less than min_percentage percent of the selected version's allocated bytes into "Other allocation sites". The default is 5%; exactly 5% remains separate. Set min_percentage=0 to disable this threshold. top still caps legend entries, including the combined remainder. Allocation heatmaps retain only observed version/site pairs, including explicit zero weights. Their complete axis labels remain in the options, with missing_cell_policy="not_observed"; absent cells are not measured zeros. Return a PerformancePlot from saved records without rerunning workloads. Standard models retain their measurement definitions (or normalized collector) and manifest tags in presentation options so renderers can label evidence without inferring its collector from units. Tradeoffs preserve raw recorded values and identify time_unit, allocation_unit, time_series_id and allocation_series_id in their options, with unit_source="recorded_series". Only the compatible, unambiguous pairs advertised by plot_catalog are available. version selects allocation/profile versions; normalized metrics use reference_version instead. Unknown plot IDs, absent plottable measurements, unsupported statistics or invalid grouping limits raise ArgumentError in the applicable plot builder. Use performance_plot_dict for transport or performance_figure after loading PerfCheckerMakie to render the model.

source

PerfChecker.performance_plot_dict Method

performance_plot_dict(plot::PerformancePlot)

Return perfchecker-plot/1 with identity, string kind, title, description, encoding, data and options. Nested dictionaries and records are retained, not deep-copied. No renderer is loaded and no report, HTML or image is written.

source

PerfChecker.performance_plot_html Function

performance_plot_html(plot::PerformancePlot;
                      asset_directory=nothing, html_directory=nothing)

Load PerfCheckerMakie and WGLMakie to render the plot as an embeddable interactive HTML string. The companion's WGLMakie extension supplies the method; without it, this generic raises MethodError. Rendering consumes saved data and does not start a measurement or write/deploy an HTML page. The caller owns embedding and any file export. By default the string includes its assets for standalone use. To share WGL assets between pages, provide both asset_directory and html_directory: Bonito writes assets only to the former and generates links relative to the latter, where the caller will save the HTML. Move or deploy both directories together without changing their relative layout. Normalized plots and flame graphs already use compact standalone HTML and do not write shared assets.

source

PerfChecker.performance_query Method

performance_query(payload::AbstractDict)

Construct a PerformanceQuery from string or symbol keys. Accept perfchecker-query/1 or an omitted schema. where contains predicate objects (field, operator, value); order_by contains objects with field and direction. Omitted fields use the constructor defaults, and omitted predicate operators and sort directions use equals and asc respectively. Unsupported schemas, nonobject predicates/order entries and invalid selections raise ArgumentError; incompatible field types can raise conversion errors.

julia> query = performance_query(Dict("resources" => ["observations"], "limit" => 2));

julia> query.limit
2

source

PerfChecker.performance_query_dict Method

performance_query_dict(predicate::QueryPredicate)
performance_query_dict(query::PerformanceQuery)

Return a dictionary suitable for JSON transport. A predicate contains field, string operator and value; a query additionally includes its schema, id, resources, where, ordered sort specifications and limit. The result preserves the query's selection semantics and is accepted by performance_query. Predicate values are retained as supplied; callers must supply JSON-compatible values when serializing. No file is written.

source

PerfChecker.plan_suite Method

plan_suite(suite::SoftwareSuite; profile=:quick,
           version_provider=get_pkg_versions, candidates=Dict(), comparisons=[])
    -> SuitePlan

Resolve package releases, development sources, Git candidates, feature variants, and comparison policies without running workloads. Registry version lookup, project-version inspection and worker-seed warnings can occur during planning.

Profiles choose the target scope: :quick selects development when enabled, otherwise the latest declared release; :ci selects representative endpoints within release groups plus development; :historical selects all declared releases plus development; :release selects releases only. Named Git candidates are added outside :release. version_provider(package_name) supplies releases for symbolic version selections and can be overridden for offline planning.

candidates maps package names or package-suite IDs as strings to extra SuiteCandidate vectors. comparisons appends policies to those in the suite and duplicate IDs are rejected. Rows with no matching package/Julia variant are retained as unavailable; multiple matching variants raise an error. This does not prepare environments, check out Git refs or call a workload.

source

PerfChecker.planned_run_id Method

planned_run_id(run::PlannedFeatureRun) -> String

Return a deterministic 20-character content-digest prefix from suite ID, package-suite ID, feature ID, target label and comparison key. Plan filters, progress events and interfaces use this selection identity. It is not a hash of the workload source, dependency resolution or completed measurement.

source

PerfChecker.plot_catalog Method

plot_catalog(bundle::RunBundle)

Return dictionaries describing plot IDs, kinds, titles and identifying metrics available from the bundle's saved observations. Include supported normalized benchmark metrics, version series/distributions/deltas, time/allocation tradeoffs, allocation site/file/stack plots and CPU/wall profiles. Entries are ordered by package/feature, preferring normalized metrics; their IDs are content-derived identities accepted by performance_plot. Time/allocation tradeoffs pair metrics only within the same collector identity. Comparisons with multiple collectors expose separate labeled entries; existing IDs are retained when only one collector supplies a valid pair. A pair requires one unambiguous series per metric, a supported time unit and allocation bytes (By). Ambiguous or incompatible pairs are omitted without hiding independent series and distributions. Tradeoff entries retain separate time_unit and allocation_unit fields and both source series IDs; values are not converted. A bundle without supported observations returns an empty list. No workload or graphics backend is started; malformed evidence errors propagate.

source

PerfChecker.preflight_passed Method

preflight_passed(report::CompatibilityReport)

Return whether every diagnostic's blocking value is false. Missing values default to blocking; an empty report passes this predicate. It checks saved preparation evidence without executing or qualifying a workload.

source

PerfChecker.preflight_suite Method

preflight_suite(plan::SuitePlan; overrides=Dict{Symbol,Any}(),
                resolver=_default_preflight_resolver,
                probe_runner=_default_preflight_probe_runner,
                progress_callback=_->nothing)
preflight_suite(suite::SoftwareSuite; profile=:quick,
                version_provider=get_pkg_versions, kwargs...)

Return a CompatibilityReport for planned target environments. Resolve ready environments inside declared constraints; reuse identical environment resolution evidence within the call. Run declared functional probes through probe_runner after successful resolution, but not the measured workload or its correctness oracle. Preparation can install/resolve dependencies and execute probe code in isolated workers with the default providers.

Diagnostics distinguish supported, unavailable feature, unsatisfiable/unknown resolution, capability-blocked, degraded nonblocking probes and probe errors. Resolver/probe exceptions become bounded blocking diagnostics where handled. overrides are passed to providers and progress_callback receives preflight progress dictionaries. Injected callbacks/providers must honor their respective contracts. The suite overload first plans targets and forwards keywords.

source

PerfChecker.prepare_pluto_dashboard Function

prepare_pluto_dashboard(path; suite_path=nothing, factory=:build_suite, kwargs...)

Load PerfCheckerPluto. Write a suite dashboard notebook and return its absolute path without starting Pluto or a workload. suite_path selects the Julia suite definition and factory its constructor; the notebook uses explicit run controls. Other keywords, including project, reports_root and force, are forwarded to write_suite_notebook.

source

PerfChecker.print_suite_plan Method

print_suite_plan([io=stdout], plan::SuitePlan; limit=typemax(Int))

Print a readable plan, including target labels and availability status, without starting workers. limit must be positive and bounds displayed rows.

source

PerfChecker.probe_julia_runtime Method

probe_julia_runtime(spec::JuliaRuntimeSpec; project=nothing)
    -> Dict{String,Any}

Launch a fresh Julia process with --compile=min and return its observed version, Git commit, bindir, llvm_version, command, UTC resolution time and declared spec under schema perfchecker-julia-runtime-probe/1. Startup and history files are disabled. project, when supplied, selects the child project.

The executable/channel must be available. Process launch failures, nonzero exits and unexpected output are raised to the caller; no runtime is installed and no workload is measured.

source

PerfChecker.probe_spec_dict Method

probe_spec_dict(probe::ProbeSpec) -> Dict{String,Any}

Return the JSON-compatible declaration with string fields id, function and category, and the Boolean blocking flag. This describes the check to execute; it contains no observed result and does not invoke the probe.

source

PerfChecker.process_memory_capabilities Method

process_memory_capabilities()

Return supported, provider, counter names, peak semantics and attribution. Linux uses /proc and optionally smaps_rollup; Windows uses PSAPI. Other platforms report unsupported counters. This describes platform capability; a protected or exited process can still be unavailable.

source

PerfChecker.process_memory_snapshot Function

process_memory_snapshot(pid=getpid(); strict=false)

Return a ProcessMemorySnapshot of operating-system counters. Unsupported platforms and inaccessible processes normally produce status=:unavailable with a message; individual missing counters remain nothing. With strict=true, unavailability raises ErrorException. RSS includes Julia and native resident pages; it is not Julia allocation volume or a workload-specific peak.

julia> snapshot = process_memory_snapshot();

julia> snapshot.pid == getpid()
true

source

PerfChecker.process_memory_snapshot_dict Method

process_memory_snapshot_dict(snapshot::ProcessMemorySnapshot)

Return a perfchecker-process-memory/1 dictionary retaining PID, monotonic timestamp, byte counters, provider, string status and message. Unknown counters stay nothing (JSON null); no new observation is taken.

source

PerfChecker.query_bundle Method

query_bundle(bundle::RunBundle, query::PerformanceQuery)

Return a perfchecker-query-result/1 dictionary containing run identity, serialized query, resource counts, and selected observations, diagnostics, artifacts, plot catalogue entries and version comparisons. Unrequested resources are present as empty vectors. Apply predicates, stable multi-field sorting and the per-resource limit independently to each requested collection.

Comparison records are derived from the saved bundle by compare_suite_versions; plot records come from plot_catalog. This reads existing evidence without launching a worker, opening artifact files, writing a report or mutating the bundle. Errors in malformed saved evidence propagate to the caller.

source

PerfChecker.query_result_dict Method

query_result_dict(bundle::RunBundle, query::PerformanceQuery)

Return query_bundle's result dictionary. This convenience entry point has the same filtering, per-resource bounds and error behavior; it does not serialize to a file or make a deep copy of nested evidence values.

source

PerfChecker.read_advice Method

read_advice(path::AbstractString) -> AbstractDict

Read a JSON advice report with schema perfchecker-advice/1 and a recommendations array. Reject an unsupported schema or malformed array. This reads recorded deterministic recommendations; it neither executes the target workload nor invokes a model. Use narrate_advice or chat_advice explicitly to request optional prose.

source

PerfChecker.read_document_blocks Method

read_document_blocks(path)

Validate a shared configuration with read_ui_configuration, then parse documentation.blocks using performance_document_block. Return a vector of PerformanceDocumentBlocks, empty when documentation or blocks are absent. A nonobject documentation raises ArgumentError; malformed blocks and file errors propagate. No bundle is loaded or modified.

source

PerfChecker.read_property_corpus Method

read_property_corpus(path)

Read a frozen property corpus and return its JSON dictionary, including cases and generation metadata. Reject missing files, unsupported schemas and a count that disagrees with the stored cases. No generator or shrinking code is run.

source

PerfChecker.read_provider_result Method

read_provider_result(path)

Parse perfchecker-provider-result/1 JSON and return a RunBundle. Validate measurement definitions, unique definition IDs, numeric observations and references, normalizing run/attempt UUIDs and record identity. Omitted UUIDs are generated; optional runtime/environment fields are retained and execution state defaults to complete. This normalization is independent of executing the provider or checking workload correctness. Unsupported schema or invalid records raise ArgumentError; file and JSON errors propagate. No files or artifact payloads are written.

source

PerfChecker.read_run_bundle Method

read_run_bundle(directory; verify_integrity=true, require_integrity=false)

Read and validate a perfchecker-run-bundle/1 directory. Integrity metadata is verified by default when present and can be made mandatory with require_integrity=true. Return a RunBundle, preserving record order and unknown fields. verify_integrity=false skips the integrity check entirely, including require_integrity; use that combination only for deliberate legacy inspection or migration. Missing JSONL documents are read as empty lists when the integrity check does not require them. Unsupported schemas and malformed definition/artifact documents raise ArgumentError; file/JSON errors propagate. This does not open artifact payloads or execute stored workloads.

source

PerfChecker.read_scenario_runs Method

read_scenario_runs(directory::AbstractString) -> Vector{RunBundle}

Read a scenario bundle directory, or its immediate child bundle directories, using read_run_bundle's integrity checks. Keep bundles whose manifest contains scenario metadata. Reject a missing directory or a directory containing no such bundles. This is not a recursive search and does not modify evidence or rerun the workload; use list_run_bundles for broader discovery.

source

PerfChecker.read_ui_configuration Method

read_ui_configuration(path)

Read JSON from abspath(path) and return a dictionary with string keys. Require schema perfchecker-ui-config/1 and a selection object containing unique string run_ids. Other fields are retained for the consuming interface; this is structural validation, not a check that selected runs exist. Invalid structure raises ArgumentError; file and JSON errors propagate.

source

PerfChecker.register_oxygen_routes! Function

register_oxygen_routes!(source; prefix="/perfchecker/v1", kwargs...)

Load PerfCheckerWeb. Register routes on Oxygen's current router for a result, bundle, result provider or interactive suite. Read-only evidence routes do not run measurements; suite/studio control routes may launch work. This registers routes without starting the HTTP server. See serve_suite for hosting.

source

PerfChecker.register_testitem_routes! Function

Register an Oxygen UI for existing TestItemRunner items. Load PerfCheckerWeb; bind the server to loopback or supply external authentication.

source

PerfChecker.reset_check_preferences! Method

reset_check_preferences!(keys::Symbol...; block_inheritance=false,
                         export_prefs=false)

Delete selected persistent check defaults from the active project and return check_preferences. With no keys, reset threads, repeat and quiet. Unknown keys raise ArgumentError before writing. Other preference keys and other packages' settings are preserved.

By default, delete local settings and allow defaults inherited from other Julia environments or Project.toml to become visible. block_inheritance=true instead uses Preferences.jl's clear markers to block inherited values at the selected storage level. export_prefs=true selects Project.toml; otherwise select LocalPreferences.toml. Resetting one level does not delete values at the other.

reset_check_preferences!(:threads) # inherit a shared value, if any
reset_check_preferences!(block_inheritance=true) # use collector defaults

This changes future checks only and does not require restarting Julia.

source

PerfChecker.resource_envelope_dict Method

resource_envelope_dict(envelope::ResourceEnvelope)

Return perfchecker-resource-envelope/1 with elapsed seconds, serialized before/after snapshots and external-requested metadata. Preserve unknown counters and provider messages; no new sample is taken or file written.

source

PerfChecker.resource_envelope_metrics Method

resource_envelope_metrics(envelope::ResourceEnvelope)

Return symbol-keyed metrics as Float64 or nothing: collector seconds, before/after memory gauges, signed gauge deltas, nonnegative new peak RSS, and external allocation/free deltas. Missing input counters yield missing derived metrics. Decreasing cumulative allocation/free totals raise ArgumentError. RSS/private/live/reserved deltas may be negative; new peak RSS is clipped at zero and measures growth of the process-lifetime high-water mark.

source

PerfChecker.resource_policy_passed Method

resource_policy_passed(evaluation::AbstractDict)

Return true exactly when the string-keyed status is "passed". Failed, unavailable and missing statuses return false; metrics are not reevaluated and empty violation lists do not independently establish success.

source

PerfChecker.run_external_command Method

run_external_command(spec::ExternalCommandSpec; bundle_root=nothing, strict=false)

Start the configured command in its directory, adding its environment plus PERFCHECKER_OUTPUT (a temporary JSON path) and PERFCHECKER_CASE_ID. Return a RunBundle. Timeout, nonzero exit or missing result file produce a failed bundle; invalid result JSON/schema and launch failures propagate. Nonempty stderr on a successful result becomes a bounded warning diagnostic. Owned process trees are stopped and temporary result/output resources cleaned before returning; arbitrary provider-created files remain provider-owned.

If bundle_root is set, write run-<run_id> there, including failed evidence. With strict=true, an unsuccessful bundle raises an error after optional writing. This explicitly executes external code; an exit code alone does not establish workload correctness or a performance-budget pass.

source

PerfChecker.run_julia_runtime_campaign Method

run_julia_runtime_campaign(specs::AbstractVector{JuliaRuntimeSpec};
    suite, reports, profile=:ci, factory=:build_suite,
    perfchecker_project=dirname(@__DIR__), timeout_seconds=3600,
    relative_limits=Dict{String,Float64}(), min_samples=1, strict=false,
    sample_statistics=Dict{String,Symbol}(), default_sample_statistic=:median,
    progress_callback=payload -> nothing, resume=false) -> JuliaRuntimeCampaign

Run the same suite definition sequentially under an explicit Julia runtime axis. Supply at least two unique runtime IDs and exactly one :baseline role; control and candidate runtimes are both compared with that baseline. Runtime selectors must already resolve to available executables/channels.

suite is loaded through load_software_suite. reports receives a separate controller project and report directory per runtime, plus campaign and source-attribution JSON/Markdown reports. Child controllers prepare their dependencies and run measurement workers, so this operation starts processes, can use package registries/depot downloads and writes reports. timeout_seconds is a positive per-runtime execution budget. Progress callbacks receive completed counts and the active runtime; callback failures are logged without aborting work.

Comparison budgets/statistics have the meanings documented by compare_bundles. strict=false returns failure records; strict=true raises after writing reports if the campaign does not pass. Invalid declarations, suite loading and other orchestration failures can still throw in either mode.

With resume=true, previously passed runs may be reused only when matching suite path, profile, runtime ID/role/selector, verified bundle integrity and the freshly probed runtime version/commit permit it. Reuse is recorded explicitly; it is not a new measurement and does not prove that an edited suite is unchanged.

specs = julia_runtime_matrix(baseline="1.10", candidates=["release"])
campaign = run_julia_runtime_campaign(specs;
    suite="perf/suite.jl", reports="perf/results/runtimes",
    relative_limits=Dict("julia.wall.time" => 0.05), min_samples=3)
runtime_campaign_passed(campaign)

source

PerfChecker.run_perfitem Method

run_perfitem(catalog, id; implementation="default", kwargs...)

Run exactly one declared performance item with the shared isolated lifecycle. Keywords are forwarded to run_scenarios, including prepared project, samples, timeout, cancellation and reports. Return the vector of RunBundles from the selected item's declared collectors, not a single scalar result. Unknown scenario/implementation identities fail during selection before a worker starts. Worker outcomes, cancellation and report side effects follow run_scenarios; this function does not adopt discovered proposals.

source

PerfChecker.run_scenarios Method

run_scenarios(catalog::ScenarioCatalog; project=catalog.root, samples=10,
              timeout=120, threads=1, cancellation=CancellationToken(),
              reports=nothing) -> Vector{RunBundle}

Execute every declared scenario/collector pair in a separate Julia worker using the existing project. samples must be positive; timeout is a finite positive per-worker wall-time budget including startup, and threads selects the worker's Julia thread count. Required collector and workload packages must already be installed in that environment. Factories prepare state outside the measurement; the operation and optional synchronization are measured, and verification checks the result outside that boundary.

Return one portable bundle per attempted pair, including explicit unavailable, invalid, error, timeout or cancelled outcomes. Source, fixtures and environment fingerprints are checked before and after a run; changed inputs invalidate its evidence. With reports, write each bundle under its run ID in that directory. Cancellation returns the attempted bundles and does not start remaining pairs. An empty catalog returns an empty vector without launching a worker.

The worker owner stops its process and cleans its own temporary transport files. Ordinary Julia unwinding invokes the scenario's cleanup callback; forcibly terminating the worker cannot run arbitrary user callbacks. Keep externally owned temporary resources recoverable independently, and inspect recorded cleanup failures before considering cancellation complete.

catalog = load_scenario_catalog("perf/scenarios.toml")
bundles = run_scenarios(catalog; project=".", samples=20,
    timeout=120, reports="perf/results/scenarios")
all(bundle_passed, bundles) # Inspect failed or unavailable bundles individually.

source

PerfChecker.run_studio_agent Function

run_studio_agent(suite; server, token, agent_id, poll_seconds=2,
                 heartbeat_seconds=30, max_jobs=typemax(Int), once=false)

Load PerfCheckerWeb. Register an agent with the specified controller, poll for jobs, execute accepted suite plans locally and upload progress/results. This performs network requests and runs workload processes; keep the server and suite under the caller's control. Use once=true for a bounded polling iteration or max_jobs to bound attempted leases, including failed attempts. Empty claims can keep polling indefinitely when once=false; max_jobs is not an idle-time limit. Heartbeat tasks are signalled to finish but are not awaited on return. Interruption and incomplete worker/trace cleanup always stop the agent, even with once=false. Controller-side cancellation of a remote lease does not interrupt its measurement; stop the agent cooperatively and allow cleanup to finish. An expired lease can be retried, so workloads must tolerate replay.

source

PerfChecker.run_suite Method

run_suite(plan::SuitePlan; executor=_default_suite_executor,
          overrides=Dict{Symbol,Any}(), strict=true,
          progress_callback=payload -> nothing) -> SoftwareSuiteResult
run_suite(suite::SoftwareSuite; profile=:quick,
          version_provider=get_pkg_versions, kwargs...) -> SoftwareSuiteResult

Execute the runnable leaves of a resolved suite plan and return their isolated worker results synchronously. Unavailable rows are retained without execution; ordinary individual failures are recorded while subsequent rows continue. strict=true raises an error carrying the completed result when suite_passed is false; strict=false returns that result for inspection. Neither mode suppresses interruption or cleanup/orchestration exceptions.

Execution has the environment, dependency, workload and callback effects documented by launch_suite. Overrides take precedence over variant and feature options. The suite overload plans first; the plan overload executes exactly its current ordered selection. No report files are exported automatically.

source

PerfChecker.run_suite_file Method

run_suite_file(path; profile=:quick, reports=nothing,
               factory=:build_suite, kwargs...)

Load and execute a suite definition. When reports is a path, write the JSON, Markdown, and JUnit representations consumed by CI and user interfaces. Return the SoftwareSuiteResult, not report paths. Other keywords are forwarded to run_suite, not to report writers. With the default strict execution, a suite failure is raised before the optional report-writing step; use strict=false to export completed failure records.

source

PerfChecker.run_suite_repl Method

run_suite_repl(plan::SuitePlan; overrides=Dict{Symbol,Any}(), reports=nothing,
               output=stdout, interactive=isinteractive(), strict=true, kwargs...)
run_suite_repl(suite::SoftwareSuite; profile=:historical,
               version_provider=get_pkg_versions, input=stdin, output=stdout,
               reports=nothing, interactive=isinteractive(), strict=true, kwargs...)

Run a plan through run_suite with terminal progress, returning its SoftwareSuiteResult. The suite form first prompts using configure_suite_repl; the plan form uses supplied overrides directly. Additional keywords go to run_suite, including prepared environment and cancellation options. interactive=true updates one progress line; otherwise progress is printed as separate lines.

When reports is a directory, write suite reports after execution. With strict=true, throw SuiteRunError for an unsuccessful result after report writing; with false, return the result for inspection. Errors preparing, executing or exporting a run propagate. This explicitly starts workloads.

source

PerfChecker.run_testitems Function

run_testitems(root=pwd(); ids=nothing, tags=[], exclude_tags=[],
              project=dirname(Base.active_project()), samples=1,
              timeout=120, threads=1, reports=nothing,
              cancellation=CancellationToken())

Execute a selection from discover_testitems with the official TestItemRunner lifecycle. Load TestItemRunner before calling. Prepare project with the target's test dependencies first; running does not install packages.

ids=nothing selects every item admitted by the performance/tag filters. Explicit IDs must be unique and known. Empty selections and nonpositive sample counts are errors. Each sample runs in a fresh process with the requested Julia threads and timeout. There is no implicit warmup or repetition.

The measured scope includes item setup, imports, assertions and module cleanup. It is suitable for checking the cost of an existing test; it does not isolate an inner operation. Use a feature workload when that narrower scope is required. Measured item workers set PERFCHECKER_TESTITEM_MODE=performance. A declaration with skip=(get(ENV, "PERFCHECKER_TESTITEM_MODE", "") != "performance") therefore runs under PerfChecker and skips in ordinary TestItemRunner or Julia VS Code runs.

Return a perfchecker-testitem-run/1 dictionary with item/sample records and a functional passed flag. performance=not_compared means no regression budget was evaluated; functional success is not a performance verdict. Source or environment changes during a sample invalidate its evidence. Failures, timeouts and cancellation remain visible in the result.

If reports is supplied, write the report into that directory without replacing an existing testitems.json. Cancellation stops the current worker and prevents further selected items from running. See testitem_filter for reserved tags.

source

PerfChecker.runtime_campaign_passed Method

runtime_campaign_passed(campaign::JuliaRuntimeCampaign) -> Bool

Return whether every run has status "passed" and every recorded comparison's "passed" flag is true. A diagnostic comparison without configured budgets does not pass. This only inspects the supplied object; it does not schedule missing runtimes or validate that a manually constructed campaign is complete.

source

PerfChecker.saveplot Function

saveplot(path, figure; overwrite=false, kwargs...)
saveplot(directory, named_figures; format=:svg, overwrite=false, kwargs...)

Load PerfCheckerMakie for methods exporting existing Makie figures to explicit SVG/PNG destinations. Load an export backend such as CairoMakie first:

using PerfChecker, PerfCheckerMakie, CairoMakie
figure = CairoMakie.Figure()
saveplot("example.svg", figure) # parent directory must exist

Existing files are preserved unless overwrite=true; directories are never replaced. The companion's collection method accepts name/Figure pairs and checks portable filenames and collisions before writing. Extra keywords go to Makie's save. Core alone provides this generic without methods, so unsupported calls raise MethodError. Exporting a figure does not rerun measurements.

source

PerfChecker.scenario_catalog_dict Method

scenario_catalog_dict(catalog::ScenarioCatalog) -> Dict{String, Any}

Return a perfchecker-scenario-catalog/1 object containing the absolute root and each scenario's identity, factory, parameters, source, fixtures, collectors and requirements. This conversion neither writes TOML nor runs the factory. Absolute paths describe this local catalog; use relative paths deliberately when preparing a catalog file for another machine.

source

PerfChecker.scenario_sync Function

scenario_sync(root=pwd(); previous=nothing)

Call discover and relate declared scenario/collector pairs to literal CI candidate configurations. Return perfchecker-scenario-sync/1 with coverage, declarations, proposals, changes, warnings, fingerprints and discovery payload. previous follows discovery's baseline dictionary/path contract. Coverage is a proposed cross-product labelled qualification="not_tested", not proof that CI invokes every scenario; adoption remains explicit. More than 10,000 combinations raises ArgumentError. No target code, CI commands or workflows are executed or written.

source

PerfChecker.select_scenarios Method

select_scenarios(catalog::ScenarioCatalog, selection::AbstractVector) -> ScenarioCatalog

Return a catalog containing the exact "id"/"implementation" pairs named by the selection dictionaries, in selection order. Reject duplicate selections and unknown identities. An empty selection returns an empty catalog. Filtering does not execute factories or promote inferred candidates to declared scenarios.

source

PerfChecker.select_suite_plan Method

select_suite_plan(plan::SuitePlan, run_ids::AbstractVector{<:AbstractString})
    -> SuitePlan
select_suite_plan(plan::SuitePlan, configuration::AbstractDict) -> SuitePlan

Return a plan containing only the requested rows, in the supplied order, preserving the original suite, profile and comparison policies. An empty selection is permitted. Duplicate or unknown IDs raise ArgumentError; unavailable rows remain unavailable if selected.

The dictionary overload requires schema perfchecker-ui-config/1 and a selection object with string-vector run_ids. It validates that selection against this plan; it does not reload files, compare a revision or execute work. Use IDs from suite_plan_dict, including rows hidden by current UI filters.

source

PerfChecker.serve_suite Function

serve_suite(source; host="127.0.0.1", port=8080, kwargs...)

Load PerfCheckerWeb to serve a suite, saved reports, bundle or scenario catalog in Oxygen. Available routes depend on the source type. The default host is loopback. Remote-control guards and keyword forwarding depend on the source overload: the interactive SoftwareSuite form requires an authenticator and allow_remote_control=true for a remote host, while the scenario catalog form is loopback-only. See the PerfCheckerWeb methods for route and server options. Loading a report view does not itself rerun its measurements.

source

PerfChecker.set_check_preferences! Method

set_check_preferences!(; export_prefs=false, kwargs...)

Persist any supplied check defaults in the active Julia project using Preferences.jl and return check_preferences. threads must be a positive Int-representable integer other than Bool; repeat and quiet must be Bool. Unknown keys and invalid values are rejected before writing. Unspecified settings are preserved. No workload path, command or credential is accepted by this API.

The default writes LocalPreferences.toml; export_prefs=true explicitly writes shareable preferences into Project.toml. Julia's environment-stack inheritance still applies, and local preferences override exported preferences. Existing explicit check options win over these defaults. Changes affect future checks, not a running suite's frozen settings. No restart is required.

set_check_preferences!(threads=2, quiet=true)
config = PerfConfig(:profile; path=pwd(), threads=1) # explicit value wins

Use reset_check_preferences! to remove selected settings or block their inheritance. Other packages' preferences are not changed.

source

PerfChecker.similar_machines Method

similar_machines(target, profiles; limit=5)

Rank machine-profile dictionaries with matching architecture, OS and word size. Return up to positive limit rows containing spec_id, label, distance, missing features and authority="specification_similarity_only". Distance averages absolute log ratios of positive available core/thread/SIMD/ memory/thread-count features, with a penalty for a different CPU identity. Profiles without any comparable numeric features are omitted; ties use spec_id. Invalid schemas or missing machine identities raise ArgumentError. This compares declared specifications, without measuring performance.

julia> profile = machine_profile(label="local");

julia> only(similar_machines(profile, [profile]))["distance"]
0.0

source

PerfChecker.studio_token_authenticator Function

studio_token_authenticator(users_or_path) -> Function

Load PerfCheckerWeb. Build a bearer-token verifier from a digest-to-identity dictionary or a TOML file with [[users]], id, token_sha256 and roles. The returned function hashes a supplied token and returns its identity or nothing. It does not create credentials, start a server or grant new roles.

source

PerfChecker.suite_dashboard Function

suite_dashboard(result::SoftwareSuiteResult; view=:normalized)
suite_dashboard(job::SuiteJob; strict=true, view=:normalized)
suite_dashboard(suite::SoftwareSuite; profile=:quick, strict=true, view=:normalized, kwargs...)

Load PerfCheckerMakie to overlay the first available benchmark workload's measurements, each divided by its minimum across versions. Use plot_catalog and performance_figure to select another workload. With view=:absolute, show minimum benchmark times for every run; this is also the fallback when no overlay is available. The result overload only renders saved measurements. The job overload waits for completion; the suite overload launches and waits for new measurements. Return a Makie figure.

source

PerfChecker.suite_dict Method

suite_dict(result::SoftwareSuiteResult) -> Dict{String,Any}

Return schema perfchecker-suite-result/1, suite/profile/timestamps, execution passed and verdict, and ordered run summaries, resource envelopes and qualification/provenance evidence. A run's summary is the first checker summary row when present, not the complete sample/profile distribution. Use a run bundle for portable raw observations. This only converts supplied evidence.

source

PerfChecker.suite_job_dict Method

suite_job_dict(job::SuiteJob) -> Dict{String,Any}

Return a perfchecker-suite-job/1 snapshot with job/suite/profile identity, status and progress. Include result through suite_dict when complete, or a captured failure message when failed. This is a nonblocking poll and does not await cancellation, write reports or execute further measurements.

source

PerfChecker.suite_job_progress Method

suite_job_progress(job::SuiteJob) -> Dict{String,Any}

Return a shallow snapshot under schema perfchecker-progress/1, with state, total/completed/remaining counts, fraction/percent, per-status counts and the active row's identity when available. Counts describe orchestration progress, not statistical confidence or correctness validation. This does not wait; nested values are not deep-copied.

source

PerfChecker.suite_job_status Method

suite_job_status(job::SuiteJob) -> Symbol

Read :queued, :running, :cancelling, :complete, :cancelled or :failed without waiting. :complete means execution returned a result, not that its feature records passed. :cancelled is reported after interruption handling; cleanup failures remain :failed and are surfaced by wait_suite.

source

PerfChecker.suite_passed Method

suite_passed(result::SoftwareSuiteResult) -> Bool

Return whether no run has status :error, :blocked or :invalid. Unavailable runs do not fail this execution predicate. Use suite_verdict to distinguish partial execution from validated results. An empty or entirely unavailable result can satisfy this execution predicate; it is not a minimum-coverage or regression-budget assertion.

source

PerfChecker.suite_plan_dict Method

suite_plan_dict(plan::SuitePlan) -> Dict{String,Any}

Return schema perfchecker-suite-plan/1, suite/profile metadata, comparison policies, and ordered rows with IDs, target provenance, entrypoints, collector, qualification declarations, compatibility, status and reason. plan_revision is a content digest of this payload, not a source/dependency fingerprint. This exposes declarations for interfaces without starting execution or writing files.

source

PerfChecker.suite_summary Method

suite_summary(result::SoftwareSuiteResult) -> TypedTables.Table

Return one row per feature run with columns suite, package, feature, version, comparison_key, status, elapsed_seconds and message. Elapsed seconds measure orchestration, including preparation, not benchmark sample time. No observations are rerun or written; use backend summary/plot APIs for measured metrics.

source

PerfChecker.suite_verdict Method

suite_verdict(result::SoftwareSuiteResult) -> Symbol

Summarize execution and qualification in priority order: execution failure, blocked capability, invalid result, partial execution, no execution, validated, execution with warnings, or executed. Ordinary successful execution without a correctness oracle remains :executed.

source

PerfChecker.suite_version_series Method

suite_version_series(bundle::RunBundle) -> Vector{Dict{String,Any}}

Group finite numeric observations carrying package, feature, version and target-kind attributes into metric series. Each series preserves its workload, comparison key, measurement definition and unit, with ordered version points. Release versions sort before development/Git targets.

Ordinary points retain sample count, median and available mean/minimum/maximum/ percentile statistics. Allocation-site bytes and counts are summed per target; their point's median field holds that total and aggregation is "sum". Line-tracking allocation fractions and CPU/wall sampled-profile observations are excluded from this scalar version-series view; their source profiles use other plot types. Observations without the required attributes are ignored.

This reduces saved evidence in memory and does not execute a suite or write files.

source

PerfChecker.summary_table Method

summary_table(result::CheckerResult) -> Table

Build a compact table for REPL display or Pluto notebooks.

The summary uses a common schema across benchmark-like and allocation-like tables. Missing metrics are reported as missing rather than inferred. Return one row per result table with package/version, row count and min/max time, memory and allocation columns. Use numeric values only and retain the source table's units: benchmark times may be nanoseconds while Chairmarks times are seconds. This is a display summary without unit normalization, confidence intervals, performance-gate evaluation or worker execution.

source

PerfChecker.table_to_csv Method

table_to_csv(table::Table, path::String)

Write a TypedTables.Table to path, creating the parent directory if needed. Return CSV.write's destination result; an existing file is replaced with the table's column header and rows. This exports supplied table values without unit conversion, bundle metadata or integrity protection. Serialization/filesystem errors propagate; no workload is run.

source

PerfChecker.table_to_pie Function

table_to_pie(table, ::Val{backend}; kwargs...)

Load PerfCheckerMakie to create a pie chart from a backend table. Implemented by the companion package for allocation tables with Val(:alloc). Sites below 5% of allocated bytes are combined by default. Use min_percentage=0 to show smaller sites; top=40 caps legend entries, including the combined remainder. Return a Makie figure; an allocation table without sites logs an error and returns nothing. The table must contain byte counts and filename/line columns. This visualizes saved values without running a check or saving an image.

source

PerfChecker.terminal_plot Function

terminal_plot(plot::PerformancePlot; width=80, height=20)
terminal_plot(bundle::RunBundle; plot_id=nothing, kind=nothing,
              version=nothing, top=20, width=80, height=20)
terminal_plot(comparison::VersionComparison; series_id=nothing, width=80, height=20)

Load UnicodePlots to render saved evidence as a text-terminal plot object. Bundle selectors choose from plot_catalog; comparison selectors choose a version series, preferring normalized metrics when no series is specified. Width/height control terminal dimensions. Unsupported/empty plot data raises ArgumentError; invalid exact selectors can raise selection errors. No workload is rerun and no plot is saved. Without the optional extension's methods, this generic raises MethodError.

source

PerfChecker.testitem_filter Function

testitem_filter(mode=:performance; tags=[], exclude_tags=[])

Build a TestItemRunner filter. Untagged items are shared. :test_only excludes performance execution; :check_only excludes functional execution. The existing :perf_only tag is an alias of :check_only, including tag selections and exclusions. Combining either performance-only tag with :test_only is rejected. A nonempty tag selection matches any requested tag. Use @run_package_tests filter=testitem_filter(:test) in functional CI. Loading PerfChecker never changes TestItemRunner's default behavior.

source

PerfChecker.to_table Function

to_table(raw_result) -> TypedTables.Table

Convert a backend-specific raw result into a table stored by PerfChecker. Backends extend this method for their raw result types. Core supplies methods for allocation, Julia CPU/wall profile and explicit/interface network records; optional benchmark collectors add their own raw-result methods. A table preserves backend columns and units without attaching run-bundle provenance. Unsupported raw types raise MethodError; no measurement or file write occurs.

source

PerfChecker.tool_catalog Method

tool_catalog(; category=nothing)

Read the bundled data/tool-catalog.json and attach current diagnostic_capabilities under analyzers. Return a dictionary; a category string or symbol filters tools by exact category, while nothing retains all tools. Unknown categories produce an empty tools list. Installed, candidate and qualified statuses remain distinct: listing a tool does not execute, install or qualify it. Bundled-file errors propagate.

source

PerfChecker.verify_run_bundle Method

verify_run_bundle(directory; require_integrity=false)

Check integrity.json against all five protocol documents: manifest, measurement definitions, observations, diagnostics and artifact descriptors. Require SHA-256 algorithm, exactly one record per permitted filename, matching byte lengths and digests. Return status="verified", verified=true and names on success. Artifact payload files under artifacts/ are outside this digest set.

Without an integrity manifest, return legacy_unverified unless require_integrity=true. Missing directories, malformed manifests, unexpected paths or size/digest mismatches raise ArgumentError. This establishes byte integrity, not workload correctness, provenance authenticity or schema validity of every observation. No files are changed.

source

PerfChecker.version_comparison_dict Method

version_comparison_dict(comparison::VersionComparison) -> Dict{String,Any}

Return schema perfchecker-version-comparison/1, the source run ID and validity, passed/verdict, warnings, target availability, metric series and comparison records. No bundle is reread and no workload or writer is invoked.

source

PerfChecker.version_comparison_passed Method

version_comparison_passed(comparison::VersionComparison) -> Bool

Return true only when version_comparison_verdict is :qualified. Successful execution without a complete, passing comparison policy returns false.

source

PerfChecker.version_comparison_verdict Method

version_comparison_verdict(comparison::VersionComparison) -> Symbol

Return :invalid_input, :regressed, :qualified or :inconclusive for a VersionComparison. Empty records and incomplete evidence are inconclusive; a successful input bundle alone does not qualify a version comparison.

source

PerfChecker.wait_investigation Method

wait_investigation(job::InvestigationJob) -> Dict

Wait for this job's asynchronous task, then return its full investigation_status snapshot. Recorded errors, incomplete results and cancellation remain visible; a returned dictionary is not itself a correctness or performance pass. Waiting does not rerun any action.

source

PerfChecker.wait_suite Method

wait_suite(job::SuiteJob; strict=true) -> SoftwareSuiteResult

Wait for the controller task to finish. Rethrow orchestration failures and throw InterruptException for cancellation. With strict=true, a failed suite raises an error carrying its result; strict=false returns completed failure records for inspection. It does not suppress orchestration exceptions.

source

PerfChecker.worker_environment Method

worker_environment(package::PackageSuite) -> String

Return the absolute seed-directory path declared for isolated measurements. This accessor does not copy or activate the project. It is distinct from the controller environment in which PerfChecker orchestrates the suite.

source

PerfChecker.workload_id Method

workload_id(feature::FeatureSpec) -> Symbol
workload_id(run::PlannedFeatureRun) -> Symbol

Return the business-workload identifier, which defaults to the feature ID but can be shared by multiple collector-specific features. This reads the declared identity; it does not infer workload equivalence from source or measured results.

source

PerfChecker.write_comparison_json Method

write_comparison_json(result::BundleComparison, path)

Write canonical JSON comparison evidence. Create parent directories, replace the destination file and return its path. The input is saved evidence; this writer does not execute measurements.

source

PerfChecker.write_comparison_markdown Method

write_comparison_markdown(result::BundleComparison, path)

Write a Markdown comparison table and verdict. Create parent directories, replace the destination file and return its path. The input is saved evidence; this writer does not execute measurements.

source

PerfChecker.write_compatibility_report Method

write_compatibility_report(result::CompatibilityReport, path)

Write JSON preparation and compatibility evidence. Create parent directories, replace the destination file and return its path. The input is saved evidence; this writer does not execute measurements.

source

PerfChecker.write_folded_profile Method

write_folded_profile(bundle, path; metric="julia.cpu.samples",
                     case_id="", version=nothing)

Write matching saved stacks as folded stack/weight lines for flame-graph tools. Create parent directories, replace the file and return its absolute path. Reject an empty selection. This exports observations; it does not sample a process.

source

PerfChecker.write_investigation_notebook Function

write_investigation_notebook(path; root=pwd(),
                            catalog=joinpath(root, "perf", "scenarios.toml"),
                            project=root, force=false)

Load PerfCheckerPluto to write an editable investigation notebook and return its absolute path. The notebook activates project, reads the declared catalogue if present, and exposes explicit launch/cancel/refresh controls for discovery, scenario runs, analyzers and optional advisors. The controller needs PerfCheckerPluto and PlutoUI plus any selected tools/providers. Existing files raise ArgumentError unless force=true; parents are created. Generation does not open Pluto, execute target code or contact an advisor. Relative paths are resolved from the caller's working directory.

source

PerfChecker.write_investigation_report Method

write_investigation_report(payload::AbstractDict, directory; force=false)

Write canonical JSON and readable Markdown for a discovery, diagnosis, advice, scenario comparison/run, tool catalogue, narrative, investigation, advisor evaluation or scenario-sync payload. The schema determines the basename (for example diagnosis.json and diagnosis.md); return the two paths. Create directory as needed. If either destination is an existing file, throw ArgumentError unless force=true; force permits replacing both reports. Unsupported schemas raise an error, and serialization/filesystem failures propagate. Writes are sequential, so an interrupted export can leave one report. This renders supplied evidence without rerunning its workloads or advisors.

source

PerfChecker.write_julia_investigation Method

write_julia_investigation(investigation::JuliaRegressionInvestigation, directory)

Write julia-investigation.json and a Markdown attribution report in directory. Create the directory, replace these report files and return both absolute paths. This exports the supplied investigation without rerunning any runtime campaign.

source

PerfChecker.write_julia_runtime_campaign Method

write_julia_runtime_campaign(campaign::JuliaRuntimeCampaign, directory;
                             include_output=false)

Write julia-runtime-campaign.json and a Markdown summary in directory and return both absolute paths. Existing report files are replaced. Captured process output is omitted from JSON unless include_output=true. No measurements run.

source

PerfChecker.write_pprof_profile Function

write_pprof_profile(bundle, path; metric="julia.cpu.samples",
                   case_id="", version=nothing, max_samples=100_000)

Load PProf and FlameGraphs. Convert saved weighted stacks into a PProf file, returning its absolute path. The conversion approximates weights with repeated samples scaled by max_samples; it is not a fresh native-process profile. Reject nonpositive sample limits and profiles without positive weights.

source

PerfChecker.write_property_corpus Method

write_property_corpus(path, cases; producer="manual", metadata=Dict(), force=false)

Persist JSON-compatible property-generated cases before benchmarking them. The result is a frozen input artifact: generation and shrinking do not run in the timed worker. Return the absolute destination path, creating parents and writing schema perfchecker-property-corpus/1, producer, UTC creation time, case count, metadata with string keys and the cases. Existing files raise ArgumentError unless force=true. Cases and metadata must be JSON-compatible; serialization and filesystem errors propagate.

julia> mktempdir() do directory
           path = write_property_corpus(joinpath(directory, "inputs.json"), [[1, 2], [3]]);
           read_property_corpus(path)["count"]
       end
2

source

PerfChecker.write_run_bundle Method

write_run_bundle(bundle::RunBundle, directory)

Write canonical protocol JSON/JSONL plus SHA-256 integrity.json and an empty artifacts/ directory, then rename a temporary sibling to the absolute destination. Return that destination. Parent directories are created; any existing destination path raises ArgumentError. Temporary output is removed on success or failure. Artifact descriptors are serialized, but referenced artifact payloads are not copied by this writer. Serialization/filesystem errors propagate. Saving records does not validate their correctness, authenticate their origin or establish fresh measurement.

julia> bundle = RunBundle(Dict{String,Any}("schema_version" => "perfchecker-run-bundle/1", "state" => "complete"), [Dict{String,Any}()], Dict{String,Any}[], Dict{String,Any}[], Dict{String,Any}[]);

julia> mktempdir() do directory
           path = write_run_bundle(bundle, joinpath(directory, "run"));
           verify_run_bundle(path; require_integrity=true)["verified"]
       end
true

source

PerfChecker.write_scenario_workflow Method

write_scenario_workflow(path; catalog="perf/scenarios.toml", project="perf",
                        implementations=["cpu"], versions=["1"],
                        platforms=["ubuntu-latest", "windows-latest", "macos-latest"])

Create a GitHub Actions YAML workflow and return its absolute path. Any existing path raises ArgumentError; parent directories are created. Catalogue/project paths must be simple relative paths without ..; implementation names must be nonempty strings without commas, and version/platform lists must be nonempty. Only the three named hosted runner platforms are supported.

When later triggered, the workflow instantiates the declared environment, selects adopted implementations, runs scenario correctness/measurement and uploads reports. Writing it does not submit, execute or qualify CI. Serialization/filesystem errors propagate.

source

PerfChecker.write_software_suite_template Function

write_software_suite_template(root=pwd(); force=false)

Create perf/suite.jl, perf/features/smoke.jl and perf/runner/Project.toml for the package named in root/Project.toml. Return those three absolute paths. Require a package Project file with a name; missing metadata or an existing generated file raises ArgumentError unless force=true permits replacement. Create parents as needed.

The smoke workload loads the package and verifies its module identity. The runner declares dependencies but is not instantiated by this writer; prepare it before executing the generated suite. This writes Julia source/configuration without measuring or qualifying the package's feature behavior.

source

PerfChecker.write_speedscope_profile Method

write_speedscope_profile(bundle, path; metric="julia.cpu.samples",
                        case_id="", version=nothing)

Export matching saved stacks as a Speedscope sampled-profile JSON document. Weights use bytes for allocation metrics and unitless samples otherwise. Create parents, replace the file and return its absolute path; reject an empty selection.

source

PerfChecker.write_suite_bundle Method

write_suite_bundle(result, root; run_id=uuid4(), attempt_id=uuid4(),
                   evidence="fresh")

Convert a completed suite result into a portable bundle under root/run-<id>. Return the absolute bundle directory. The atomic writer rejects an existing destination. The evidence label is supplied by the caller; writing a bundle does not establish that previously cached observations were freshly measured.

source

PerfChecker.write_suite_json Method

write_suite_json(result::SoftwareSuiteResult, path)

Write JSON suite evidence including per-run qualification. Create parent directories, replace the destination file and return its path. The input is saved evidence; this writer does not execute measurements.

source

PerfChecker.write_suite_junit Method

write_suite_junit(result::SoftwareSuiteResult, path)

Write JUnit XML, mapping error/blocked/invalid runs to failures and unavailable runs to skipped cases. Create parent directories, replace the destination file and return its path. The input is saved evidence; this writer does not execute measurements.

source

PerfChecker.write_suite_markdown Method

write_suite_markdown(result::SoftwareSuiteResult, path)

Write a Markdown suite summary with verdict and run statuses. Create parent directories, replace the destination file and return its path. The input is saved evidence; this writer does not execute measurements.

source

PerfChecker.write_suite_notebook Function

write_suite_notebook(path; suite_path=nothing, factory=:build_suite,
    profile=:quick, project=dirname(Base.active_project()),
    result_path="results/suite-result.json", reports_root="results/notebook",
    force=false)

Load PerfCheckerPluto and generate an editable Pluto notebook. It selects a package, workload, collector and target, with explicit Launch, Cancel, Refresh and Save controls. Generation does not start Pluto or execute a workload. Without suite_path, the notebook only displays an existing saved result.

project must be a prepared controller environment containing PerfChecker, PlutoUI and the selected collectors. Relative report paths resolve beside the notebook; relative suite and project paths resolve from the caller's directory. Create parent directories and return the absolute notebook path. Reject an existing destination unless force=true explicitly permits replacement.

source

PerfChecker.write_suite_reports Method

write_suite_reports(result::SoftwareSuiteResult, directory;
                    formats=(:json, :markdown, :junit, :bundle, :version_series,
                        :version_comparison_json, :version_comparison_markdown),
                    relative_limits=Dict(),
                    min_samples=1, sample_statistics=Dict(),
                    default_sample_statistic=:median)

Export a completed suite and return written paths. Defaults include JSON, Markdown, JUnit, a run bundle, version series and version-comparison reports. The comparison options control evidence reduction and budgets. Ordinary report files are replaced; bundles use new run directories. No workload is rerun. relative_limits uses metric-string keys and fractional budgets; sample_statistics, default_sample_statistic and positive min_samples are forwarded to compare_suite_versions. Without limits, version comparisons remain diagnostic/inconclusive even for a passed execution. The return value is a vector of written file or bundle-directory paths.

# Export an already completed result, including its failures if present.
paths = write_suite_reports(result, "perf/results/local";
    relative_limits=Dict("julia.wall.time" => 0.05), min_samples=3)

source

PerfChecker.write_template Method

write_template(kind::Symbol; path=nothing, force=false) -> String

Write a Julia performance-checking template and return its path.

Supported template kinds are :benchmark, :chairmark, :alloc, and :pluto. The generated files use PerfConfig; no external configuration file format is introduced. Missing path uses the kind's filename under perf/; the returned string preserves that supplied/default path rather than forcing an absolute path. Create parents, rejecting existing files unless force=true. Unknown kinds raise ArgumentError; filesystem errors propagate. Writing a starter does not execute it or install its collector dependencies.

source

PerfChecker.write_version_comparison_json Method

write_version_comparison_json(result::VersionComparison, path)

Write canonical JSON version-comparison evidence. Create parent directories, replace the destination file and return its path. The input is saved evidence; this writer does not execute measurements.

source

PerfChecker.write_version_comparison_markdown Method

write_version_comparison_markdown(result::VersionComparison, path)

Write a Markdown version-comparison report. Create parent directories, replace the destination file and return its path. The input is saved evidence; this writer does not execute measurements.

source

PerfChecker.write_version_series_json Method

write_version_series_json(result::VersionComparison, path)

Write canonical JSON series, target availability and normalized plot descriptions under schema perfchecker-version-series/1. Create parent directories, replace the destination file and return its path. The input is saved evidence; this writer does not execute measurements.

source