AOS records workflow, authorization and execution evidence. Use this guide to compare that evidence with a separate collector and check whether missing events, conflicting outcomes or clock uncertainty could mislead an investigation.
Available in 0.2.0-rc15. Compare supplied records through the CLI, API or console. Analysis is read-only; uploaded records are not stored. Collector identity and clock calibration are verified separately.
Run a local fault exercise
After installing AOS, activate the same virtual environment and run:
aos observability validate --output ./observability-run --tenant free-evaluation
python -m json.tool observability-run/validation.json
Use a new output directory. The command starts a disposable loopback HTTP service, makes two real calls, records a server-side effect journal separately from client observations and stops the service. One call has a deliberate 40 ms delay. It also analyzes copies of the captured evidence with an event removed, an event duplicated, a timestamp shifted and clock calibration removed. It does not change your machine’s clock, firmware or network settings.
Expect status: passed, effect_count: 2, and reports for five cases. The overall pass means the expected faults were detected. Individual fault cases should be inconsistent or inconclusive. Client and server run in one process. Recorded durations describe this local exercise; comparisons across deployments need a separate benchmark.
The directory contains private evidence.json and validation.json files. Existing output is never overwritten. No adoption reporting is sent by these commands.
Compare through CLI or console
Offline CLI:
aos observability analyze ./observability-run/evidence.json ./comparison.json
Exit code 0 means consistent supplied evidence, 3 means inconsistent or inconclusive evidence, and 2 means invalid input or file handling failure.
For a running AOS session, generate evidence using the same tenant shown in the console session. For example, the local getting-started session uses free-evaluation. An operator or compliance auditor can submit it:
export AOS_SESSION="$PWD/interactive-run/operator-session.json"
aos product --output remote-comparison.json observability analyze --file ./observability-run/evidence.json
The remote command’s exit code describes the API request; inspect data.status for the evidence result.
Console: connect using your operator session, choose Audit → Compare observations, select evidence.json, and choose Submit. The status message gives matched, missing and order-conflict counts. The complete report downloads as aos-observability-report.json. It uses the same authenticated API as the CLI: POST /v1/observability/analyze. Tenant mismatches and unauthorized roles are rejected.
Connect external measurements
Normalize selected records into aos.observability-input/v1. The batch contains tenant_id, a UUID run_id, observations and witnesses. Each array accepts at most 500 records; CLI and console uploads are limited to 1 MiB. Match records using shared event IDs from both collectors. Similar timestamps alone do not establish a match.
Each record has:
| Field | Meaning |
|---|---|
event_id, trace_id, execution_id |
UUID identifiers shared by both collectors for the same event and execution. |
source_id |
Short collector identifier; different names alone do not prove independence. |
layer |
agent, model, runtime, network, firmware, hardware or physical. |
signal |
Short signal name, such as temperature or reference.effect. |
outcome |
observed, success, failure or unknown. |
timestamp |
Timezone-aware timestamp from the source. |
clock_offset_ms |
Source clock minus reference clock, or null when uncalibrated. |
uncertainty_ms |
Bound on the corrected timestamp, or null when unknown. |
value, unit |
Optional finite measurement pair. Units: ms, s, C, W, V, percent, count, ratio. |
parent_event_id |
Optional explicit causal predecessor within the same trace and execution. |
Use the generated evidence.json as a complete schema example. Measurements compared across collectors must use the same unit and exact normalized value; this version has no sensor-tolerance model. Independently sampled values may differ legitimately and require investigation. Repeated event IDs are flagged, not silently merged. Missing witnessed events, conflicting outcomes/values, cycles, broken links and definite reversed clock intervals are reported. Overlapping intervals or unknown calibration produce uncertainty rather than a false ordering conclusion.
A report includes an input SHA-256 for reproducibility. A hash is not a signature or proof of origin. Protect the original evidence and validate collector identity, signatures and calibration separately. Identifiers must not contain secrets, personal identifiers or customer payloads. No arbitrary metadata or credential fields are accepted.
Runtime signal provenance
Runtime snapshots now include collection_mode and per-field measurement_status where available. The psutil collector distinguishes measured memory/GPU data, reported scheduler/latency data, estimated child-process counts and unavailable metrics. Compatibility numeric defaults remain, but zero with unavailable is not a measured zero. An unavailable provider is labeled unavailable; deliberate mock mode is synthetic. Older providers may be unspecified.
These labels do not change admission policy automatically. Require appropriate measured inputs in deployment-specific controls before making a safety or capacity decision.
External data collection
Use your infrastructure collectors to capture hardware, firmware and physical measurements, including their calibration and timing uncertainty. Normalize the selected records into the input schema above.
AOS checks consistency between those records. Sensor collection, clock calibration and root-cause investigation remain part of the surrounding monitoring system. Validate workflow results separately from the completeness and accuracy of its telemetry.