Executive Summary
Separate execution outcome from root cause. Outcomes are passed, failed or blocked. Classifications route investigation to the right owner; they do not change the outcome or release decision. Use unclassified when evidence is insufficient instead of blaming the application or dismissing a problem as a flaky test.
Classification Rules
| Category | Evidence Required | Example | Follow-Up Owner |
|---|---|---|---|
product_bug |
Valid fixture/environment; application violates an agreed contract | Confirmed signature not persisted despite successful response | Application owner |
environment_issue |
Target/configuration/connectivity or runtime dependency failure | Acceptance allowlist rejects the intended target | LTI/platform owner |
data_mismatch |
Correlated expected and observed values disagree; fixture/mapping ownership established | Sandbox list or message-type mapping differs from approved contract | Data and affected service owner |
payment_provider_issue |
Sanitized sandbox provider result establishes provider/integration failure | Approved sandbox method rejected at provider boundary | Payment integration owner |
test_flake |
Reproducible nondeterminism in test machinery with unchanged valid target/data | Proven stale locator timing, not an application regression | LTI scenario owner |
unclassified |
Evidence missing, contradictory or ambiguous | Timeout, HTTP 500, assertion failure without diagnosis | LTI triage owner |
Blocked is an execution state, not a root cause. For example, missing credentials can be blocked/environment_issue; an unexplained approval dependency can remain blocked/unclassified. A recovered flow remains distinct from the original failed operation.
Triage Procedure
- Confirm environment, candidate SHA, scenario version and run correlation.
- Preserve redacted evidence and the original outcome before any rerun.
- Record category, reason, named owner and source evidence in the controlled task. Classify only the affected check, not the entire service.
- Permit a rerun only when side effects are understood and cleanup/idempotency is safe.
- Close the investigation with the change and validation evidence, or retain an explicit unresolved dependency.
Existing Sentry labels (test_locator_fragility, environment_dependency, expected_filtered, product_defect) are intake hints, not confirmed taxonomy. Do not mechanically treat expected_filtered as success or test_locator_fragility as a proven flake.
Reporting Contract
The offline summary accepts optional reviewed failure_category values from this table on each non-passing check. It does not edit the source report. Without a reviewed category it recognizes only exact TargetEnvironmentError, URLError and ConnectionError classes as environment signals. Timeouts, assertions and all unknown classes remain unclassified. Messages are never parsed for keywords or emitted to the summary.
This classification is triage guidance, not provider adjudication or permission to promote a release. Outcome counts and the existing gate remain unchanged. See summary usage.