- Executive Summary
- Author A Scenario
- Preview And Execute
- Interpret And Troubleshoot
- Evidence And Handoff
- Recovery And Cleanup
Executive Summary
Use a small versioned scenario with synthetic data, explicit acceptance targets and observable expected results. Preview it before scheduling. A blocked prerequisite is not a product failure or a pass. Retain one correlated result and enough redacted evidence for the application owner to reproduce it safely.
This runbook adds no production authorization. Commands below must run in an approved acceptance runtime with its normal settings and secret bindings, never by copying production credentials locally.
Author A Scenario
- Read the feature specification and existing matrix template selected by
generate_scenario_matrix --help. Reuse a matching scenario before adding another. - Identify list/locale, entry point, one-time/recurring mode, provider, expected persistence changes and external side effects. Use stable non-personal scenario slugs.
- Record the acceptance target manifest, synthetic fixture owner, expected result and cleanup contract. Leave missing integration approval blocked, with an owner.
- Add ordered steps using the execution contract. API steps are GET/HEAD; browser steps support navigation and the existing click/fill/select/wait assertions. Do not embed credentials or real donor data.
- Budget the whole scenario to at most 600 seconds and 20 steps; each step is at most 60 seconds. Reads may retry within the bounded contract; a submit/payment action must not be blindly replayed.
- Add deterministic unit tests for success, failure, missing prerequisites and production-target rejection. Run the CI-listed modules with
main.settings_ci, not uncontrolled legacy live tests. - Raise a PR with the spec, scenario changes and redacted example output. Application/data owners review assertion and cleanup semantics before a hosted run.
Preview And Execute
These commands preview configuration or produce a matrix; they do not approve a live run:
python manage.py generate_scenario_matrix --output /tmp/lti-matrix.json
python manage.py prepare_acceptance_signature_scenarios
python manage.py prepare_acceptance_donation_scenarios
Review every blocked item and target before using --apply. The signature command requires a future timezone-qualified --schedule-at; use the same explicit scheduling convention for donations. --apply writes to the acceptance catalogue/queue and is not a dry run.
Use the existing serialized acceptance scheduler or approved regression workflow to execute. The read-only-smoke suite checks reachability only. The cross-stack-donation-recovery suite prepares and executes approved donation scenarios and therefore has sandbox side effects. Do not run it simply to test this documentation.
Capture candidate SHA, scenario version, execution timestamp, target identities and source report URL. Confirm the run has reached a terminal state; a queued task or generated matrix is not execution evidence.
Interpret And Troubleshoot
| Observation | First Check | Next Action |
|---|---|---|
| Rejected target or unexpected hostname | Target manifest and effective backend/FRAPI URL | Stop; platform owner corrects acceptance configuration |
| No executable scenarios | Approval/fixture/provider prerequisites | Record blocked, ask the prerequisite owner |
| Signature succeeds but email gate fails | Separate signature result from Iterable mapping/delivery evidence | Keep signature outcome; escalate the email gate separately |
| UI timeout or missing control | Screenshot, page state, bounded network metadata | Classify as unclassified until product/environment/selector cause is established |
| Donation success without persistence evidence | Baseline, correlation ID, permitted lag, provider sandbox result | Hold release; do not submit again before reconciling the first attempt |
| Recurring scheduler failure | Existing run state and owner-approved schedule | Do not start a parallel runner; retain evidence and investigate |
| Read-only smoke passes on 401/404 | Reachability semantics | Do not report business behavior as tested |
Use the taxonomy for routing. A timeout is not automatically a flake. A passing rerun alone cannot erase the original failure.
Evidence And Handoff
Render a retained report without contacting any service:
python -m test_interface.run_summary report.json --previous previous.json --output summary.md
Attach the redacted summary to the controlled release/incident record. Include source run and ScheduledTest references, candidate SHA, environment, outcome, owner and next action. Never paste raw payloads, personalized URLs, payment credentials, cookies or donor records into Asana.
Recovery And Cleanup
Stop further scheduling if targets or side effects are unsafe. The preparation commands' --disable options affect their pending generated scenarios, not completed evidence or remote records; review scope before use. Do not delete unrelated tests or use broad SQL cleanup.
Reset only the approved synthetic identities and records correlated to the run, following the synthetic-data contract and Ultra cleanup runbook. Provider cancellation, email changes and remote cleanup require their own approval. Preserve release/incident evidence under the retention policy.