LTI Operations

Acceptance deployment and rollback

1. Run Deploy LTI dev from the stable workflow on main and select the feature branch in deploy_branch.
2. The workflow applies the reviewed public acceptance target manifest to both lti-dev and lti-dev-cron while preserving separately managed secrets and unrelated runtime configuration.
3. The workflow builds an immutable image, deploys a no-traffic candidate, and calls authenticated /system_health/readiness. Promotion is refused unless the database and acceptance target manifest are healthy.
4. After traffic promotion, verify the scheduler job uses the same image SHA and run one bounded signature smoke before enabling any provider scenario.
5. To roll back, rerun Deploy LTI dev with deploy_branch=main. The workflow restores the stable application image and scheduler image. Runtime secrets and target configuration are preserved.

If target readiness is unhealthy, do not bypass it. Correct the reviewed dev manifest in the workflow, redeploy a candidate, and retain the failed workflow as evidence. No step in this runbook changes a production service or production database.

Runtime Inventory

LTI currently has two operating models:

• testing.product.citizengo.net: legacy VM deployment using Apache/mod_wsgi, MySQL, and a root cron invoking run_cron.sh.
• lti-dev and lti-prod: Cloud Run services using PostgreSQL, with lti-dev-cron and lti-prod-cron Cloud Run Jobs.

The Cloud Run services are the migration target. The VM remains the public production authority until an explicit parity and cutover decision is approved. Do not treat a successful lti-prod deployment as a VM production cutover.

Health Contracts

All diagnostics except the legacy resource widget require a deployment-health Google identity. Never place tokens in a URL or log their values.

Scheduler Triage

1. Confirm the Cloud Scheduler trigger is enabled and points to the intended Cloud Run Job.
2. Confirm the latest Job execution completed.
3. Read /system_health/readiness and compare oldest_due_age_seconds with stale_after_seconds.
4. Review ScheduledTask and ScheduledTest in Django administration. A performed row is immutable; repair by scheduling a new row.
5. Review Cloud Run Job logs for lock acquisition, selected record IDs, handler outcome, recurrence, and Asana reporting outcome.
6. Do not run the VM cron and Cloud Run Job against the same database during a cutover unless the database advisory lock has been verified on both paths.

The supported acceptance trigger is lti-dev-cron-trigger in europe-west1. It invokes the lti-dev-cron Cloud Run Job every three minutes. The legacy LTI_Test_Scheduler job in europe-west4 targeted the removed /runcrons HTTP endpoint and is paused. Do not resume it; restore the supported trigger instead if acceptance scheduling stops.

FRAPI Queue Monitoring

The queue monitor runs in LTI acceptance and reads frapi_cgo.request through the existing direct, read-only FRAPI database identity. It does not mutate the queue and does not read request payloads or identifiers.

The acceptance scheduler idempotently creates one pending monitor on its next tick. Operators can also seed or bring the first run forward explicitly:

python manage.py ensure_frapi_queue_monitor --run-now

The command and automatic seed both refuse non-acceptance targets. To execute the first check in the same bounded operation, add --execute. Subsequent successful, warning, or unhealthy runs schedule one successor at the configured minute interval.

Operator triage for an unhealthy run:

1. Read the sanitized task log or authenticated /system_health/frapi/queue response.
2. Confirm whether stale work or the recent failure threshold caused the state.
3. Use the aggregate module/service groups to identify the responsible worker.
4. Inspect FRAPI and worker logs through the approved production support path.
5. Do not retry, update, or delete queue rows from LTI. Recovery remains owned by the FRAPI operator.

If the task reports FRAPI queue database connection failed, verify the non-secret Cloud SQL attachment on both lti-dev and lti-dev-cron, then verify the secret-backed read-only database settings without printing their values. A connection-category failure is an LTI runtime dependency, not a FRAPI queue-health result.

If the task reports FRAPI queue database read timed out, first confirm the deployed monitor uses the indexed query contract from spec 016. Do not increase the timeout as the first response: retain the bounded read, inspect the query plan through the approved FRAPI operator path, and treat repeated timeouts as a monitor-performance incident.

Before rolling LTI acceptance back to a revision that predates this task type, disable the pending recurrence while the feature image is still active:

python manage.py ensure_frapi_queue_monitor --disable

Confirm that the command deleted the pending monitor, then redeploy stable main. Completed history can remain. No FRAPI rollback is needed because the monitor is read-only.

Payment Gateway Monitoring

The existing Payment Gateway Healthcheck maintenance task is the reporting automation. It reads FRAPI donation status counts for Stripe, PayU, PayPal, and GoCardless, writes monthly standard and rejection reports, recurs daily, and reports failed executions to Asana.

This task is a transaction-status monitor, not a provider synthetic probe. It does not prove that a new checkout can be completed. Provider sandbox journeys must remain separate scheduled tests with synthetic identities and test instruments.

Daily operator checks:

1. Verify a pending recurring Payment Gateway Healthcheck task exists.
2. Verify its latest performed predecessor succeeded within 26 hours.
3. Review /pg_report for missing providers, abrupt volume changes, and rejection spikes.
4. Treat a missing run, query failure, or stale queue as an incident even when the web service remains healthy.

Cloud Run IAP Rollout

Use direct Cloud Run IAP on lti-dev first. This avoids a load balancer and keeps the run.app endpoint behind Google sign-in.

1. Inventory service invokers, scheduler identities, deploy identities, and health-check identities.
2. Enable direct IAP on lti-dev.
3. Grant IAP access to an approved CitizenGO Google group. Prefer a group over the entire domain.
4. Verify browser sign-in, scheduled Job execution, deployment smoke, Agent API clients, and staff administration.
5. Keep Django users and permissions as the authorization source during the first phase. Do not delete users or password fields.
6. In a separate reviewed change, map the verified IAP identity to a Django user and then disable password login.
7. Roll out to the Cloud Run production candidate only after acceptance evidence is approved. This does not change the VM deployment.

CI/CD Recovery

GitHub Actions must be operational before accepting a merge:

• CI is required for lint, Django validation, unit tests, local-agent tests, and both Docker build contracts.
• Automatic Cloud Run production deployment runs only after CI succeeds for the same main commit.
• A failed or unstarted Actions run is not a passing gate.
• Configure branch rules to require the CI jobs and one human approval.
• Replace the long-lived GCP_SA_KEY with Workload Identity Federation in a dedicated credential-hardening change.

Incident Ownership

• LTI web/database/scheduler: Software team on-call.
• Donation journey or FRAPI provider diagnostics: Donations platform owner.
• Salesforce verification: Salesforce integration owner.
• GitHub Actions billing, organization rules, and Google IAM: Engineering manager or organization administrator.
• Asana alert delivery: LTI owner, with manual escalation if reporting itself fails.

Known Schema Compatibility Repair

Older Agent API installations may retain the obsolete token_prefix, scope_used, and payload_hash columns as required PostgreSQL fields. Current models do not write those fields, so stale constraints can reject API client or audit inserts even though the request itself continues. Migration 0019_relax_legacy_agent_api_columns preserves existing values and only makes those obsolete columns nullable. Validate this migration on lti-dev before promoting the same image.