Start from the SNS notification and the harmonia-<stage>-operations dashboard. Logs and queue depth explain symptoms; DynamoDB claims, receipts, and verification records remain the authority for what happened.
Contain the affected path
If an alarm can affect external side effects, pause the affected job or disable its capability before retrying anything. Do not delete SQS messages, purge a queue, redrive a dead-letter queue, or reissue a provider call until the persisted operation identity and effect state are known.
Triage by alarm
Reconcile effect-bearing work
For a claimed action without a conclusive receipt, read the official provider by the persisted idempotency and target identifiers. Record exactly one of: confirmed not dispatched, confirmed dispatched, confirmed provider result, or still unknown. Only the first state can return the action to an executable queue automatically.
Do not use a DLQ redrive as effect authorization. After the root cause is fixed, redrive only messages whose durable claim state permits another delivery; the worker must still reacquire the normal fence.
Close the incident
Resolve the incident only when the alert is healthy, affected queues are draining, service revisions are stable, persisted operations are terminal or visibly escalated, and an independent read proves the intended result. Record the exact commit, task/runtime revisions, trace and operation IDs, alarm history, root cause, remediation, and remaining gaps in private evidence. Last modified on September 14, 2026
