Review service incidents and notification delivery

Worked example

Example: the receiver accepted a recovery but its response was lost

A missing HTTP acknowledgement leaves delivery uncertain. Preserve the notification ID and check the receiver before deciding what happens next.

Procedure

Follow the task in order

1. 1
   Open the existing service diagnosisOpen the bell near your user menu and choose Inspect service or Health & incident history. Services and instance diagnostics also open the shared health dialog. Brief preparation or archive write delays are qualified before warning; an unavailable health check does not prove recovery. Expand Incident history for the recorded observer/controller episode.
2. 2
   Review personal recipientsAn administrator expands Notification recipients, selects up to ten active people and saves the revision-bound selection. Only future incident openings and recoveries are queued. A selected recipient reads notices through the bell and marks their own notice as read. Unselected users do not receive the notice. A failed check retains last known notices.
3. 3
   Configure an approved external receiverOn the observer controller configure an exact HTTPS destination and a signing-key reference before restarting it. Keep ENV/Infisical credentials node-local. No arbitrary URL or secret is entered into the health dialog. Open that observer controller to review its targets.

   Node-local configuration fragment; provide the referenced key through the existing secret provider

   {
     "serviceNotifications": {
       "webhooks": [
         {
           "id": "operations",
           "label": "Operations receiver",
           "url": "https://receiver.example.invalid/openhub-events",
           "signingSecret": {
             "provider": "env",
             "key": "OPENHUB_WEBHOOK_SIGNING_KEY"
           }
         }
       ]
     }
   }

4. 4
   Review and enable the webhookExpand External notifications · Webhook, select the configured destination and choose Review and save webhook. New incident/recovery events enter a durable queue. Existing events are not backfilled. Destination or key changes require a fresh review and invalidate pending delivery from the old selection. Disable the channel to stop pending sends after an already approved in-flight attempt.
5. 5
   Read the delivery resultExpand Recent deliveries. Acknowledged by receiver requires a matching receiver acknowledgement; a transport timeout or exhausted retry budget is not proof of rejection. The first automatic round is limited to eight attempts. HTTP requests have bounded time/response size, redirects are rejected and retryable failures use backoff. Delivery status is separate from the current incident and personal inbox.
6. 6
   Review an uncertain or failed deliveryAs an administrator select Review delivery, check this exact ID at the receiver and record one decision: verified acceptance, close without resending, or allow up to eight more attempts with the same ID. Verified acceptance is shown as an administrator decision, not fabricated HTTP acknowledgement. Retrying requires the original target/key review to remain valid. At most two extra rounds are allowed; the total attempt counter is retained and cannot exceed 24.
7. 7
   Recover after a lost decision responseUse Retry same decision. It reuses the immutable request receipt, even if a worker subsequently completed the delivery. Do not create another request to bypass an unknown outcome. A definitively rejected stale decision reloads current state for a new review. In-flight, already acknowledged or resolved deliveries cannot be changed by a stale decision.

Expected result

What you should have

Boundaries

Important boundaries

• Delivery is at least once. Remote exactly-once effects require receiver-side durable deduplication; repeating the same ID must not repeat its business effect.
• The minimal envelope contains service/controller/incident identifiers and event time, not topic paths, measurement values, credentials or recipient addresses.
• Incident, notification and decision evidence is retained by default. Recent UI lists are bounded; they are not a complete export. No automatic destructive retention purge is enabled.
• Use the observer controller for webhook administration. This guide does not claim cross-controller transport failover, email/SMS or direct Slack/Teams compatibility.

FAQ

Common questions

Related guides

Continue from here