Skip to content

Out-of-Order Webhooks: Prevent Stale Business Status Updates

Read this article in ไทย 中文

Out-of-Order Webhooks: Prevent Stale Business Status Updates

Arriving later does not mean newer

A system receives a payment confirmation, then returns to awaiting payment without a cancellation. One possible cause is an older webhook arriving later and overwriting newer state. Investigate event ordering and state-transition rules rather than only increasing server speed. This also differs from receiving the same event more than once.

A webhook sends an event notification from a source system to a destination. Event creation, delivery, queue acceptance and processing completion happen at different times. Stripe explicitly does not guarantee delivery in creation order and describes retrieving missing objects through its API. This is one provider example; read the actual provider contract before designing the integration.

Separate duplicates from stale updates

Hypothetically, event A describes an order awaiting payment and B describes it as paid. If B is processed first, an event-ID duplicate check still accepts A because it has a different ID, allowing state to move backwards even without duplicate records. Conversely, ordering alone does not prevent issuing the same receipt or dispatching goods twice. Both controls are needed.

Record at least the event ID, business-object ID, event type, source creation time when available, arrival time, processing result and downstream correlation ID. This reveals which event changed which record; a generic success message is insufficient. Limit personal data and log access to what is necessary.

Choose an authoritative state and explicit rules

If the source provides a reliable API for current state, a webhook can trigger retrieval of the latest object before updating the destination under explicit rules. This reduces reliance on old snapshots but adds requests and must handle latency, rate limits or state not yet visible. If retrieval fails, retain a reasoned pending task with controlled retries instead of guessing success.

Where the source guarantees a monotonic version for the same object, compare versions before writing. Do not infer order from event-ID appearance or arrival time alone. Timestamps may tie or come from different clocks, and business state is not always linear: refunds or reopened work may be valid. A blanket rule forbidding all backwards-looking changes can reject legitimate events.

Queues need concurrency controls too

Even with a queue, two workers can read the same state and overwrite each other. Developers should serialize work per object or use write-time version checks and database transactions as appropriate. Verify webhook signatures according to provider guidance and durably record work before acknowledging receipt. An acknowledgement does not prove the business operation has finished.

External effects such as dispatching goods, sending email or issuing documents need a reference to completed work and reconciliation when the destination response is ambiguous. Do not replay all events into production without checking existing effects. The Chawinnix further-reading article on duplicate submissions explains that control, which must be combined with the ordering and state rules discussed here.

Test business outcomes, not only HTTP success

Test normal order, reversed order, duplicates, concurrent delivery and failed API reads. For the same test order, verify that final state agrees with the source, external effects occur only as intended and unresolved work remains visibly pending. Preserve before-and-after results and logs to distinguish data defects from worker defects.

Before launch, the process owner should approve the authoritative source, allowed transitions, ownership of pending tasks and how to stop external effects when discrepancies appear. Chawinnix lists IT systems and automation services. Sanitized sample logs and an integration diagram provide a clearer starting point than simply reporting that the system is slow.

Conceptual event cards following separate paths through a checkpoint into a database

Sources

Back to all articles

Start the conversation

Tell us briefly what concerns you and our team will get back to you. What you send stays confidential.