Handling Out-of-Order Webhook Events

Handling out-of-order webhook events safely

Webhooks can arrive out of order — retries, network delays, and independent endpoint runs mean you must design consumers to handle late or duplicate events. This guide explains practical Laravel patterns to handle out-of-order webhook events and keep state correct.

You will implement an apply-if-newer pattern (store occurred_at), add idempotency/dedupe, and test locally and end-to-end. The primary keyword “handle out of order webhook events” is used in the recommended design.

Why ordering is not guaranteed

Understand the sender behavior so you don’t assume ordering.

Retries use exponential backoff (late arrivals happen)

Retries are normal and will cause older attempts to show up after newer ones.

Multiple endpoints + independent runs per endpoint

Different endpoints or consumers may process events at different speeds, producing apparent reordering.

Micro checklist:

• Expect at-least-once delivery semantics.
• Do not assume monotonically increasing arrival times.
• Store occurred_at in your consumer to compare event freshness.

Trigger two events from the Sample Project and confirm attempts + timing in Message Log.

Choose your ordering contract

Pick one contract and implement it consistently across services.

Strict ordering (rarely worth it)

Requires global coordination (locks, single-queue). Only choose when business logic absolutely depends on strict ordering.

“Last-write-wins with versions”

Store a version/timestamp and apply updates only if the incoming event is newer.

Event-sourcing style apply-if-newer

Persist events to an inbox and project state from the canonical event stream; apply only when appropriate.

Common gotcha: Treating the first-arrived event as canonical—use occurred_at or an explicit version to decide.

Minimum safe design

Implement these minimum protections to remain correct under retries and reordering.

Idempotency / dedupe for at-least-once delivery

Use a dedupe key (message id or hash of payload) to ensure side effects are not applied more than once.

Store occurred_at and “entity version”

Keep the event timestamp or version in your state so you can reject or ignore older events.

Micro checklist:

• Persist the raw event to an inbox table.
• Record occurred_at on state rows.
• Compare and apply only if incoming occurred_at > current last_event_at.

Laravel reference implementation

Practical, minimal example: store inbound events and apply only if newer.

Inbox table

Store raw events in webhook_inbox for audit and replay.

Apply rules with optimistic checks

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

// Example assumes your payload has: entity_id, occurred_at (ISO8601), and status.

Route::post('/webhooks/sendpromptly', function (\Illuminate\Http\Request $r) {
    $raw = $r->getContent();
    $data = json_decode($raw, true) ?: [];

    $entityId = $data['payload']['entity_id'] ?? null;
    $occurredAt = $data['payload']['occurred_at'] ?? null;

    if (!$entityId || !$occurredAt) {
        // treat as bad contract (you may DLQ instead of rejecting)
        return response()->json(['error' => 'missing_fields'], 422);
    }

    \DB::transaction(function () use ($entityId, $occurredAt, $raw) {
        \DB::table('webhook_inbox')->insert([
            'entity_id' => $entityId,
            'occurred_at' => $occurredAt,
            'body' => $raw,
            'created_at' => now(),
        ]);

        // Apply only if event is newer than current state (LWW pattern)
        $current = \DB::table('entity_state')->where('entity_id', $entityId)->lockForUpdate()->first();

        if (!$current || strcmp($occurredAt, $current->last_event_at) > 0) {
            \DB::table('entity_state')->updateOrInsert(
                ['entity_id' => $entityId],
                ['last_event_at' => $occurredAt, 'updated_at' => now()]
            );
        }
    });

    return response()->json(['ok' => true], 200);
});

Suggested diagram: A sequence showing “deliveries (t1,t2,t3)” arriving out of order, write to inbox, then project state only if occurred_at is newer.

Testing + failure modes

A) Direct local test (simulate out-of-order)

Send newer then older (same entity_id) and confirm state remains at the newer timestamp:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

curl -i -X POST http://localhost:8000/webhooks/sendpromptly \
  -H "Content-Type: application/json" \
  -H "X-SP-Timestamp: 1700000000" \
  -H "X-SP-Signature: <valid>" \
  -d '{"event_key":"x","payload":{"entity_id":"E1","occurred_at":"2026-02-14T10:05:00Z","status":"paid"}}'

curl -i -X POST http://localhost:8000/webhooks/sendpromptly \
  -H "Content-Type: application/json" \
  -H "X-SP-Timestamp: 1700000001" \
  -H "X-SP-Signature: <valid>" \
  -d '{"event_key":"x","payload":{"entity_id":"E1","occurred_at":"2026-02-14T10:01:00Z","status":"pending"}}'

Expected: state remains at 10:05:00Z.

B) SendPromptly end-to-end validation

Trigger events via the API and confirm delivery runs in Message Log.

Common failure modes

1. Assuming order ⇒ “pending” overwrites “paid”.
2. No idempotency ⇒ retries duplicate side effects.
3. No stored occurred_at ⇒ can’t resolve late arrivals safely.
4. Processing inline (slow) ⇒ timeouts ⇒ retries ⇒ even more reordering.
5. Mixed clocks / wrong timezone parsing ⇒ incorrect comparisons.
6. Rejecting older events with 5xx ⇒ pointless retries.

Related

• Retries and success criteria (2xx)
• Idempotency model (24-hour TTL) for ingestion
• Dedupe patterns for webhook consumers
• Debugging replays in logs

If state looks wrong, inspect per-attempt timing in Message Log and verify your apply-if-newer rule.

Conclusion

• Expect at-least-once delivery and design for out-of-order arrivals.
• Persist raw events to an inbox and project state using occurred_at comparisons.
• Prefer apply-if-newer (LWW) or event-sourced replays over strict ordering.
• Keep processing fast and idempotent to avoid retry storms.
• Use Message Log and the Sample Project for deterministic tests.