Rotate Webhook Signing Secrets Without Downtime

Rotate Webhook Signing Secrets Without Downtime (Laravel Pattern)

Secret rotation is routine — but if you swap webhook signing secrets instantly you can break production traffic. This guide shows a pragmatic Laravel pattern to rotate webhook signing secrets without downtime using a “dual-secret window”, safe comparisons, and an operational cutover checklist so deliveries continue to succeed.

You will implement: middleware that verifies against multiple secrets, a short cutover runbook, and smoke tests to confirm 2xx runs in Message Log. The primary keyword “rotate webhook signing secret without downtime” is used in the recommended strategy below.

What SendPromptly signs (and what you must verify)

Follow these rules for correct verification and to avoid accidental mismatches with rotated secrets.

Signature headers you should validate (X-SP-Timestamp, X-SP-Signature)

• X-SP-Timestamp — Unix timestamp used for replay-protection.
• X-SP-Signature — HMAC-SHA256 over the raw request body (format: hex). Use hash_equals() to compare.
• Prefer X-SP-Message-Id for dedupe/correlation when present.

Webhook signature headers + delivery rules.

HMAC-SHA256 over the raw request body + constant-time compare

Compute HMAC on the raw body and compare with hash_equals(); never sign parsed JSON (parsers can reorder/normalize).

Micro checklist:

• Use $request->getContent() for the raw body.
• Compute hash_hmac('sha256', $raw, $secret).
• Compare with hash_equals().
• Reject missing/invalid timestamp headers.

Why rotation breaks prod (common root causes)

Rotation failures usually come from configuration or timing mistakes — not the signing algorithm.

Only one secret configured

If receivers are updated to the new secret before senders are issuing it (or vice versa) deliveries fail with invalid_signature.

Old and new secrets cut over at different times

Half the fleet accepts only the old secret and the other half only the new secret — this split-brain produces intermittent failures.

Mini incident: A payments service rotated its webhook secret during a deploy; one AZ received the new secret first and began rejecting X-SP-Signature values from SendPromptly until the remaining servers were redeployed.

The “dual-secret window” strategy (recommended)

Accept signatures signed by either the previous secret or the new secret during a short cutover window, then remove the old secret once you confirm traffic is using the new secret. This is the safest way to rotate webhook signing secret without downtime.

Accept signatures from {old, new} during cutover

• Configure receivers with an ordered list of active secrets (new first or old first — your choice as long as both are accepted).
• Verify the incoming signature against any secret in the list using hash_equals().

Remove old after you confirm traffic on new

• After confirming deliveries succeed with the new secret (Message Log verification + metrics), remove the old secret from the list.

Micro checklist:

• Add new secret to SP_WEBHOOK_SECRETS alongside the old.
• Deploy verification that checks against the comma-separated list.
• Confirm 2xx deliveries for new-secret traffic in Message Log.
• Remove the old secret after at least one successful run from the Sample Project.

Laravel middleware implementation

Implement your receiver to accept a comma-separated list of secrets and verify the signature against any entry.

Parse secrets list

Supply both secrets in your environment (example: SP_WEBHOOK_SECRETS=old_secret,new_secret). The middleware below expects a comma-separated env var.

Verify against any secret

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

<?php
// app/Http/Middleware/VerifySendPromptlyWebhook.php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;

class VerifySendPromptlyWebhook
{
    public function handle(Request $request, Closure $next)
    {
        $raw = $request->getContent();

        $sig = (string) $request->header('X-SP-Signature', '');
        $ts  = (string) $request->header('X-SP-Timestamp', '');

        if ($sig === '' || $ts === '') {
            return response()->json(['error' => 'missing_signature_headers'], 401);
        }

        // Optional best practice: reject very old timestamps (replay protection).
        // (Pick your tolerance window; keep it consistent across services.)
        if (!ctype_digit($ts)) {
            return response()->json(['error' => 'invalid_timestamp'], 401);
        }

        $secrets = array_filter(array_map('trim', explode(',', (string) env('SP_WEBHOOK_SECRETS', ''))));
        if (empty($secrets)) {
            return response()->json(['error' => 'webhook_secret_not_configured'], 500);
        }

        $ok = false;
        foreach ($secrets as $secret) {
            $expected = hash_hmac('sha256', $raw, $secret); // raw body per docs
            if (hash_equals($expected, $sig)) {
                $ok = true;
                break;
            }
        }

        if (!$ok) {
            return response()->json(['error' => 'invalid_signature'], 401);
        }

        return $next($request);
    }
}

Common gotcha: Put this middleware early in the kernel so nothing consumes the request body first; also ensure SP_WEBHOOK_SECRETS contains a comma-separated list with no extra logging of secrets.

1
2

Route::post('/webhooks/sendpromptly', fn () => response()->json(['ok' => true], 200))
    ->middleware(\App\Http\Middleware\VerifySendPromptlyWebhook::class);

Operational runbook

Follow this short, repeatable cutover checklist when rotating secrets.

Step-by-step cutover checklist

1. Add the new secret to SP_WEBHOOK_SECRETS (keep the old secret present).
2. Deploy receivers that accept multiple secrets (no traffic interruption expected).
3. Update SendPromptly to include the new secret (or schedule the rotation on SendPromptly side).
4. Wait and confirm deliveries signed with the new secret — watch Message Log and delivery metrics.
5. After at least one successful 2xx run for the new-secret delivery, remove the old secret from SP_WEBHOOK_SECRETS and redeploy.
6. Monitor for invalid_signature spikes for the next hour.

Micro checklist:

• Start with config change only (no removals).
• Verify new-secret deliveries in Message Log before removal.
• Roll back immediately if signature errors spike.

How to validate in Message Log

• Filter by the webhook endpoint and check recent delivery attempts.
• Confirm at least one successful attempt that corresponds to the new-secret timestamp/window.

Trigger a delivery from the Sample Project, then confirm the new-secret traffic in Message Log.

Testing

A) End-to-end SendPromptly test (ingestion → delivery runs)

Trigger a real SendPromptly event (see Getting Started). Expected:

• Ingestion returns 201 on success.
• Webhook delivery success shows HTTP 2xx; failures retry with exponential backoff.
• Confirm runs in Message Log and verify the run used the new secret.

See Send test event + confirm in Message Log: /docs/getting-started/.

B) Local “signature only” smoke test

Generate signature locally (replace old_or_new_secret_here with one secret from your SP_WEBHOOK_SECRETS list):

1
2
3
4
5
6
7
8
9

BODY='{"ping":"rotation-test"}'
SIG=$(php -r '$b=getenv("BODY");$s=getenv("SECRET");echo hash_hmac("sha256",$b,$s);' \
  BODY="$BODY" SECRET="old_or_new_secret_here")

curl -i -X POST http://localhost:8000/webhooks/sendpromptly \
  -H "Content-Type: application/json" \
  -H "X-SP-Timestamp: 1700000000" \
  -H "X-SP-Signature: '"$SIG"'" \
  --data "$BODY"

Expected: 200 {"ok":true}

Secret rotation failure modes

1. Rotation swaps secret instantly (no overlap) → webhook failures.
2. Service A updated, service B not → split-brain verification.
3. Verifying against parsed JSON instead of raw body → mismatches.
4. Logging secrets/signatures during debugging → security leak risk.
5. Comparing with == instead of hash_equals() → timing risk.
6. Timestamp isn’t validated at all → replay vulnerability (best practice).

Related

• Webhook signature headers + delivery rules
• Token scope + required ingestion headers
• Send test event + confirm in Message Log
• Laravel verification cookbook

Conclusion

• Keep an overlap window when rotating — accept both secrets during cutover.
• Verify signatures against a comma-separated SP_WEBHOOK_SECRETS list using hash_equals().
• Never log raw secrets or signatures in plain text.
• Confirm rotation success in Message Log before removing the old secret.
• Use the Sample Project to validate before and after removal.

When you remove the old secret, run one more Sample Project event and confirm clean 2xx runs in Message Log.