Webhooks are a distributed-systems problem

Every webhook integration is a distributed-systems problem wearing a trench coat. Deliveries are at-least-once, arrive out of order, and retry on timeouts you already processed. Design for that from line one and “webhooks are unreliable” stops being an outage category.

Idempotency: first writer wins

Dedup on the provider’s event id before you do any work:

const fresh = await redis.set(`evt:${tenant}:${id}`, '1', { NX: true, EX: 86_400 });
if (!fresh) return ack(); // duplicate delivery — acknowledge, do nothing

Ordering and replay

Sequence per entity, not globally — a stale update to one record shouldn’t be reordered against another. And keep the raw events: a durable log lets you replay a window deterministically to heal state after a bug, instead of reconstructing it by hand. Idempotency makes replay safe; ordering makes it correct.