Building Resilient Event-Driven Architectures with Postgres and Serverless Queues

Resilient Event-Driven Architecture Starts with the Database

Event-driven systems make it easier to decouple services, absorb bursts, and run work asynchronously. But the moment a database write and a message publish happen in separate systems, a familiar failure appears: the database commits while publishing fails, or publishing succeeds while the database transaction rolls back.

A robust design treats PostgreSQL as the system of record and a serverless queue such as Upstash QStash as a delivery mechanism. The transactional outbox pattern connects them: write business data and an event record in the same PostgreSQL transaction, then relay committed events to the queue. Consumers process events asynchronously and safely tolerate redelivery.

This is a practical foundation for serverless architecture—but it is important to be precise about the guarantee. The design delivers events at least once, not exactly once. Reliability comes from durable state, retries, and idempotency rather than assuming a message can never be duplicated.

Architecture: PostgreSQL Outbox, QStash Delivery

A typical flow looks like this:

1. An API request updates application tables and inserts an outbox event in one transaction.
2. A relay claims pending outbox rows and publishes them to QStash.
3. QStash delivers an HTTP request to a consumer, retrying according to its configured policy.
4. The consumer verifies the request, applies its business effect, and records the event as processed.
5. Failed or repeatedly unsuccessful work is inspected, replayed, or sent to a dead-letter workflow.

PostgreSQL is authoritative for the business change and the fact that an event needs delivery. QStash handles asynchronous HTTP delivery, retries, and traffic smoothing. This is not the same as an append-only event-streaming platform: QStash is a managed message-delivery service, not a substitute for a retained, partitioned event log when consumers need long-term offset management or broad replay across a stream.

Create an Outbox Table

Keep events immutable once written. Include a stable ID, event type, schema version, creation time, and enough payload for the consumer to perform its work. Avoid placing secrets or unnecessary personal data in messages.

CREATE TABLE outbox_events (
  id            uuid PRIMARY KEY,
  event_type    text NOT NULL,
  schema_version integer NOT NULL DEFAULT 1,
  payload       jsonb NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now(),
  published_at  timestamptz,
  attempts      integer NOT NULL DEFAULT 0,
  lease_token   uuid,
  lease_until   timestamptz,
  last_error    text
);

CREATE INDEX outbox_pending_idx
  ON outbox_events (created_at)
  WHERE published_at IS NULL;

Write the domain change and its event together. If either statement fails, PostgreSQL rolls back both.

BEGIN;

UPDATE orders
SET status = 'confirmed'
WHERE id = $1;

INSERT INTO outbox_events (id, event_type, payload)
VALUES (
  $2,
  'order.confirmed',
  jsonb_build_object('orderId', $1, 'eventId', $2)
);

COMMIT;

The event ID should remain stable across every delivery attempt. Consumers use it as an idempotency key.

Build a Relay That Survives Crashes

A relay can run as a scheduled serverless function or a continuously running worker. It should claim a small batch, release the database transaction, publish each event, and then mark successful publications. Never hold row locks open while making network requests: slow delivery would tie up database connections and block other workers.

FOR UPDATE SKIP LOCKED allows multiple relay instances to claim different rows. A lease makes claimed work recoverable if a worker dies. The following query illustrates the claim step; production code should assign a fresh lease token and lease expiry in the same transaction.

BEGIN;

SELECT id, event_type, payload
FROM outbox_events
WHERE published_at IS NULL
  AND (lease_until IS NULL OR lease_until < now())
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 100;

-- Update the selected rows with a lease token and expiry here.
COMMIT;

After committing the claim, the relay publishes to QStash. On success, it sets published_at only if the lease token still matches. On failure, it records the error and releases or expires the lease with bounded backoff. Token checks prevent a slow, expired worker from overwriting the state of a newer claimant.

There is an unavoidable crash window: QStash may accept a message just before the relay crashes, leaving the outbox row apparently unpublished. The relay will publish it again. That is expected. The consumer must be idempotent.

Make Consumers Idempotent and Secure

For a database-backed consumer, record processed event IDs in the same transaction as the business effect. A uniqueness constraint turns duplicate delivery into a harmless no-op.

CREATE TABLE processed_events (
  event_id uuid PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);

The consumer transaction should insert the event ID with conflict handling, check whether it was newly inserted, and apply the effect only in that case. For external side effects—such as charging a payment provider—use that provider’s idempotency-key support where available. A database dedupe table cannot make an unrelated external API call atomic.

Verify QStash request signatures before trusting the body. Configure the consumer to return a success response only after durable processing, and keep request handling within the platform’s timeout limits. Long-running work should be split into smaller events or handed to another durable job. Use bounded retries and alert on dead-letter messages; retries are a recovery mechanism, not a substitute for diagnosis.

Production Trade-Offs and Operations

Ordering, Throughput, and Backpressure

Multiple relay workers and queue deliveries improve throughput but do not guarantee global ordering. If order matters for a particular entity, include an aggregate ID and sequence number, then enforce ordering or reject stale events at the consumer. Partitioning work by entity can reduce contention, but increases operational complexity.

Tune batch size and concurrency against PostgreSQL connection limits, consumer capacity, and downstream rate limits. QStash flow control can help regulate delivery, while database-side leases prevent multiple relays from competing for the same row. Monitor outbox age, pending row count, delivery attempts, consumer latency, and dead-letter volume. Alert on oldest pending event age, not just queue depth: a small number of stuck events can be more urgent than a large, healthy backlog.

Retention, Replay, and Schema Evolution

Published outbox rows are useful for audit and recovery, but an ever-growing table harms indexes and maintenance. Define a retention policy, archive if required, and delete in bounded batches. Keep payloads versioned and make consumers tolerant of additive fields. For intentional replay, retain the original event ID when repeating the same logical effect; create a new event ID when initiating a genuinely new operation.

Use PostgreSQL NOTIFY only as a wake-up hint if lower relay latency is useful. Notifications are not a durable queue: the outbox remains the source of truth, and polling or another recovery path must catch missed wake-ups.

The Core Design Principle

PostgreSQL provides atomic business state plus durable intent. QStash provides managed asynchronous HTTP delivery. The outbox closes the dual-write gap; leases support concurrent relays; idempotent consumers make at-least-once delivery safe. Together, these pieces produce a resilient event-driven system without pretending that distributed failures—or duplicate messages—can be eliminated.

Sources & Technical References

• PostgreSQL: Transactions
• PostgreSQL: SELECT and `SKIP LOCKED`
• PostgreSQL: `NOTIFY`
• Upstash QStash documentation
• Upstash QStash: retries
• Upstash QStash: request signing
• Upstash QStash: flow control
• AWS Prescriptive Guidance: Transactional outbox pattern
• CloudEvents specification
• Martin Fowler: Idempotent Receiver