Outbox

import "github.com/gp-system/gpsystem/events/outbox"

events/outbox is the delivery guarantee behind dispatch: instead of handing the event straight to Redis, it writes it into a Postgres table in the caller's transaction, and a relay forwards committed rows to the queue. In a generated project this is the default dispatcher: not an optional extra, but the starting point.

The dual-write problem

Take the shop's PlaceOrder: the order insert and the orderPlaced event must happen together or not at all. But two writes into two systems (Postgres + Redis) cannot be atomic:

• Enqueue the event before the commit, and if the transaction rolls back, the worker sends a confirmation for an order that never happened.
• Enqueue it after the commit, and if the process dies in between (a deploy, an OOM kill, a node failure), the order exists but the event is lost: the customer never gets an e-mail, and nothing tells you something went wrong.

An after-commit hook alone doesn't close this gap: it can skip a rollback, but a crash between the commit and the hook running still loses the event. In the kit the pattern comes stock: the outbox writes the event into the same database, in the same transaction as the business change, one commit that persists everything or nothing.

How it works

From here asynq takes over: fan-out to the listeners, retries, archiving; see events. The chain is at-least-once all the way, which is why listener idempotency remains a precondition.

Store and dispatcher

store := outbox.NewStore(pg.NewDB(pool)) // accepts a pg.DBTX
dispatcher := outbox.NewDispatcher(store) // an events.Dispatcher

The Store is a single method (Insert(ctx, env)), and NewStore builds on the kit's pgx executor: because pg.DB joins the transaction carried by the ctx, an insert made inside WithinTransaction automatically runs in the caller's transaction: the service needs to know nothing about the outbox. Called outside a transaction it is a plain insert; the event is still delivered, the atomicity guarantee just doesn't apply.

Projects using bun wire the events/outbox/bunx adapter, which joins the bun transaction instead:

import outboxbunx "github.com/gp-system/gpsystem/events/outbox/bunx"

dispatcher := outbox.NewDispatcher(outboxbunx.NewStore(bunDB))

In the generated main.gos this wiring is ready-made (the first add event inserts it); in the shop's HTTP entry point it looks like this:

deps := shop.Dependencies{
    DB:         pg.NewDB(pool),
    Transactor: pg.NewTransactor(pool),
    Dispatcher: outbox.NewDispatcher(outbox.NewStore(pg.NewDB(pool))),
}

The relay

relay := outbox.MustNewRelay(ctx, cfg.Outbox, pool, cfg.Worker.Redis)

err := worker.Run(ctx, cfg.Worker, register,
    worker.WithOutboxRelay(relay), // the worker runs it and shuts it down
    // ...
)

NewRelay / MustNewRelay opens its own queue.Client (verified with a PING), and Run(ctx) polls until the context is cancelled. You typically call neither by hand: the generated worker passes it via worker.WithOutboxRelay, and the worker lifecycle starts and stops it.

What's worth knowing about the poll loop:

• A full batch triggers an immediate re-poll. When a round processed OUTBOX_BATCH_SIZE rows, it doesn't sleep but polls again right away: bursts drain quickly, and POLL_INTERVAL is only the idle latency.
• Errors back off exponentially. A failing round (say, a Redis outage) retries with a doubling wait, capped at 30 seconds, and the first successful round returns to the normal cadence.
• Partial success is not lost. If an enqueue fails mid-batch, the rows already submitted get marked; the error doesn't replay the whole batch.

Scalability: N replicas, no double delivery

The relay runs in every worker replica, with no coordination. Two mechanisms make that safe:

1. FOR UPDATE SKIP LOCKED means two replicas never claim the same row;
2. the deterministic TaskID (= event_id) means a task submitted twice inside the crash window deduplicates in Redis: OUTBOX_TASK_RETENTION is the time window the dedupe lives for.

That's why you can scale the worker horizontally without a second thought; see worker → scaling.

The table and the migration

outbox_events arrives as a goose migration: new project writes it to migrations/20200101000100_outbox.sql (add worker backfills it into older projects); the schema's source of truth in the package is outbox.MigrationSQL. The essence:

CREATE TABLE outbox_events (
    id           bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    event_id     uuid        NOT NULL UNIQUE,
    event_name   text        NOT NULL,
    payload      jsonb       NOT NULL,
    metadata     jsonb       NOT NULL DEFAULT '{}'::jsonb,
    created_at   timestamptz NOT NULL DEFAULT now(),
    published_at timestamptz
);
CREATE INDEX outbox_events_unpublished_idx ON outbox_events (id) WHERE published_at IS NULL;

metadata carries the envelope's trace context, so the trace stays continuous through the outbox; the partial index keeps the poll fast even when the table grows large over time. Run it before starting the worker: go run ./cmd/migrate up (see migrations).

Configuration

The project config composes outbox.Config under the OUTBOX_ prefix (ready-made in the generated config.go):

Patterns used

This page is the full write-up of the Design patterns catalog's Transactional outbox entry: the FOR UPDATE SKIP LOCKED batch-claiming and the dual-write problem, with code and the shop PlaceOrder example, live here. Canonical external description: microservices.io: Transactional outbox.