← All reference docs

Store drivers: lifting the single-instance cap (#14)

The v1 registry is the filesystem store over gcsfuse, pinned to one writer (docs/registry-verification.md, terraform/README.md). This is the design for the backends that remove that cap: GCS for the immutable objects, Postgres for the contended mutable state. It is scoped so the implementation is a driver change behind a seam, not a kernel rewrite — the higher-level logic (hashing, JSON, the journal chain, metadata merge, the proof-queue lease) stays in Store; only the byte-level I/O moves behind an interface.

The seam

Everything Store does to the filesystem reduces to a small byte-level surface. Extract it as an interface; the current code becomes the fs implementation.

type backend interface {
    // Content-addressed, immutable. Writing the same hash twice is idempotent
    // (identical bytes), so writes need no coordination and never conflict.
    GetObject(hash string) ([]byte, bool, error)
    PutObject(hash string, b []byte) error
    ListObjects() ([]string, error)

    // Per-hash metadata (mutable, but keyed by hash — cross-hash writes never
    // conflict; same-hash writes are serialized by the mutation lock).
    GetMeta(hash string) ([]byte, bool, error)
    PutMeta(hash string, b []byte) error

    // The name index — the contended mutable state. ReadNames+WriteNames is a
    // read-modify-write that MUST be atomic against concurrent repoints.
    ReadNames() ([]byte, error)
    WriteNames(b []byte) error

    // The journal — append-only, the chain computed in-kernel. ReadJournal
    // returns the exact bytes VerifyLog anchors its byte offsets on.
    ReadJournal() ([]byte, error)
    AppendJournal(line []byte) error

    // The proof queue (lease semantics live here — the fs backend does it with
    // rename+mtime; Postgres with FOR UPDATE SKIP LOCKED).
    EnqueueProof(job []byte, hash string) error
    ClaimProof(now time.Time, ttl time.Duration) ([]byte, bool, error)
    CompleteProof(hash string) error
    ProofDepth() (int, error)

    // The mutation mutex (fs: an O_EXCL lock file; Postgres: pg_advisory_lock).
    Lock() (unlock func(), err error)
}

Store keeps hashDef, the metadata-merge in StoreObject, the chain math in AppendLog/VerifyLog, and the queue job shape — all backend-agnostic. The rest of the kernel already goes through Store methods, so no call site outside the store layer changes.

GCS object backend

Objects and meta are content-addressed blobs — the easy, dumb, cheap part.

Postgres backend — the mutable, contended state

One instance, three tables, real transactions.

create table names   (name text primary key, hash text not null, updated timestamptz);
create table journal (seq bigserial primary key, entry jsonb not null, chain text not null);
create table proofq  (hash text primary key, job jsonb not null, gate bool,
                      leased_at timestamptz);   -- null = available

The cross-store commit

GCS and Postgres aren't in one transaction, but content addressing removes the need. A put commits in two steps, ordered so a crash is always safe:

  1. PutObject/PutMeta to GCS (idempotent, content-addressed).
  2. One Postgres transaction: repoint the name + append the journal entry.

A failure between them leaves an orphan object (addressable, unreferenced) — harmless and GC-able — but never a dangling name or a torn journal. The name only moves inside the transaction, which only runs after the bytes are durably stored.

Lifting the cap

With the mutable state transactional, the serve service drops min=max=1: min_instance_count = 0, max_instance_count = N. The worker Job scales out the same way (the queue is now real single-dispatch). OATH_STORE_LOCK (the fs mutex) is unset — Postgres is the coordinator. This is exactly the enable_database = true path the Terraform already gates; wiring the serve container to the DB (the env the v1 config deliberately omits) is the last step.

Migration

oath migrate-store reads an fs store and populates GCS + Postgres:

  1. Every objects/* and meta/* → GCS (idempotent; re-runnable).
  2. names.json → names rows.
  3. log.jsonl → journal rows preserving bytes and order, so the migrated journal's chain still verifies end-to-end (the migration asserts VerifyLog passes against the reconstructed stream before committing).
  4. proofq/* → proofq rows.

Content addressing makes it safe to run repeatedly and to run READS against the old store while backfilling.

Conformance

SPEC §8.1's filesystem layout stays the normative reference. A driver is conformant iff, for the same inputs, it yields byte-identical hashes, verdicts, and — crucially — a journal whose VerifyLog passes and whose chain values match the fs store's. That's a new conformance dimension: run the existing store tests (journal chain, tamper detection, put/repoint, proof-queue lifecycle) against each backend via the interface, plus a differential test fs-vs-driver.

Testing

The seam makes this tractable without cloud dependencies:

Status (what's built)

Building and selecting it

go build -tags cloud -o oath .        # cloud driver compiled in (adds the pg driver dep)
OATH_BACKEND=cloud \
OATH_OBJECT_BUCKET=my-objects \
OATH_DB_DSN='postgres://…' \
OATH_DB_DRIVER=postgres \             # matches the registered database/sql driver
  oath serve --http :8080

The default build (no -tags cloud) is unchanged and zero-dependency; OATH_BACKEND=cloud errors there rather than silently falling back. The deployment image adds a driver via a blank import in a //go:build cloud file it supplies, e.g. import _ "github.com/lib/pq" (kept out of this repo so the tree stays dependency-free; the driver name in OATH_DB_DRIVER must match).

Now built (beyond the seam)

Cutover runbook

The two gates are deliberately separate so the DB can be provisioned and migrated into while serve still runs on the filesystem store — enable_database provisions, activate_cloud_backend flips. Never flip before migrating (you would serve an empty registry); cloud_active = activate && enable, so a flip without the DB is a safe no-op.

  1. Provision — terraform apply -var enable_database=true. Stands up Cloud SQL + the DSN secret; serve/worker keep using the fs store.
  2. Migrate — OATH_STORE=<fs store> OATH_OBJECT_BUCKET=<bucket> OATH_DB_DSN=<dsn> oath migrate-store (a cloud-tagged binary; the DSN uses the DB's public IP or the proxy since it runs outside Cloud Run). It copies the store into GCS + Postgres and verifies the migrated journal before exiting.
  3. Flip — terraform apply -var enable_database=true -var activate_cloud_backend=true. Now OATH_BACKEND=cloud + the DSN + Cloud SQL socket are wired onto serve and the worker (the image already carries the backend). Then drop min_instance_count to 0 and raise max — Postgres is the coordinator, so the single-writer cap and OATH_STORE_LOCK are no longer needed.

Still to do

Rendered verbatim from docs/store-drivers.md in the repository. The markdown is the single source; this page is a copy checked for drift in CI, so what you read here is what an implementer reads.