High Availability

Roster high availability has two independent layers:

1. A Roster cluster with at least two identical instances behind a load balancer.
2. A PostgreSQL cluster with a writable primary and at least one read replica.

PostgreSQL commonly calls the writable node the primary or writer. Some providers call the same role the principal. This page uses primary.

PostgreSQL high availability and multi-instance Roster deployments require Enterprise. See Licensing.

Before you begin

Have the following ready before enabling HA mode:

• one immutable Roster image version for every instance
• a valid Enterprise license
• a PostgreSQL 17 or 18 service with a stable writer endpoint, at least one read-only endpoint, automatic promotion, backups, and tested recovery
• schema-owning migration credentials and separate restricted runtime credentials
• a load balancer with HTTPS and a stable public hostname
• shared S3 or SFTP storage for CSV connector files
• centralized logs and monitoring

If any of these prerequisites are unavailable, keep Roster in standalone mode until the platform is ready.

Deployment sequence

1. Provision the PostgreSQL database and roles.
2. Run one migration-only job against the writer endpoint and require a successful exit.
3. Start at least two Roster instances with the shared runtime configuration on this page.
4. Configure the load balancer to route only to instances that pass /health/ready.
5. Confirm login, authenticated REST and MCP requests, worker processing, and connector refreshes through the load balancer.
6. Run controlled Roster-instance and PostgreSQL-primary failover tests before accepting the deployment.

Architecture

Request and data flow One public edge, stateless Roster instances, shared PostgreSQL

Clients

Web REST CLI MCP

Public edge Load balancer Routes traffic only to ready instances

/health/ready

Roster cluster Two or more identical instances

Instance A Roster Ready

Instance B Roster Ready

↓ Writes + transactions Consistent operations

↓ Ordinary reads Drizzle replica routing

Primary

Writer endpoint Read / write ROSTER_DATABASE_URL

→ PostgreSQL
replication

One or more

Read replicas Read only ROSTER_DATABASE_REPLICA_URLS

Primary path Replica read path Healthy Roster instance

The load balancer distributes requests across healthy Roster instances. Every Roster instance connects to the same PostgreSQL writer endpoint and the same set of read-replica endpoints.

The writer endpoint must always target the current primary. When the primary fails, the PostgreSQL service promotes an eligible replica and moves the writer endpoint to the promoted node.

ROSTER_DATABASE_URL must always target the writer. Configure read-only endpoints separately with ROSTER_DATABASE_REPLICA_URLS.

Roster cluster

Run at least two identical Roster instances in separate failure domains when the hosting platform supports it. Place them behind a redundant load balancer.

Every instance uses the same:

• immutable Roster image version
• final public HTTPS ROSTER_AUTH_URL
• ROSTER_AUTH_SECRET
• ROSTER_PROVIDER_SECRET_ENCRYPTION_KEY
• PostgreSQL writer endpoint
• PostgreSQL read-replica endpoints
• runtime configuration and Enterprise license

Set a unique ROSTER_INSTANCE_ID for each instance, or allow Roster to infer one from the runtime environment. Sticky sessions are not required.

Load balancer health checks

Use GET /health/ready for load-balancer routing. Readiness returns 200 only when the runtime configuration is valid, the PostgreSQL writer is reachable, and the connector configuration is safe for HA. It returns 503 while a dependency is unavailable.

Use GET /health/live only for process liveness. Liveness does not check the database or other dependencies.

Readiness checks the PostgreSQL primary because a healthy Roster instance must be able to accept writes. Configure monitoring for replica health and replication lag through the PostgreSQL service.

During PostgreSQL failover, readiness may temporarily return 503. The load balancer should keep probing each Roster instance and restore traffic after the writer endpoint becomes healthy again.

PostgreSQL cluster

Provision a PostgreSQL cluster with:

• one writable primary
• at least one read replica in a different failure domain
• automatic failure detection and replica promotion
• a stable writer endpoint that follows the current primary
• stable read-only endpoints for the configured replicas
• backups and point-in-time recovery appropriate to the deployment

Roster validates that HA mode uses PostgreSQL, but it cannot inspect the database cluster topology or confirm that automatic failover is configured. Verify primary promotion and writer-endpoint failover with the database provider before production use.

Replication mode, recovery-point objective, and recovery-time objective are database-provider decisions. Select a PostgreSQL plan that meets the required durability and failover guarantees.

Read replica routing

Roster distributes eligible ordinary reads across the configured replicas. Writes, transactions, readiness checks, authentication data, and security decisions such as session, API-token, role, identity-provider, and OAuth validation stay on the primary. If the primary is unavailable, those checks fail closed until the database service promotes a replica and restores the writer endpoint.

Replica routing is eventually consistent. A read immediately after a write can briefly return older data until PostgreSQL replication catches up. Monitor replication lag and choose a PostgreSQL service whose lag bounds meet the application’s requirements.

When multiple URLs are configured, Roster selects among them for each eligible read. The endpoints must remain reachable from every Roster instance. Prefer managed read endpoints that remove unavailable replicas, or update the runtime configuration when a direct replica endpoint is taken out of service.

Runtime configuration

Configure every Roster instance with:

NODE_ENV=production
ROSTER_DEPLOYMENT_MODE=ha
ROSTER_DATABASE_URL=<postgresql-writer-endpoint>
ROSTER_DATABASE_REPLICA_URLS=<comma-separated-postgresql-read-replica-urls>
ROSTER_DATABASE_MIGRATION_MODE=skip
ROSTER_AUTH_URL=<your-roster-public-url>
ROSTER_AUTH_SECRET=<shared-auth-secret>
ROSTER_PROVIDER_SECRET_ENCRYPTION_KEY=<shared-provider-encryption-key>
ROSTER_LICENSE_KEY=<enterprise-license-jwt>
ROSTER_BOOTSTRAP_ADMIN_PASSWORD=<fresh-database-only-strong-password>

Store these values in the hosting platform’s secret manager and provide the same values to every Roster instance. An existing database whose bootstrap administrator already exists does not need ROSTER_BOOTSTRAP_ADMIN_PASSWORD. After first-start verification, remove that bootstrap value from the deployment configuration. A mounted ROSTER_LICENSE_FILE can be used instead of ROSTER_LICENSE_KEY.

HA mode fails fast when it detects SQLite, missing PostgreSQL replica URLs, migration-on-start, missing shared secrets, or an enabled local-file CSV connector.

Database migrations

Run one migration-only release job before updating the Roster cluster. The job uses a schema-owning credential and connects to the PostgreSQL writer endpoint:

ROSTER_DATABASE_URL=<schema-owner-writer-endpoint>
ROSTER_DATABASE_MIGRATION_MODE=only

The migration job does not use ROSTER_DATABASE_REPLICA_URLS.

Runtime instances use a separate restricted credential with ROSTER_DATABASE_MIGRATION_MODE=skip. Do not run migrations independently on each Roster instance or against a read-only replica.

See Production Database Migrations for the complete rollout sequence.

Shared runtime coordination

PostgreSQL stores rate-limit counters and worker coordination shared by every Roster instance. Workers can run on every instance and use leases, heartbeats, fencing tokens, and bounded retries.

Worker delivery is at least once. Connector refresh operations must remain idempotent so another instance can safely reclaim work after a failed lease.

Each configured PostgreSQL endpoint has its own pool. With one primary and R replica URLs, allow up to Roster instances × (1 + R) × ROSTER_DATABASE_POOL_MAX runtime connections, plus migration and operational connections. Retain an edge rate limit for volumetric traffic before requests reach Roster.

Shared storage and logs

Do not use local-file CSV connectors in HA mode. Use S3 or SFTP so every Roster instance reads the same connector content.

Centralize stdout logs and monitoring across all Roster instances. Back up any remaining /data content that holds connector files, worker journals, or other runtime state.

Failure behavior

Failure	Expected behavior
One Roster instance stops	The load balancer removes it and continues routing to healthy instances.
PostgreSQL primary stops	The database service promotes a replica and moves the writer endpoint. Roster readiness can remain unavailable until connections recover.
PostgreSQL replica stops	Writes can continue, but reads routed to an unavailable direct replica endpoint fail until the endpoint recovers or is removed.
Replication is delayed	Reads routed to a replica can temporarily return older data. Monitor lag and restore the required bound before relying on those reads.
Migration job fails	Do not deploy the new Roster image. Inspect the migration and database state first.

Production acceptance

Do not direct production traffic to the cluster until all of the following are confirmed:

• at least two Roster instances pass readiness through the load balancer
• login and authenticated requests continue when either Roster instance is removed
• PostgreSQL promotes a replica and the writer endpoint recovers without a ROSTER_DATABASE_URL change
• ordinary reads tolerate the expected replication lag
• one connector refresh completes without duplicate or lost work during worker failover
• backups, monitoring alerts, and the isolated restore procedure have owners and a tested schedule

Ongoing operations

Monitor the deployment as one system rather than as independent containers:

• alert when fewer than two Roster instances pass /health/ready
• monitor PostgreSQL writer availability, replica reachability, replication lag, connection-pool saturation, storage growth, and transaction conflicts
• monitor the last successful backup, available recovery window, and the age of the last isolated restore drill
• centralize stdout logs and watch repeated worker retries, exhausted jobs, and connector failures across every instance
• review ROSTER_DATABASE_POOL_MAX whenever the number of Roster instances or configured replica URLs changes

Before a Roster upgrade, take a fresh PostgreSQL backup, run the migration-only job, require its successful exit, and then roll out the same immutable image to all Roster instances. Keep the previous image and database recovery point until readiness, login, authenticated REST/MCP requests, worker processing, and connector health are accepted.

Run a controlled failover drill after initial setup and periodically afterward. Stop or fail the PostgreSQL primary through the provider’s supported process, confirm a replica is promoted and the writer endpoint moves, then verify Roster readiness and authenticated writes recover without changing ROSTER_DATABASE_URL. Also remove one Roster instance and one read replica at a time to verify load-balancer and reader-endpoint behavior. Perform destructive drills only in an isolated environment unless the production change process explicitly authorizes them.

Test Roster instance removal, PostgreSQL primary promotion, read-replica routing and lag, health-check routing, shared sessions, rate limits, and worker lease recovery before the deployment is considered production-ready.