Paxos Engineers 371ms Cutover for 21TB Postgres Ledger Migration

Paxos, the stablecoin infrastructure firm behind PYUSD and USDP, has printed technical particulars on how its engineering staff partitioned a 21-terabyte ledger database desk with out taking techniques offline—a feat that required simply 371 milliseconds for the ultimate cutover whereas crypto markets continued buying and selling across the clock.

The corporate’s ledger desk had grown to roughly two-thirds of Aurora’s per-table dimension restrict, giving engineers a few 12 months earlier than writes would begin failing. For a agency processing stablecoin transactions that may’t tolerate knowledge loss or delays, conventional migration approaches involving prolonged upkeep home windows weren’t viable.

The Technical Strategy

Relatively than copying billions of rows into a brand new construction—the usual partitioning playbook—Paxos constructed the partitioned structure across the present desk. The unique desk grew to become a “historical past” partition whereas new time-range partitions caught incoming knowledge. To exterior techniques, nothing modified; Postgres dealt with routing internally.

The catch? Postgres must confirm each row satisfies partition constraints earlier than attaching a desk, which suggests a full desk scan. On a 21TB desk, that is not fast.

Paxos break up this into two phases. First, they added the constraint as NOT VALID—a quick operation that skips verification. Then they ran VALIDATE CONSTRAINT individually, permitting reads and writes to proceed throughout the scan.

9 Hours of Ache

The validation scan took simply over 9 hours. Throughout that point, the lock prevented autovacuum from cleansing up lifeless tuples, inflicting tail latency to climb steadily. The primary try failed when write spikes exceeded timeout thresholds.

On the second try, Paxos coordinated with market makers to pause buying and selling exercise quickly and relaxed timeout thresholds. P50 latency stayed comparatively flat, however P95/P99 degraded considerably as lifeless tuples gathered.

“In hindsight, this was probably the most operationally demanding a part of the migration—not the cutover, however the scan that made the cutover attainable,” the engineering staff wrote.

Why This Issues for Crypto Infrastructure

The migration highlights a rising problem for crypto infrastructure suppliers. Not like conventional finance with scheduled upkeep home windows, crypto markets run constantly. Database tables backing ledger techniques cannot merely go offline for rebuilding.

Paxos additionally tackled a refined partitioning gotcha: uniqueness constraints throughout partitions. In Postgres, distinctive constraints on partitioned tables solely work globally in the event that they embody the partition key. In any other case, two inserts with an identical idempotency keys might land in numerous partitions and each succeed—a catastrophe for a monetary ledger that would apply the identical transaction twice.

The answer concerned idempotency checks outdoors the balance-update lock, including lower than 5 milliseconds of latency in inside testing.

Testing at Manufacturing Scale

Staging environments could not replicate the issue. Paxos used Aurora manufacturing cloning to create a full-sized take a look at database, then constructed a “reverse historical past” SQL generator that replayed actual transaction patterns backward to keep away from triggering overdraft failures.

The strategy displays broader business motion towards zero-downtime database migrations. Related methods have been documented for tables starting from 1.5TB to multi-terabyte scale, typically utilizing views to summary transitions from purposes.

For Paxos, the partitioned construction now permits one-liner archiving by means of DETACH PARTITION and removes the looming dimension ceiling as a constraint. The corporate says it is starting a sequence of posts on engineering challenges behind its infrastructure—a sign that stablecoin operators are more and more competing on technical credibility alongside regulatory compliance.

Picture supply: Shutterstock