Design a Payment System
Problem
Design the payment backend for a large online marketplace. When a buyer checks out, the system charges the buyer's card through an external payment processor, records the money movement, and later pays out the sellers their share minus a platform fee. A buyer must never be charged twice for the same order, and every cent must be accounted for, even when networks time out or services crash mid-request. The marketplace processes 10 million payments per day and must support refunds.
- Do we store card numbers ourselves, or does a processor tokenize them so we stay out of most PCI scope?
- Which processors do we integrate with, and do we need to fail over between them?
- Are payouts to sellers immediate, or batched daily or weekly?
- Which currencies are supported, and do we handle currency conversion?
- Are partial refunds and partial captures needed?
- What is the acceptable delay before a payment shows as succeeded: synchronous at checkout, or can it settle asynchronously?
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Charge a buyer for an order through an external payment processor.
- Guarantee that retrying a payment request never charges the buyer twice.
- Record every money movement so balances can always be explained and audited.
- Pay out sellers their balance minus platform fees on a schedule.
- Support full and partial refunds.
- Notify the order service and the buyer of payment status changes through webhooks or events.
- Reconcile internal records against processor settlement reports every day.
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- 10 million payments per day: about 115 per second on average, with holiday peaks near 2,000 per second.
- Correctness over availability: a payment may be delayed or rejected, but never duplicated or lost.
- Checkout payment latency under 2 seconds at p99, dominated by the processor round trip.
- Ledger storage: each payment produces 4 to 6 ledger entries of about 200 bytes, or roughly 12 GB per day and 4 TB per year, retained for 7 years for audit.
- Raw card data never touches our servers; card details are tokenized by the processor's client SDK.
- 99.99% availability for the payment API, with graceful degradation to "pending" when a processor is down.
A payment service accepts checkout requests keyed by an idempotency key, persists the intent in a strongly consistent database before calling the external processor, and drives each payment through an explicit state machine. Every confirmed money movement is written to an append-only double-entry ledger, from which seller balances and payouts are derived. Asynchronous workers handle processor webhooks, retries of uncertain outcomes, payouts, and daily reconciliation against the processor's settlement files.
Loading diagram...
Checkout. The client tokenizes the card with the processor's SDK and sends POST /payments with the order id, amount, token, and an Idempotency-Key header. The payment service inserts a payment row in state CREATED keyed by the idempotency key; if the key already exists, it returns the stored result instead of charging again. The executor calls the processor with the same key as the processor's own idempotency key, then moves the payment to SUCCEEDED or FAILED and returns the result.
Uncertain outcomes. If the processor call times out, the payment moves to UNKNOWN. A retry worker queries the processor by idempotency key until it learns the real outcome. The processor's webhook is a second path to the same answer; state transitions are conditional updates, so whichever path arrives first wins and the other is a no-op.
Recording money. On each terminal transition, the payment service writes an event to an outbox table in the same transaction as the state change. A relay publishes it to the stream, and the ledger service posts balanced entries: debit buyer funds receivable, credit seller payable and platform revenue.
Payouts and reconciliation. A daily payout job sums each seller's payable balance, creates a payout record, and sends a bank transfer. Overnight, the reconciliation job compares every ledger entry against the processor's settlement report and flags mismatches for review.
Owns the lifecycle of every payment and refund. It is the only writer of payment state, and it enforces idempotency and legal state transitions.
- Java or Go service, stateless, behind the API gateway.
- PostgreSQL for payment records, chosen for serializable transactions and unique constraints.
Diagram
Loading diagram...
API
POST /paymentswithIdempotency-Keyheader and{ orderId, amount, currency, paymentToken }returns{ paymentId, status }.GET /payments/:idreturns the current status.POST /payments/:id/refundswithIdempotency-Keyand{ amount }returns{ refundId, status }.
Data Model
payments(id, idempotency_key UNIQUE, order_id, buyer_id, amount_minor, currency, status, processor_ref, version, created_at, updated_at)refunds(id, payment_id, idempotency_key UNIQUE, amount_minor, status, processor_ref)outbox(id, aggregate_id, event_type, payload, published_at)
Scalability
At 2,000 payments per second peak, a single primary with read replicas handles the write volume. Beyond that, shard by buyer_id or payment_id hash; the idempotency key lookup stays on one shard because the key embeds the buyer id.
Tradeoffs
Amounts are stored as integers in minor units (cents) to avoid floating point rounding. The transactional outbox is chosen over publishing directly to Kafka after commit, because a crash between the commit and the publish would lose the event and the ledger would silently drift. Two-phase commit across the database and the stream was rejected as operationally fragile.
Calls the external processor and translates its responses into state transitions. It isolates processor-specific APIs behind one interface so a second processor can be added for failover.
- Internal library plus worker pool, with per-processor adapters.
- Circuit breakers and timeouts tuned per processor.
Scalability
Processor calls are I/O bound, so a modest pool of workers with async HTTP clients handles thousands of concurrent requests. Rate limits imposed by the processor are respected with a token bucket per merchant account.
Tradeoffs
The executor never retries a charge blindly on timeout. It forwards our idempotency key to the processor and resolves UNKNOWN states by querying status, because a blind retry against a processor without idempotency could double charge. Automatic failover to a second processor is only allowed for requests that definitely did not reach the first one, such as connection refused errors.
Append-only double-entry bookkeeping. Every transaction posts two or more entries whose debits and credits sum to zero, so any balance can be recomputed from history.
- PostgreSQL with insert-only tables and a check that each transaction balances, or a purpose-built ledger database such as TigerBeetle at very high volume.
Data Model
accounts(id, owner_type, owner_id, currency, kind)where kind is asset, liability, revenue, or expense.ledger_transactions(id, source_event_id UNIQUE, description, created_at)ledger_entries(id, transaction_id, account_id, amount_minor, direction)where direction is debit or credit.account_balances(account_id, balance_minor, last_entry_id)as a cached projection.
Scalability
Entries are append-only, so writes never contend on updates except for the cached balance row. Hot accounts such as platform revenue are split into sub-accounts that are summed at read time. Old partitions are moved to cheaper storage by month.
Tradeoffs
Double-entry is chosen over a single balance column per user because it makes every balance auditable and self-checking: the sum of all entries is always zero. Entries are never updated or deleted; a mistake is corrected with a reversing transaction. The source_event_id unique constraint makes ledger posting idempotent when the stream redelivers an event.
Computes each seller's available balance and moves money to their bank account on a schedule.
- Scheduled batch job plus a worker that calls the bank or processor payout API.
Data Model
payouts(id, seller_id, amount_minor, currency, status, period_end, bank_ref, idempotency_key UNIQUE)
Scalability
Payouts are batched daily, so even millions of sellers produce a bounded job that can be partitioned by seller id range across workers.
Tradeoffs
Funds are held for a settlement window (for example 2 days) before payout to absorb refunds and chargebacks. Paying out instantly would improve seller experience but expose the platform to losses when a buyer disputes a charge after the seller has already been paid.
Daily comparison of our ledger against the processor's settlement report and the bank statement. It is the safety net that catches bugs, missed webhooks, and processor errors.
- Spark or a SQL batch job reading settlement files from object storage.
Scalability
Matching 10 million rows per day is a join on processor reference, which a batch engine finishes in minutes.
Tradeoffs
Reconciliation runs asynchronously instead of verifying each payment inline, because processors publish settlement data only once per day. Mismatches are auto-corrected only for known classes (such as a late webhook); everything else goes to a manual review queue, since an automated fix to financial records is riskier than a short delay.