Design an Object Storage Service like S3
Problem
Design a service where customers create buckets and store objects (files) by key, then read them back over HTTP. Objects range from a few bytes to 5 TB, and large objects must be uploadable in parallel parts that can be retried individually. The service must store 100 PB of data across billions of objects and handle 1 million requests per second. Stored data must be extremely durable (99.999999999%, or eleven nines), and a newly written object must be readable immediately by any client.
- Do we need strong read-after-write consistency for overwrites and listings, or only for new objects?
- Is object versioning required?
- What is the read to write ratio, and are most objects small or large?
- Do we need multiple storage classes, such as a cheaper tier for rarely accessed data?
- Is the service in one region with several availability zones, or replicated across regions?
- What access control model do we need: per bucket policies, per object ACLs, or both?
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Create and delete buckets.
- Put, get, and delete objects by bucket and key.
- Upload large objects in parts that can be sent in parallel and retried individually.
- Read-after-write consistency: a successful put is visible to every subsequent get.
- List objects in a bucket by key prefix with pagination.
- Support object versioning and access control per bucket.
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Store 100 PB of logical data across billions of objects.
- Handle 1 million requests per second, mostly reads, with throughput of many TB per second for large objects.
- Eleven nines of durability, which means surviving the loss of disks, servers, and an entire availability zone.
- 99.99% availability for reads and writes.
- Time to first byte under 100 ms at p99 for small objects.
- Storage overhead well below 3x: with 8 data and 4 parity fragments, raw capacity is 1.5x, or 150 PB.
Requests enter through a stateless API tier that authenticates them and splits work between two subsystems: a metadata service that maps bucket and key to object locations, and a data service that stores immutable chunks on storage nodes. Each chunk is erasure coded across storage nodes in three availability zones, and the metadata entry is committed only after the data is durable, which gives read-after-write consistency. Background services scrub data for corruption, rebuild lost fragments, and garbage collect deleted chunks.
Loading diagram...
Put a small object. The client sends PUT /bucket/key. The front end authenticates the request and checks the bucket policy, then asks the placement service for a set of 12 storage nodes spread across three zones. It streams the body, splits it into 8 data fragments, computes 4 parity fragments, and writes all 12 in parallel. Once at least 11 fragments are stored and fsynced, it commits a metadata record (bucket, key, version -> chunk ids, size, checksum) to the metadata store and returns 200 with an ETag.
Put a large object. The client calls CreateMultipartUpload, uploads parts of 8 MB to 5 GB with UploadPart (in parallel, each stored as its own chunks), and calls CompleteMultipartUpload with the list of part ETags. The metadata service then atomically creates the object record pointing at all parts.
Get an object. The front end reads the metadata record, locates the chunk fragments, and reads any 8 of the 12 fragments (preferring data fragments in the local zone, so decoding is usually unnecessary). It verifies checksums and streams the bytes. Range requests read only the chunks covering the range.
Delete. The metadata record is replaced with a delete marker (or removed), and the garbage collector later frees chunks that no longer have any reference.
Stateless HTTP tier that implements the public S3-style REST API.
- Go or Rust servers behind DNS-based and L4 load balancing.
- SigV4-style request signing verified against the auth service.
API
PUT /:bucketandDELETE /:bucket.PUT /:bucket/:key,GET /:bucket/:key(withRange),DELETE /:bucket/:key.GET /:bucket?prefix=&continuation-token=lists objects.POST /:bucket/:key?uploads,PUT /:bucket/:key?partNumber=&uploadId=,POST /:bucket/:key?uploadId=for multipart uploads.
Scalability
The front end holds no state, so it scales horizontally with request rate. It streams bodies rather than buffering them, so memory per request stays small even for multi-gigabyte objects.
Tradeoffs
Routing all data through the front end costs an extra network hop compared with clients writing directly to storage nodes. In exchange, storage nodes stay simple and private, encoding happens in one place, and clients need no knowledge of the internal topology.
Maps every bucket and key to its current version and chunk locations.
- A strongly consistent, horizontally partitioned store, such as a Raft-replicated key-value store or a distributed SQL database like FoundationDB or Spanner-style systems.
Data Model
buckets(bucket_name PK, owner_id, region, versioning, policy, created_at)objects(bucket_id, key, version_id, is_latest, size, etag, storage_class, chunk_ids[], created_at, is_delete_marker), ordered by(bucket_id, key, version_id).multipart_uploads(upload_id PK, bucket_id, key, parts[])
Scalability
Billions of keys with a few hundred bytes each is several TB of metadata, which is partitioned by range of (bucket_id, key) so prefix listings read contiguous ranges. Hot ranges are split automatically when their request rate grows.
Tradeoffs
Hash partitioning would spread load more evenly but make prefix listing a scatter-gather across every partition. Range partitioning keeps listings efficient, and automatic splitting handles hot prefixes. A strongly consistent store is required because read-after-write consistency for overwrites and listings depends on it; an eventually consistent store would be cheaper but would reintroduce stale reads.
Stores object bytes as immutable chunks spread across storage nodes and zones.
- Storage nodes with dense HDDs (for capacity) and a small SSD for write journaling.
- Reed-Solomon erasure coding with 8 data and 4 parity fragments.
Diagram
Loading diagram...
Data Model
- Fragment file on disk: header with
chunk_id,fragment_index,checksum, followed by the fragment bytes.
Scalability
With fragments spread four per zone, the loss of an entire zone still leaves 8 fragments, enough to rebuild every chunk. Small objects are packed together into shared chunks so a 1 KB object does not cost 12 separate disk writes and files.
Tradeoffs
Three full replicas are simpler and faster to read and repair, but cost 3x storage, which at 100 PB is 300 PB of raw disk. Erasure coding at 1.5x saves half the hardware in exchange for more CPU and more network traffic during repair. Very hot or tiny objects can still use replication, chosen per storage class.
Decides which storage nodes receive the fragments of each new chunk.
- An in-memory service with a consistent view of node capacity and health, fed by node heartbeats.
API
allocate(sizeBytes, storageClass) -> [{ node, fragmentIndex }]reportHealth(nodeId, freeBytes, status)
Scalability
Placement decisions are cached in batches by front ends, so the service handles allocation for whole groups of chunks rather than every request.
Tradeoffs
Weighting placement by free space and spreading across racks and zones prevents correlated failures from destroying too many fragments of one chunk. Pure random placement is simpler but slowly unbalances disks as hardware is added.
Background services that keep the durability promise over years.
- Distributed workers scheduled by a job coordinator, with rate limits per node.
Scalability
Scrubbers read every fragment on a regular cycle (for example every two weeks) and compare checksums to catch silent corruption. When a disk fails, repair reconstructs its fragments from survivors across many nodes in parallel, so a 20 TB disk is rebuilt in hours rather than days. Faster repair directly improves durability, because it shortens the window in which a second failure could hurt.
Tradeoffs
Garbage collection uses mark and sweep against the metadata store instead of deleting chunks synchronously during DELETE. Synchronous deletion would make deletes slower and risk freeing a chunk that a concurrent multipart completion still references. Delayed collection costs some temporary storage but is far safer.