Large Payload Strategies

Size tiers — different tools

Scale	Pattern
KB–few MB	Normal REST/JSON body
MB–100 MB	Streaming body, gzip, direct upload
100 MB–5 GB	Multipart, presigned object storage URLs
GB–TB	Async export jobs, parallel multipart, dedicated transfer
10+ TB	Physical appliance, cross-account replication

Never stream 500 GB through your API server's memory — bypass with direct-to-storage upload. How S3 consistency, lifecycle tiers, and Parquet layout work → Databases → Object Storage & Data Lakes. How bronze-layer ingest jobs run → Data Engineering.

Presigned URLs

API authenticates user → returns time-limited PUT URL to S3/GCS → client uploads directly to object storage → callback/webhook or poll for completion.

Wins: no API bandwidth tax, horizontal scale, resumable via multipart.

Multipart upload (S3 pattern)

1. CreateMultipartUpload → uploadId
2. Upload parts in parallel (5–100 MB each) with part numbers
3. CompleteMultipartUpload with ETags

Failed part → retry that part only. ListParts for resume.

Same idea: GCS compose, Azure block blobs.

Resumable uploads (tus)

Open protocol: POST create upload → PATCH chunks with Upload-Offset. Server tracks offset; client resumes after disconnect.

Used by Vimeo, Cloudflare; better UX than rolling your own.

HTTP streaming download

Transfer-Encoding: chunked for server → client streams without known Content-Length.

Range requests (Range: bytes=0-1048575) for resume and parallel download segments.

Async export pattern

POST /exports → 202 + job ID → worker writes to object storage → GET /exports/{id} returns status: ready + download URL.

Used by Stripe data exports, analytics platforms, GDPR dumps.

Compression

Content-Encoding: gzip / zstd for JSON/text. Don't compress already-compressed (JPEG, parquet). Negotiate via Accept-Encoding.

Checksums and integrity

Content-MD5, x-amz-checksum-sha256 on upload — detect corruption. Critical for compliance and multi-part assembly.

Delta / rsync patterns

Only transfer changed blocks:

• rsync algorithm for file sync
• CDC in replication (change data capture) — different layer; see Data Engineering
• Binary diff patches for mobile app updates

Rate limiting large transfers

Per-user bandwidth quotas, concurrent upload limits, storage lifecycle policies.

Interview scenario

"Move 2 TB from on-prem to S3 weekly" — answer: initial Snowball, then S3 Transfer Acceleration or Direct Connect + incremental sync; API only orchestrates metadata.