Offline & Physical Transfer

When the network loses to a truck

Rule of thumb (often attributed to AWS):

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

100 TB over 1 Gbps ≈ 9+ days theoretical — real world slower. Snowball ships appliance, loads at datacenter line rate, AWS ingests internally.

Cloud physical transfer

Service	Scale	Pattern
AWS Snowcone / Snowball / Snowmobile	TB–PB	Encrypted appliance, chain of custody
Azure Data Box	TB	Similar appliance model
GCP Transfer Appliance	TB	Rackable storage device

Use when: limited bandwidth, one-time migration, air-gapped environments, compliance constraints on public internet transfer.

Online large transfer (when physical is overkill)

• AWS DataSync / Transfer Family — managed agents, scheduled sync
• gsutil -m / azcopy — parallel object copy
• Cross-region replication — ongoing sync after initial seed

Initial seed often still physical; replication handles delta.

Bandwidth math (staff quick estimate)

time_seconds ≈ bytes × 8 / bits_per_second 100 TB = 8×10^14 bits 10 Gbps → ~22 hours ideal; ~2–3× real world

Factor: TCP overhead, encryption, small files, API list operations.

Security for bulk data

• Encryption at rest on appliance (KMS keys)
• Tamper-evident seals
• Logging chain of custody
• VPC endpoints / PrivateLink so data never traverses public internet after cloud ingest

Application API role

Your API does not move the bytes — it:

1. Issues credentials / job tickets
2. Tracks manifest and checksums
3. Triggers pipeline when object lands in s3://landing/
4. Notifies via webhook when import completes

Pairs with Data Engineering ingest patterns (bronze layer landing zone).

Replication vs migration

• Migration — one-time cutover, validate counts/checksums
• Replication — ongoing; conflict resolution, lag monitoring

Edge cases

• Many small files — tar/parquet bundle before transfer; list operations dominate
• Geo-distributed users — CDN for download; regional upload endpoints
• Compliance — data residency; transfer within region only

Senior answer structure: quantify size/time, compare network vs physical, describe orchestration API, mention integrity verification and incremental sync.