Engineering Knowledge Base

S3

Properties1
tagsaws, s3, storage

3 min read

S3

Object storage for any amount of data, accessed over HTTP(S). Objects (up to 5 TB) live in buckets (globally unique names, region-scoped). Flat namespace — “folders” are just key prefixes.

Use Cases

  • Big data analytics, data lakes
  • Static website / content distribution (with CloudFront)
  • Backup, archive, disaster recovery
  • Application assets, mobile & game state

Durability & Availability

  • 11 9s of durability (99.999999999%) across all storage classes — designed for 1 object lost per 10,000 objects every 10M years.
  • Standard availability: 99.99% (~53 min downtime/year). IA classes drop to 99.9%, One Zone-IA to 99.5%.
  • Strong read-after-write consistency for all operations (PUT/GET/LIST), automatic since Dec 2020.

Storage Classes

Pick by access frequency and retrieval latency. Reference

ClassAvailabilityMin durationUse case
Standard99.99%—Frequently accessed, low latency
Standard-IA99.9%30 daysInfrequent, rapid when needed (backups, DR)
One Zone-IA99.5%30 daysRe-creatable / secondary backups (single AZ)
Intelligent-Tiering99.9%—Unknown/changing access — auto-tiers, no retrieval fees
Glacier Instant Retrieval99.9%90 daysArchive, ms retrieval, ~quarterly access
Glacier Flexible Retrieval99.99%90 daysArchive, minutes-to-hours retrieval
Glacier Deep Archive99.99%180 daysLowest cost, 12–48h retrieval, compliance archive

Glacier retrieval tiers

  • Instant Retrieval — milliseconds.
  • Flexible Retrieval — Expedited (1–5 min), Standard (3–5 h), Bulk (5–12 h, free).
  • Deep Archive — Standard (12 h), Bulk (48 h).

Intelligent-Tiering moves objects between access tiers automatically based on usage (Frequent → Infrequent after 30 days → Archive Instant after 90 days; optional Archive/Deep Archive tiers from 90–180+ days). Small monitoring fee; no retrieval charges.

Cost control

Use Lifecycle rules to transition objects to cheaper classes and expire them on a schedule — don’t move data by hand.

Lifecycle Rules

  • Transition actions — move objects to another class after N days.
  • Expiration actions — delete objects (or old versions, or incomplete multipart uploads) after N days.
  • Scope by prefix or object tags.

Versioning

  • Keeps every version of an object; protects against accidental delete/overwrite.
  • Enabled at the bucket level (can be suspended, never fully disabled once on).
  • A DELETE adds a delete marker; the prior version is recoverable.
  • Pairs with MFA Delete for extra protection on permanent deletes.

Replication

  • Asynchronous copy to another bucket. Versioning required on both source and destination.
  • CRR (Cross-Region) — compliance, latency, cross-account.
  • SRR (Same-Region) — log aggregation, prod↔test sync.
  • Replicates only new objects after enablement; use S3 Batch Replication for existing objects.

Encryption

  • SSE-S3 — S3-managed keys (AES-256), default for all new objects.
  • SSE-KMS — KMS keys, adds audit trail + access control (watch KMS request quotas).
  • DSSE-KMS — dual-layer KMS for stringent compliance.
  • SSE-C — customer-provided keys.
  • Enforce in-transit encryption with a bucket policy requiring aws:SecureTransport.

Security

  • Block Public Access is ON by default at the account and bucket level — the single most important guardrail.
  • Access via bucket policies (resource-based), IAM policies (identity-based), and ACLs (legacy, avoid).
  • Access Points — named endpoints with their own policies for shared datasets.
  • Presigned URLs — time-limited access to a specific object without IAM credentials.
  • Object Lock — WORM (write-once-read-many) for compliance/legal hold.

Other Features

  • Event notifications → SQS, SNS, Lambda, or EventBridge on object create/delete/etc.
  • Transfer Acceleration — upload via CloudFront edge locations.
  • Multipart upload — required for large objects; recommended above ~100 MB (single PUT max 5 GB).
  • S3 Select — SQL over a single object (CSV/JSON/Parquet) to retrieve only needed rows/columns.
  • Requester Pays — shift data-transfer cost to the requester.

Snippets

# Sync a local dir to a bucket (delete removed files)
aws s3 sync ./dist s3://my-bucket/ --delete
 
# Presign a download URL valid for 1 hour
aws s3 presign s3://my-bucket/report.pdf --expires-in 3600
 
# Enable default encryption (SSE-KMS)
aws s3api put-bucket-encryption --bucket my-bucket \
  --server-side-encryption-configuration '{
    "Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}'

Graph View

Backlinks