Enterprises running Apache Iceberg on cloud object stores now face two connected problems: rising S3 bills and query performance degradation caused by many small files and long snapshot histories. This guide gives data engineers a pragmatic, production-ready process to implement cost-aware retention and compaction for Iceberg lakehouses on AWS S3. You’ll get measurement techniques, policy templates, safe execution steps, automation patterns and monitoring recommendations — concrete actions you can deploy in Q4 2026.

Why this matters now

Iceberg adoption surged in 2024–2026, and large tables with high ingest rates frequently accumulate millions of small Parquet/ORC files, heavy manifest lists, and long snapshot chains. These factors increase storage costs, amplify list/GET request volume (pay-per-request), and slow metadata-driven queries (planning time, manifest scanning). Addressing this requires coordinated work across S3 lifecycle rules, Iceberg snapshot management, and periodic compaction jobs that respect rollback and compliance needs.

Overview of the approach

  1. Inventory: measure storage, object counts and small-file ratios per table/prefix with S3 Inventory + Athena.
  2. Policy design: classify tables/partitions into hot/cold retention tiers and define snapshot retention windows.
  3. Compaction: implement partition-aware compaction jobs (Spark/EMR, Dataproc, or Flink rewrite) that rewrite small data files into target-size files.
  4. Snapshot and orphan cleanup: expire old snapshots and remove orphaned files safely after compaction.
  5. S3 lifecycle: transition cold data to cheaper tiers and retain metadata according to rollback needs.
  6. Automation & monitoring: orchestrate via Airflow/Step Functions and monitor storage, file sizes, and query latency.

1) Inventory: quantify the problem

Before any change, get a reliable baseline so you can prioritize tables that deliver the biggest cost wins.

  • Enable S3 Inventory for the bucket(s) that hold your Iceberg tables. Configure daily CSV/Parquet output in a separate bucket/prefix for Athena queries.
  • Create an Athena table over the S3 Inventory output. Use Athena to aggregate object sizes and counts by prefix (your Iceberg table root).

Example Athena query (conceptual):

SELECT
  regexp_extract(key, '^(.*?/db/table_name/)', 1) as table_prefix,
  count(*) as file_count,
  sum(size) as total_bytes,
  avg(size) as avg_bytes
FROM s3_inventory
WHERE key LIKE 'company-data/warehouse/%'
GROUP BY 1
ORDER BY total_bytes DESC;

What to look for:

  • Tables where avg_bytes < 64 MB (indicates many small files).
  • High number of objects per GB (many small files add request costs and manifest overhead).
  • Metadata directory size and number of metadata files under table_root/metadata/ (long snapshot history).

2) Policy design: retention tiers and constraints

Create a policy matrix that balances cost, rollback needs, and regulatory requirements. Example tiers:

  • Hot (last 7–14 days): rapid rollback and time-travel required for debugging and BI freshness. Keep frequent snapshots and avoid deep snapshot expiry.
  • Warm (15–90 days): lower probability of rollback. Keep daily snapshots, expire older snapshots progressively.
  • Cold (>90 days): rarely accessed. Compact and move to cheaper S3 storage classes; keep minimal metadata needed by retention laws.

Guidelines:

  • Never expire snapshots before your maximum rollback window (e.g., 7 days) unless you have a higher-level backup.
  • Regulatory/legal retention overrides cost-savings—separate compliance-critical tables into a different lifecycle.
  • Track per-table SLAs: interactive BI tables should tolerate fewer background compactions during business hours.

3) Compaction strategy: partition-aware and cost-conscious

Compaction rewrites many small data files into fewer, larger files to reduce object count, manifest size and request overhead. The safest approach is partition-aware compaction because it bounds IO and avoids long-running full-table rewrites.

Design decisions

  • Target file size: 128 MB–512 MB for parquet is a common tradeoff (smaller files improve parallelism; bigger files reduce metadata overhead).
  • Compaction frequency: heavy ingest tables may compact hourly/daily; less active tables weekly/monthly.
  • Per-partition vs. global compaction: prefer per-partition compaction to limit compute costs and keep jobs parallelizable.

Partition-level compaction pattern (Spark example)

High-level steps for each partition to compact:

  1. Read the partition into a DataFrame (filter to a specific partition key and date range).
  2. Estimate partition byte size (from S3 Inventory or by reading file metadata).
  3. Compute number of output files: ceil(partition_bytes / target_file_bytes).
  4. Repartition DataFrame to that many files, then write back into Iceberg using an append with a new snapshot (or use "overwrite by expression" if your engine supports safe partition overwrite).

Conceptual PySpark pseudocode:

# PSEUDO: estimate partition bytes externally (s3 inventory or metadata)
target_bytes = 256 * 1024 * 1024
num_files = ceil(partition_bytes / target_bytes)
df = spark.read.format('iceberg').load('catalog.db.table').filter("date='2026-10-01'")
df = df.repartition(num_files)
df.write.format('iceberg').mode('append').save('catalog.db.table')

Notes:

  • Prefer creating a new snapshot and letting Iceberg manage atomics rather than in-place file deletes.
  • To avoid double counting, schedule compaction jobs to run during low-traffic windows or throttle concurrency.
  • For very large partitions, break compaction across smaller sub-partitions or buckets to control job memory.

4) Snapshot expiry and orphan file removal

After compaction you must prune the old snapshots and remove files no longer referenced in the current metadata. Two Iceberg actions are frequently used:

  • Expire old snapshots (keeps the table metadata lean and reduces manifest lists).
  • Remove orphan files (delete data files that are no longer referenced by any snapshot).

Recommended safe sequence per table/partition:

  1. Run compaction and create a new snapshot.
  2. Wait a short safety window (e.g., 1–2 hours) to ensure compaction succeeded and downstream processes have acknowledged the new snapshot.
  3. Expire snapshots older than your retention window (e.g., older_than = current_timestamp - INTERVAL '7' DAY).
  4. Run orphan file removal to delete unreferenced objects from S3.

Important: test expiry and removal in staging. Keep a recovery plan (e.g., cross-region replication of critical tables or snapshot exports) before you delete data files for the first time.

5) S3 lifecycle rules and metadata handling

Iceberg stores table-level metadata (metadata/ directory) that is far smaller than data files but must be handled carefully. Use S3 lifecycle rules that differentiate by prefix:

  • Data files (table root / partition prefixes): apply transitions to Infrequent Access and Glacier/Deep Archive for cold partitions after your warm window.
  • Metadata files (table_root/metadata/): retain for a shorter period than data if you are comfortable with limited time-travel; otherwise retain metadata longer. Do not transition metadata to Deep Archive unless you accept slower restore times for time-travel or rollback.

Example lifecycle strategy:

  • Data: transition to INTELLIGENT_TIERING after 30 days, then to DEEP_ARCHIVE after 365 days for very cold partitions.
  • Metadata: keep in standard storage for 90 days, then transition to INTELLIGENT_TIERING; do not deep-archive metadata needed for recent rollbacks.

6) Automation: orchestration and safe guards

Orchestrate compaction, snapshot expiry and lifecycle changes with an idempotent DAG. A typical Airflow/Step Functions DAG:

  1. Inventory task: query S3 Inventory/Athena and compute candidate partitions for compaction.
  2. Decider task: apply policy rules to choose partitions and compaction parameters (target size).
  3. Compaction task(s): per-partition Spark job with concurrency limits.
  4. Verification: validate row counts, checksums, table metrics.
  5. Expire snapshots & remove orphan files.
  6. Tag objects and update S3 lifecycle rules if moving partitions to cold tier.

Safety features:

  • Dry-run mode to show estimated bytes saved and metadata changes without deleting objects.
  • Canary partition compaction to verify correctness and measure query latency impact.
  • Retry with exponential backoff and alerting on verification mismatch.

7) Monitoring and KPIs

Track these metrics per table and partition:

  • Storage bytes and cost (daily)
  • Object count and average object size
  • Snapshot count and age distribution
  • Manifest count and planning time for representative queries
  • Compaction job success rate and compaction CPU/IO cost

Alert thresholds examples:

  • avg_file_size < 32 MB for a table for 7 consecutive days
  • snapshot_count > 50 (indicates unpruned history)
  • unexpected reduction in row counts after compaction verification

8) Trade-offs and operational notes

  • Compaction increases short-term compute cost to reduce longer-term storage and request costs. Evaluate ROI per table.
  • Small files can improve query parallelism. Don’t blindly increase file sizes for latency-sensitive interactive queries.
  • Snapshot expiry reduces time-travel capability. Coordinate with downstream consumers before aggressive pruning.
  • Testing in a staging copy of a table is essential. Use representative partitions to validate data fidelity and query performance.

9) Quick checklist to roll this out

  1. Enable S3 Inventory and build Athena aggregation views for table-level metrics.
  2. Classify tables into retention tiers and document rollback windows.
  3. Build a partition-aware compaction job template (Spark/PySpark or native Iceberg action), with dry-run and verification steps.
  4. Automate expire-snapshots and remove-orphan-files after compaction with a safety delay.
  5. Implement S3 lifecycle rules with prefix filters for data vs metadata directories.
  6. Expose metrics to Prometheus/CloudWatch and create budget alerts tied to storage cost by prefix.
  7. Run a canary on a non-critical table and measure cost and query latency impact.

Real-world example: rough numbers

In one enterprise case, a 10 TB Iceberg table composed of 1.2M small files (avg 8 MB) was compacted into files averaging 256 MB. Object count dropped ~95% and S3 request costs fell by ~70% month-over-month. The team paid for a few hours of transient cluster time for compaction; ROI was achieved within two months through lower per-request and listing costs.

Conclusion

Reducing S3 costs for Iceberg lakehouses is not a single tool change — it’s a process: measure accurately, make tiered retention decisions, run safe partition-level compactions, prune snapshots, and automate with monitoring and safeguards. With disciplined rollout and a clear safety-first approach you can cut storage and request costs while preserving the rollback and time-travel capabilities that make Iceberg valuable.

Next steps: enable S3 Inventory, run the Athena queries in this guide to find 3–5 high-impact tables, and prototype a canary compaction for a single date partition in staging this month.