Data teams deploying ML in production increasingly need feature pipelines that guarantee freshness—tight service-level agreements (SLAs) for how recent features must be relative to inference time. In 2026, teams commonly combine Kafka for event ingestion, Apache Flink for low-latency transforms, Iceberg as the durable offline store, and Feast (or similar) for online serving. This guide walks through a practical, operational approach to build reproducible, low-latency feature pipelines that meet freshness SLAs.

Why focus on freshness?

Model performance often degrades when features are stale. A freshness SLA is typically defined per feature or feature set (e.g., “feature last-updated within 30 seconds of inference”). Achieving that repeatedly requires engineering for low end-to-end latency, deterministic processing, and robust observability. This guide assumes you need sub-minute freshness for critical online features and hourly or daily freshness for less critical offline features.

High-level architecture

We’ll implement a pattern used in production by many teams in 2026:

  • Event ingestion: Kafka (Confluent or cloud-managed MSK/Cloud Pub/Sub Kafka) receives source events and CDC streams.
  • Stream processing: Apache Flink performs joins, aggregations, deduplication, and feature computations with exactly-once semantics.
  • Offline store: Apache Iceberg tables store feature history and support fast analytical queries and backfills.
  • Online store / serving: Feast (online store: Redis/KeyDB or DynamoDB) exposes low-latency feature lookups to models.
  • Orchestration: A scheduler (Airflow, Dagster) coordinates backfills, schema migrations, and ML model training pipelines.
  • Observability: Prometheus/Grafana for infrastructure metrics, custom metrics for freshness, and a data quality engine (Great Expectations, Soda) for invariants.

Step 1 — Define SLAs, keys, and TTLs

Begin by specifying for each feature:

  • Primary entity key (user_id, device_id, order_id).
  • Freshness SLA (e.g., 30s, 5m, 24h).
  • Retention/TTL in the online store (how long to keep feature values).
  • Acceptable staleness policy for model fallback behavior.

Document these in your feature registry (Feast or internal catalog). These become inputs for monitoring and alert thresholds.

Step 2 — Ingest events and CDC with Kafka

Use Kafka topics partitioned by entity key to preserve ordering per key. For relational sources, use Debezium to emit CDC streams; for events, instrument producers to include event_time and event_id.

Practical tips:

  • Use compacted topics for latest-by-key streams where appropriate.
  • Include an explicit event_time and monotonic sequence or UUID for dedupe.
  • Set retention for raw event topics long enough to cover possible replays/backfills (e.g., 7–30 days), and keep compacted copies of primary keys.

Step 3 — Compute features in Flink with exactly-once guarantees

Flink is well-suited for low-latency, stateful transformations that produce features. Key engineering goals here are deterministic state, fault-tolerance, and checkpointing tuned for latency.

Core Flink design patterns

  • Per-key state using RocksDB backend to handle high cardinality.
  • Event-time processing with watermarks to bound lateness.
  • Windowed and stateful aggregations for counts, rates, and time-decayed features.
  • Deduplication using event_id or CDC transaction id.
  • Emit both (a) online updates to Feast's ingestion API or online store and (b) append/compact writes to Iceberg for offline history.

Configuration guidance

  • Enable frequent, small checkpoints (e.g., 10–30s) if you need low recovery RPO; use incremental RocksDB checkpoints to reduce storage.
  • Use exactly-once sinks where possible. For writes to Iceberg, use Flink’s Iceberg connector with transactional semantics and commit coordination.
  • Right-size Flink parallelism based on per-key skew and state size. Use rescaling if traffic patterns vary.

Example Flink SQL (conceptual)

Below is a conceptual Flink SQL snippet to compute a 1-minute window rate and upsert into Iceberg:

<code>
CREATE TABLE kafka_events (
  user_id STRING,
  event_time TIMESTAMP(3),
  event_id STRING,
  attr DOUBLE,
  WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH ( ... 'connector'='kafka' ... );

CREATE TABLE iceberg_features (
  user_id STRING,
  feature_name STRING,
  value DOUBLE,
  feature_time TIMESTAMP(3),
  PRIMARY KEY (user_id, feature_name) NOT ENFORCED
) WITH ( ... 'connector'='iceberg' ... );

INSERT INTO iceberg_features
SELECT
  user_id,
  'event_rate_1m' AS feature_name,
  COUNT(*) AS value,
  WINDOW_END AS feature_time
FROM TABLE(
  TUMBLE(TABLE kafka_events, DESCRIPTOR(event_time), INTERVAL '1' MINUTE)
)
GROUP BY user_id, WINDOW_START, WINDOW_END;
</code>

Use MERGE INTO where possible for idempotent upserts and to maintain single-row-per-(user_id,feature_name) semantics.

Step 4 — Offline store: Iceberg design and partitioning

Iceberg is a durable analytical store with snapshot isolation and performant scans. Use it as your canonical historical repository for features and for batch materialization/backfills.

  • Design tables with a compound key: (entity_id, feature_name, feature_time).
  • Partition by date (e.g., feature_time_day) plus a hash-bucket on entity_id for high-cardinality keys to avoid small-file problems.
  • Enable table-level metadata management with a catalog (Nessie or Hive/Glue) and schedule regular compaction/major-merges to reduce small files created by streaming writes.

Step 5 — Serving: real-time writes to online store (Feast)

For sub-100ms lookup latency at inference time, write the most recent feature values into an online store.

  • From Flink, call Feast ingestion APIs or write directly to the online store (Redis, DynamoDB, or KeyDB). Batch writes should be avoided for the critical features path.
  • Use versioned keys or timestamps to resolve out-of-order arrivals: only accept writes with feature_time newer than existing value.
  • Set TTLs aligned with your retention policy and storage cost model.

Step 6 — Backfills and historical recompute

Backfills are a frequent source of complexity. Adopt a two-path model:

  1. Streaming updates keep the online store fresh for live traffic.
  2. Periodic batch recomputes (Spark, Flink batch, or Flink batch SQL) write full historical features into Iceberg and into the Feast offline store for model training.

When overwriting historical data, use transactional replace-at-point or MERGE semantics in Iceberg so live reads are not interrupted. Coordinate backfills with the model training pipeline via the orchestrator to prevent training on partial data.

Step 7 — Monitor freshness and enforce SLAs

Observability is essential. Implement the following metrics and alerts:

  • Feature freshness latency = inference_time - latest_feature_time (per feature/key). Emit a percentile distribution and max.
  • Event processing lag = Kafka consumer lag (time-based) per partition and per job.
  • Flink job health: checkpoint success rate, restore time, state size growth.
  • Online store hit rate and lookup latency (p50/p95/p99).

Example alert: trigger PagerDuty if median freshness for a critical feature exceeds 50% of SLA for 5 minutes, or if max freshness exceeds SLA. Also alert on abnormally high null rates or drift signals (see next section).

Step 8 — Data quality and drift detection

Feature correctness is as important as freshness. Implement:

  • Invariant tests (null ratio, value ranges, cardinality checks) continuously on both online writes and offline Iceberg tables.
  • Schema-change detection and automated dry-run validations before applying major changes.
  • Drift detectors that compare online feature distributions with training-time distributions and emit alerts when shifts exceed thresholds.

Operational knobs and scaling considerations

Operational trade-offs you’ll encounter:

  • Checkpoint interval vs. recovery time: more frequent checkpoints reduce potential data reprocessing but increase I/O.
  • State backend sizing: plan RocksDB memory and local SSDs on Flink TaskManagers according to expected state size per key.
  • Compaction cadence for Iceberg: frequent compaction reduces small files but consumes CPU; choose compaction windows aligned to write patterns.
  • Cost control: tier hot online store memory for critical features and store less-critical features only in Iceberg with slightly relaxed SLAs.

Testing and CI/CD for feature pipelines

Automate testing at multiple levels:

  • Unit tests for transform logic with synthetic input streams.
  • Integration tests that run Flink jobs in a lightweight cluster and validate writes to a temporary Iceberg catalog and a mock online store.
  • Contract tests: validate schema and semantics in a staging environment and run backfill rehearsals before production merges.

Checklist before production rollout

  • Defined SLAs and documented fallback behavior
  • Idempotent ingestion and deduplication strategies
  • Exactly-once sinks and checkpointing configured
  • Proper Iceberg partitioning and compaction policy
  • Online store TTLs and write consistency checks
  • End-to-end monitoring for freshness, lag, and data quality
  • Backfill strategy and training-data freeze window

Real-world example (concise)

A fintech company needed sub-30s freshness for fraud-score features. They partitioned Kafka by account_id, ran Flink streaming jobs with RocksDB and 15s checkpoints, wrote latest features into Redis via Feast, and kept Iceberg as the source of truth. They implemented a guardrail that rejected online writes older than the stored timestamp and added Prometheus alerts for p99 freshness. After these changes they reduced fraudulent transactions by 14% due to improved real-time scoring.

Conclusion

Meeting freshness SLAs requires a combination of streaming architecture, transactional writes to a durable offline store, a low-latency online serving layer, and robust monitoring. The Kafka → Flink → Iceberg → Feast pattern is battle-tested in 2026, but success depends on careful state management, deduplication, compaction strategy, and automated observability. Start small with a single critical feature, iterate on checkpointing and compaction settings, and expand once SLAs are consistently met.

Recommended next steps: prototype end-to-end on a narrow feature set, instrument freshness metrics from day one, and run a backfill rehearsal before scaling to hundreds of features.