Under 10ms serving with feature store design for ML engineers
- 2 hours ago
- 11 min read
A feature store is the dual-store architecture, a registry, and a materialisation engine working together to guarantee point-in-time correct features for training and near-instant serving in production. The single design principle that matters most: separate the read pattern for training from the read pattern for serving, but compute both from one shared transformation definition. Adopt this pattern once more than one team reuses the same features or once training-serving skew starts corrupting model performance.
TL;DR:
The online store’s latency target is under 10 milliseconds for most production inference needs, typically achieved with in-memory or hybrid key-value stores.
Offline storage benefits from columnar formats like Parquet or Delta Lake, which dramatically reduce training query times compared to row-based databases.
Maintaining strict parity in transformation logic and regular automated drift testing between offline and online paths is crucial to prevent training-serving skew.
Reusing features across teams makes a feature store worthwhile, especially when model freshness demands rapid, near real-time updates rather than only batch processing.
Operational practices such as incremental backfills, gap detection, and region-aware replication are vital for reliable, scalable feature store management at scale.
Table of Contents
What is feature store design and why does it need two stores?
Feature store design starts with four subsystems, not one database with a clever schema. A managed feature store architecture needs a feature registry, an offline store, an online store, and a materialisation engine, and each one solves a different failure mode that ad hoc feature pipelines eventually hit.
The registry is the catalogue: it tells engineers what a feature means, who owns it, and where it comes from. The offline store holds full historical feature values for training, joined against labels at the correct point in time. The online store holds only the latest values, indexed by entity, ready for a model to fetch during inference. The materialisation engine is the connective tissue that computes features once and pushes them into both destinations on a schedule.
These four pieces interact constantly. A training job queries the offline store for a historical snapshot; a production model queries the online store for the current value of that same feature, computed by the same logic. If those two paths diverge even slightly, you get training-serving skew, the single most common cause of a model that scores well in evaluation and then underperforms in production.
The non-functional requirements that should drive your architecture decisions early are freshness (how quickly a feature must reflect new events), latency (how fast the online store must respond under load), and consistency (how tightly training and serving values must match at any given timestamp). Get these requirements agreed with stakeholders before you pick a single technology, because they determine almost everything downstream: your materialisation cadence, your storage engine, and your monitoring thresholds.
How should you design the offline feature store?
The offline store’s job is bulk historical reads for training, and that workload rewards a completely different storage choice than the online store. Columnar formats on object storage such as Parquet or Delta Lake outperform row-based databases here because training jobs typically scan a handful of columns across millions of rows, and columnar layouts skip the ones they don’t need.
Point-in-time correctness lives or dies on how you join features to labels. An AS-OF join pulls the feature value that existed at or before the label’s event timestamp, never after it. Get this wrong and your model trains on information it wouldn’t have had in production, a form of data leakage that inflates offline accuracy and quietly wrecks live performance. Practically, this means partitioning your offline tables by event date and entity key, so an AS-OF lookup can prune irrelevant partitions instead of scanning the whole history.
Backfills are where offline stores earn their complexity. When you add a new feature or fix a computation bug, you need to recompute historical values across every affected partition, which can mean rewriting months of data. Plan backfill jobs to run incrementally, partition by partition, with checkpointing so a failure halfway through doesn’t force a full restart. Query performance tuning at this layer usually comes down to sensible partition pruning and avoiding wide joins across unrelated entity types. A well-partitioned offline store on Parquet can serve training pulls that would otherwise take hours in minutes.
What latency and architecture does the online store need?
The online store exists for one reason: serve the freshest feature value for a given entity in single-digit milliseconds. Sub-10ms p99 lookups are the practical target most production inference paths need to hit, which rules out anything built for analytical scans and points you toward a key-value store, often backed partly or entirely in memory.
Choosing that store is a trade-off between cost and speed. Fully in-memory engines give the lowest latency but the highest per-gigabyte cost, so many teams keep hot entities in memory and let colder ones fall back to a disk-backed tier with slightly higher latency.
Entity-key design matters more than people expect. If a model needs twenty features about one customer, you want one round trip that returns all twenty, not twenty separate lookups. Grouping co-located features under a single entity key and fetching them as one record cuts network overhead dramatically and is one of the cheapest performance wins available.
For teams operating across regions, replication strategy needs deciding early: do you replicate the full online store to each region, or route requests back to a primary? Cross-region replication adds cost and consistency lag; a single-region online store with regional caching is often simpler. TTL policies should match your freshness SLA, expiring stale entity records rather than serving outdated values indefinitely.
How do batch and streaming materialisation compare?
Materialisation is where features actually get computed and pushed into both stores, and the batch versus streaming choice should follow your freshness requirement, not fashion. Batch materialisation, running hourly or daily, suits features that change slowly, like a customer’s lifetime spend. Streaming materialisation, computing features as events arrive, suits anything that needs to reflect activity within seconds, like a fraud signal based on the last transaction.
Dual-write reliability is the part most teams underestimate. Writing a feature to both the offline and online store is two separate operations, and either one can fail independently. The standard pattern, and one worth adopting as a default rather than an afterthought, is writing to the offline store first, then the online store, with gap detection that flags any online write missing its offline counterpart. When gaps appear, replay from the offline history to repair the online store rather than trying to patch it live.
Idempotency matters just as much as ordering. Materialisation jobs will retry after failures, and a retry that double-counts a feature update is worse than the failure itself, so every write should be safe to repeat. On scheduling, per-feature materialisation with independent SLAs avoids a single slow feature blocking an entire batch, though grouping co-located features into one job reduces compute overhead. Materialisation compute, particularly for backfills and retries, tends to become the largest ongoing operational cost in a mature feature store, so budget for it explicitly rather than treating it as a rounding error.
How do you guarantee training-serving consistency?
Training-serving consistency comes down to one rule: define a feature’s transformation logic once, and run that exact definition in both the offline pipeline and the online serving path. Maintain two separate implementations, even conceptually identical ones written by different teams, and they will drift within a quarter.
Event-time constraints prevent lookahead bias, which is the subtler cousin of leakage. An AS-OF join enforces that a training example only ever sees feature values timestamped before its label event, but you also need to enforce that the online store never serves a value computed from future data relative to the request time, which sounds obvious until a delayed batch job backfills a value out of sequence.
Parity testing catches drift between the two paths before it reaches production. A practical version: for a sample of entities, pull the feature value from both stores at the same timestamp and diff them daily. Any consistent mismatch signals either a transformation bug or a materialisation lag worth investigating immediately. This single check, run automatically, catches more training-serving skew than any amount of code review, because skew tends to appear only once real production traffic and real timing hit the pipeline.
What metadata and governance controls does a feature registry need?
Every feature in the registry needs a “feature card”: owner, schema, computation logic, freshness SLA, and lineage back to source data. Skip any of these and reuse becomes guesswork. Without a documented owner, nobody knows who to ask when a feature starts behaving oddly. Without lineage, you can’t trace a bad prediction back to the raw data that caused it.
Versioning needs clear rules about what’s immutable and what isn’t. A feature’s schema and computation logic should be versioned explicitly, with old versions kept queryable for models still depending on them, while metadata like the owner or description can be updated in place without a version bump. Treat a change to computation logic as a new feature version, never a silent overwrite. A model trained against version 3 of a feature should never silently start receiving version 4 in production.
Access control should follow standard role-based patterns: read access for consumers, write access restricted to the owning team, and an audit trail logging who changed what and when. Usage tracking, recording which models consume which features, turns a deprecation from a guessing exercise into a two-minute query.
How do you monitor feature freshness and catch drift early?
Feature-layer monitoring catches problems earlier than model-level monitoring alone, because a feature can degrade for weeks before a model’s aggregate accuracy metric moves enough to trigger an alert. Two categories of signal matter here: freshness and distribution.
Freshness monitoring tracks the gap between a feature’s last update timestamp and now, alerting when that gap exceeds its declared SLA. A feature with a five-minute freshness SLA that hasn’t updated in twenty minutes is a silent failure until something checks for it. Distribution monitoring tracks null rate, mean, standard deviation, and population stability metrics like PSI or KL divergence against a rolling baseline, catching upstream schema changes or broken source pipelines before they poison every downstream model.
When a check fails, quarantine the affected feature version rather than letting it flow into serving, and roll back to the last known good version while the team investigates. A tool like Sentient Concepts’ data drift detection guidance covers the broader statistical toolkit for setting these thresholds sensibly rather than chasing every minor fluctuation with a page alert.
What operational practices keep a feature store reliable at scale?
Partitioning strategy differs by store. Offline tables partition by event date and entity type for pruning efficiency; online stores shard by entity key hash to spread load evenly across nodes and avoid hot spots when one entity type dominates traffic.
Scaling the online store usually means horizontal sharding plus autoscaling tied to request volume, since read-heavy inference traffic spikes unpredictably around business events. For organisations running multiple workspaces or regions, a hub-and-spoke registry model lets a central hub hold the discoverable catalogue while spokes consume features locally, avoiding messy peer-to-peer replication across teams.
Backfill planning deserves a written runbook, not improvisation. Recompute incrementally by partition, checkpoint progress, and always validate a sample against the old values before rolling the new backfill out fully. Incremental recompute, rather than full history rewrites, keeps both compute cost and blast radius under control when something needs fixing.
When does a team actually need a feature store?
The clearest signal is reuse: once two or more teams are independently rebuilding the same customer or transaction features, a feature store pays for itself in avoided duplication and avoided skew. A second signal is freshness pressure, when a model needs feature values that are minutes old rather than a nightly batch.
The trade-off is operational cost against speed to production. Feast, an open-source option, suits teams with engineering capacity to run their own infrastructure and no urgent streaming requirement. A managed, opinionated platform suits teams that need streaming freshness SLAs immediately and would rather pay for that capability than build it.
The practical migration path starts small: one registry, batch materialisation, and a handful of shared features. Expand into streaming and a dedicated online store only once freshness or latency requirements genuinely demand it, not because a roadmap slide said so.
A design checklist for building a feature store
Work through these steps roughly in order, adjusting for your team’s existing infrastructure:
Define non-functional requirements first: freshness SLA per feature category, p99 latency target for online lookups, and consistency tolerance between offline and online values.
Choose offline storage (Parquet or Delta on object storage) and online storage (a key-value store meeting your latency target) as separate, independently optimised backends.
Design the feature registry schema, including owner, computation logic, schema version, freshness SLA, and lineage, before writing a single feature.
Plan materialisation scheduling per feature, decide batch versus streaming per feature category, and write the dual-write and gap-detection logic before onboarding your first real feature.
Define your backfill runbook, including checkpointing and sample validation, before you need to run one under pressure.
Build parity tests between offline and online values and wire freshness and distribution monitoring into alerting from day one, not as a later addition.
Document RBAC and audit logging requirements alongside the registry schema, not as a bolt-on after the first security review.
Pro Tip: Build the parity test and the freshness monitor before you onboard your second feature, not your fiftieth. Retrofitting monitoring onto an established registry means months of undetected skew you’ll never fully account for.
Sentient Concepts’ view on building feature platforms that hold up
Feature store projects fail less often on architecture diagrams and more often on the operational grind that follows: the third dual-write failure at 2am, the backfill that takes longer than planned, the freshness SLA nobody’s watching. Sentient Concepts approaches feature platform work as a continuous engagement rather than a handover, staying accountable through data and platform engineering, deployment, and the managed operations phase that follows launch.
That continuity matters most in finance and manufacturing, where feature pipelines feed document automation and predictive models that can’t tolerate silent skew. Clients working with Sentient Concepts on data and platform engineering get architecture built for their actual freshness and latency requirements, not a generic template, plus the operational continuity to keep monitoring, backfills, and governance running long after the initial build.
For teams weighing whether to run feature store operations in house or hand off the ongoing burden, managed AI operations covers exactly that gap: SLA management, drift monitoring, and remediation without adding permanent headcount.
— Thomas Samuel
Sources
FAQ
What Is a Feature Store in Machine Learning?
A feature store is a system that computes, stores, and serves machine learning features consistently for both training and production inference, combining a registry, an offline store, an online store, and a materialisation engine.
Feast vs Tecton: Which Should You Choose?
Choose Feast when you have engineering capacity for self-managed infrastructure and no urgent streaming requirement; choose a managed platform like Tecton when streaming freshness SLAs and faster time-to-production justify the added operational cost.
Why Does Training-Serving Skew Happen?
Training-serving skew happens when the offline and online paths compute a feature using different logic, different timing, or inconsistent point-in-time joins, causing a model to see different values in production than it saw during training.
What Latency Should an Online Feature Store Target?
Most production inference paths need sub-10ms p99 lookups from the online store, which typically requires an in-memory or hybrid key-value store rather than a general-purpose database.
How Often Should Features Be Materialised?
Materialisation frequency should match each feature’s freshness SLA individually. Slow-changing features suit hourly or daily batch jobs, while features tied to real-time decisions need streaming materialisation that updates within seconds.
Recommended