For the past decade, data lakes have been the default repository for AI training data. Centralize everything in S3 or ADLS, throw Spark at it, and let data scientists query away. That model worked when teams were small and models were simpler. But in 2025, as AI pipelines ingest petabytes from dozens of sources and models require continuous fine-tuning, the monolithic data lake is cracking under its own weight. Stale schemas, silent data drift, and cross-team contention for compute resources have become daily friction points. A growing number of engineering leads are turning to data mesh—a decentralized architecture that treats data as a product owned by individual domains—to unblock their training pipelines. This article breaks down why data mesh is gaining traction, where it outperforms data lakes for AI workloads, and what it costs to make the switch.
A data lake centralizes storage and often compute, which creates a single point of contention. When five ML teams each need to read 50 TB of log data simultaneously for retraining a recommendation model, the lake's metadata layer becomes a bottleneck. Hive Metastore or Glue Catalog queries start taking seconds instead of milliseconds. Worse, schema changes made by one team can silently corrupt another team's training dataset. At a 2024 ML ops conference, a lead engineer from a mid-size fintech reported that 30% of their training pipeline restarts were caused by schema drift in shared lake tables that no single team owned. Data lakes also encourage a "dump everything, fix later" culture, which leads to low data quality. Training on stale or incomplete data produces models that degrade in production—sometimes catastrophically for fraud detection or real-time pricing. The central governance team, usually understaffed, ends up as a gatekeeper, slowing every pipeline change by days or weeks.
When data quality issues arise in a lake, root cause analysis is painful. You have to trace lineage through ingestion jobs, transformation steps, and catalog updates that may have been written by different teams months apart. Data mesh forces each domain to publish data products with their own schema, freshness SLAs, and ownership. This makes it straightforward to assign blame and fix issues at the source, which directly reduces the number of failed training runs.
Data mesh, first formalized by Zhamak Dehghani, flips the lake model on its head. Instead of a central store, each business domain (e.g., payments, fraud, recommendations) owns its data and exposes it as a curated product. For AI pipelines, this means the fraud team produces a "fraud_features" dataset with documented schemas, freshness guarantees, and access policies. The recommendation team consumes that product directly, without needing permission from a central data team. This reduces pipeline latency because consumers don't wait for a central ETL job to finish—they pull the latest version when needed. In practice, companies like Zalando and Intuit have reported cutting data preparation time for ML models by 40–60% after shifting to a mesh model. The key technical enabler is a data catalog that supports federated governance: each domain retains autonomy, but global policies (PII redaction, retention) are enforced at the catalog level rather than at the storage level.
In a mesh, each data product carries its own metadata. For AI use cases, the most important metadata fields are schema version, distribution statistics (min, max, null ratio), and training/validation split timestamps. Tools like DataHub or Apache Atlas can capture this lineage automatically when a domain publishes an update. This dramatically simplifies reproducibility: when a model's accuracy drops, you can replay the exact dataset versions used during training. Data lakes can achieve similar lineage tracking, but it requires a centralized team to enforce conventions across all domains—a near-impossible task at scale.
Latency is the silent killer of AI training pipelines. Every minute a data scientist waits for a Spark job to scan a lake table is a minute they are not iterating on model architecture. In a controlled migration at a large e-commerce company, the analytics engineering team measured the time from a new product launch (e.g., a new discount code format) to that data being available for model retraining. Under the data lake model, it averaged 3.2 days—central ETL had to be updated, tested, and deployed. After moving the discount data to a domain-owned product with a streaming output, that latency dropped to 15 minutes. For time-sensitive models like dynamic pricing or inventory forecasting, that gap translates directly into revenue. Data mesh also eliminates the "thundering herd" problem where multiple training jobs simultaneously hammer the same lake partition. Each consumer reads from the data product's serving layer, which can be optimized independently—often a low-latency store like Apache Iceberg or Delta Lake with clustering on training-relevant keys.
Critics argue that data mesh duplicates storage, raising costs. That's true on the surface: each domain may maintain its own copy of transformed data. But for AI pipelines, the economics are more nuanced. In a data lake, every training job pays for a full scan of a lake table (often hundreds of terabytes), even if it only needs a subset of columns or partitions. With data mesh, the domain exports a pre-joined, columnar-optimized dataset sized for the model's actual input dimensions. For a fraud detection model that consumes 500 features, the curated product might be 5 TB instead of the lake's 200 TB. Training jobs run faster, and compute costs drop proportionally. A 2025 survey by a cloud cost management firm found that teams using a mesh approach for model training reduced their compute spend by an average of 35%, mainly because they stopped scanning irrelevant data. The trade-off is higher storage costs for the curated copies (often 2–3x raw storage), but for organizations where compute dominates the bill, the net saving is positive.
There is a threshold below which mesh overhead outweighs benefits. If your organization has fewer than three ML teams, or if model inputs change less than once a month, the central lake with a single curator is more economical. The mesh adds operational overhead: each domain needs someone to maintain the data product pipeline, write documentation, and respond to consumer queries. For small teams, that person is the same person building the models—and the context switching is painful.
Let's make this concrete. A media streaming company with 40 million users ran its content recommendation models off a central data lake built on S3 and Spark. The training pipeline ingested users' watch history, catalog metadata, and real-time interaction logs. The lake had grown to 50 TB, and retraining a model required scanning 35 TB of data, taking 6 hours. Data freshness was a constant complaint: new content wasn't reflected in recommendations for 48 hours. Their migration to data mesh took 9 months and followed three steps:
The migration cost roughly $200k in engineering time, but the company recouped that in 5 months through lowered compute spend and faster iteration cycles.
Data mesh hands control to domain teams, which is a double-edged sword for AI. One domain might define "active user" as someone who logged in within 7 days; another domain uses 30 days. When both products feed into the same multi-task model, the inconsistency can degrade performance. Centralized lakes avoid this by enforcing a single definition at the ingestion layer. The mesh solution is a global semantic layer—a shared ontology that domains agree on, enforced at the catalog level. Tools like dbt's model contracts or Great Expectations can validate that each data product adheres to agreed-upon ranges and formats before it's published. Without this, mesh can devolve into chaos. At a large bank, one team published a "loan_risk" product with a column named "fico_score" that actually contained a modified score inside their domain. The consumer model trained on this product and failed catastrophically in production. The lesson: governance in a mesh is not about controlling access; it's about agreeing on semantics and validating them automatically.
Begin with three mandatory checks for every data product used in training: schema match (column names and types), distributional boundaries (null rate below 5%, min/max within expected range), and freshness (last updated time within agreed SLA). Automate these checks as part of the product publishing pipeline, and alert both the producer and all consumers on failure.
Data mesh is not a universal upgrade. It works best when your organization has multiple distinct business domains, each with its own data semantics and velocity, and when your training pipelines are suffering from data staleness or cross-team interference. If your current pain is primarily compute cost or slow query performance on a single large table, you might be better served by converting the lake to a lakehouse format like Delta Lake with Z-ordering and partition pruning. But if the bottleneck is organizational—teams stepping on each other's schemas, long wait times for central ETL changes, or inability to trace data quality issues—then mesh addresses the root cause. The trend in 2025 is clear: as AI becomes embedded in every product feature, the data infrastructure must mirror the decentralization of the teams building those features. Data mesh is one answer, and it's gaining production adoption not because it's trendy, but because it solves a specific, expensive problem.
If you are evaluating a migration, start small: pick one domain with a well-understood dataset and convert it to a data product. Measure the time from data generation to model training start, and track the number of training pipeline failures caused by data issues. Run that experiment for two months before expanding. You'll likely find that the upfront investment in domain tooling and semantic validation pays for itself in fewer failed training runs and faster model iteration.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse