The Modern Data Stack has matured significantly. After several years of explosive tooling growth, the dust is settling and we can see clearly which tools have earned their place in production and which were overhyped.
Here’s my current take after running these tools in real engagements.
What’s Actually Working#
dbt — Still the Foundation#
dbt has won the transformation layer. The combination of SQL templating, testing, documentation, and lineage in a single framework is simply unbeatable for most use cases. Every data platform I build today has dbt at the center.
The critical workflow: raw → staging → intermediate → mart. Simple, traceable, testable.
Airflow vs Prefect — It Depends#
Airflow remains the enterprise standard. The ecosystem is massive, the community is huge, and managed offerings (Astronomer, MWAA, Cloud Composer) have removed much of the operational pain.
Prefect is the better developer experience. If you’re a smaller team or starting fresh, Prefect 2.x is genuinely enjoyable to work with. Dynamic task mapping and the Python-native API are a real step forward.
Snowflake vs Databricks — Choose Your Battle#
Snowflake if you’re primarily doing SQL analytics and BI. The separation of compute and storage, instant scaling, and zero-ops model are hard to beat.
Databricks if you need a unified platform for both data engineering and ML. The Delta Lake format, Unity Catalog, and tight Python/Spark integration make it the better choice for ML-heavy workloads.
What I’m Watching Closely#
Apache Iceberg is winning the open table format war. Snowflake, Databricks, and every major cloud provider are embracing it. If you’re building a new lakehouse, build on Iceberg.
DuckDB for local development and smaller datasets is surprisingly powerful. It’s changed how I prototype pipelines.
What’s Overhyped#
Real-time everything. Most businesses genuinely don’t need sub-second latency. A well-designed batch pipeline refreshing every 15 minutes solves 90% of analytics use cases at a fraction of the complexity and cost.
Want to discuss how to build a data platform that’s right for your team’s needs and budget? Get in touch.