The data steward has an impossible job. Here’s why and what we can do to empower them. — In his seminal 2017 blog post, The Downfall of the Data Engineer, Maxime Beauchemin wrote that the data engineer had the worst seat at the table. Data technology and teams have changed tremendously since that time, and now the Preset CEO and creator of Apache Airflow and Apache Superset has…

Data lineage should be less like a treasure map and more like Google Maps — Data lineage isn’t new, but automation has finally made it accessible and scalable — to a certain extent. In the old days (way back in the mid-2010s), lineage happened through a lot of manual work. This involved identifying data assets, tracking them to their ingestion sources, documenting those sources, mapping…

The data warehouse is the foundation of the modern data stack, so it caught our attention when we saw Convoy head of data Chad Sanderson declare, “the data warehouse is broken” on LinkedIn. Of course, Chad isn’t referring to the technology, but how it’s being used. As he sees it…

On average, companies experience one data issue for every 15 tables in their warehouse. Here are 8 reasons why and what you can do. — Say it with me: your data will never be perfect. Any team striving for completely accurate data will be sorely disappointed. Unit tests, anomaly detection, and cataloging are important steps, but technology alone will not solve your data quality problem. Like any entropic system, data breaks. And as we’ve learned…

Unpacking the newest data role on the modern data team: the data product manager — This article was co-written with Atul Gupte. The companies we talk to are diligently building their data product or platform. This includes migrating to Snowflake, integrating with Databricks, moving towards a data mesh, or generally investing in their data stack. Increasingly, we are seeing data departments modernize their team structure…

And when it makes sense to get started. — One of the most frequent questions I get from customers is: “when does it make sense to invest in data quality and observability?” My answer is, “it depends.” The reality is that building a data platform is a multi-stage journey and data teams have to juggle dozens of competing priorities…

Here are 5 important questions to answer when migrating to the self-service data platform of your dreams — Building a data platform — i.e., a central repository for all your company’s data, which enables the acquisition, storage, delivery, and governance of that data while maintaining security across the data lifecycle — has become a rite of passage for today’s data teams. Data platforms are critical in a modern…

How to build a data certification program so everyone knows what to expect and what data to trust. — Data trust starts and ends with communication. Here’s how best-in-class data teams are proactively certifying tables as approved for use across their organizations. Say it with me: data engineers are not data catalogs. You would be hard pressed to find “answering multiple Slack messages every week about which tables are…

Your team wants to “treat data like a product.” Great! Now what? — For the past few decades, most companies have kept data in an organizational silo. Analytics teams served business units, and even as data became more crucial to decision-making and product roadmaps, the teams in charge of data pipelines were treated more like plumbers and less like partners. In the 2020s…