Outlining the past, present, and future of architecting reliable data systems. When we launched the data observability category in 2019, the term was something I could barely pronounce. Four years later, the category has squarely established itself as a core layer of the modern data stack. Data Observability is a…

Spinning up a data platform doesn’t have to be complicated. Here are the 5 must-have layers to drive data product adoption at scale. — Like bean dip and ogres, layers are the building blocks of the modern data stack. Its powerful selection of tooling components combine to create a single synchronized and extensible data platform with each layer serving a unique function of the data pipeline. Unlike ogres, however, the cloud data platform isn’t…

If you want to build data trust, you can’t just gut-check the final output. You need to monitor data as it is ingested and flows across your data pipeline. Here’s why. I link to Benn Stancil in my posts more than any other data thought leader. I might not always…

The characteristics and value of reactive vs. proactive data teams — Fundamentally, there are two different types of data teams in this world. There are those who are reactive to the wants of the organization, and then there are those who proactively lead the organization towards its needs. The first is helpful, but a cost center. The second is a value…

I wanted to share this article we put together that illustrates how real data teams are benefiting from data observability. Data observability, an organization’s ability to fully understand the health and quality of the data in their systems, has become one of the hottest technologies in modern data engineering. In…

Data teams are more important than ever before — but they need to get closer to the business. Here’s how we can right the ship. — Over the past decade, data teams have been simultaneously underwater and riding a wave. We’ve been building modern data stacks, migrating to Snowflake like our lives depended on it, investing in headless BI, and growing our teams faster than you can say reverse ETL. Yet, much of the time we…

The post-modern data stack is coming. Are we ready? — If you don’t like change, data engineering is not for you. Little in this space has escaped reinvention. The most prominent, recent examples are Snowflake and Databricks disrupting the concept of the database and ushering in the modern data stack era. As part of this movement, Fivetran and dbt fundamentally…

Some organizations approach data quality like a crash diet. Instead, try these 6 steps for sustainable data quality at scale. — I like to think of operationalizing data reliability within the context of physical fitness. You can get in shape with hard work, but staying in shape requires good habits. And above all else it’s a mindset and a lifestyle. You also have to sweat. At least a little bit. No…

And how to avoid falling into the same traps — Everywhere we go in the cloud data space today, we’re hearing one message loud and clear: “you should be thinking about data governance”. It’s a sentiment that we wholeheartedly endorse, but we like to take it a little bit further — you should be thinking about data governance differently. In…