To succeed with generative AI at scale, we need to give LLMs the diligence they deserve. Enter RAG and fine tuning. — The hype around LLMs is unprecedented, but it’s warranted. From AI-generated images of the Pope in head-to-toe Balenciaga to customer support agents without pulses, generative AI has the potential to transform society as we know it. And in many ways, LLMs are going to make data engineers more valuable —…

Getting started with LLMs? Here are 5 popular applications data teams at OpenAI, Vimeo, and other companies are putting into practice today. — The hype around generative AI is real, and data and ML teams are feeling the heat. Across industries, executives are pushing their data leaders to build AI-powered products that will save time, drive revenue, or give them a competitive advantage. And tech giants like OpenAI, Google, Amazon, and Microsoft have…

Outlining the past, present, and future of architecting reliable data systems. When we launched the data observability category in 2019, the term was something I could barely pronounce. Four years later, the category has squarely established itself as a core layer of the modern data stack. Data Observability is a…

Spinning up a data platform doesn’t have to be complicated. Here are the 5 must-have layers to drive data product adoption at scale. — Like bean dip and ogres, layers are the building blocks of the modern data stack. Its powerful selection of tooling components combine to create a single synchronized and extensible data platform with each layer serving a unique function of the data pipeline. Unlike ogres, however, the cloud data platform isn’t…

If you want to build data trust, you can’t just gut-check the final output. You need to monitor data as it is ingested and flows across your data pipeline. Here’s why. I link to Benn Stancil in my posts more than any other data thought leader. I might not always…

The characteristics and value of reactive vs. proactive data teams — Fundamentally, there are two different types of data teams in this world. There are those who are reactive to the wants of the organization, and then there are those who proactively lead the organization towards its needs. The first is helpful, but a cost center. The second is a value…

I wanted to share this article we put together that illustrates how real data teams are benefiting from data observability. Data observability, an organization’s ability to fully understand the health and quality of the data in their systems, has become one of the hottest technologies in modern data engineering. In…

Data teams are more important than ever before — but they need to get closer to the business. Here’s how we can right the ship. — Over the past decade, data teams have been simultaneously underwater and riding a wave. We’ve been building modern data stacks, migrating to Snowflake like our lives depended on it, investing in headless BI, and growing our teams faster than you can say reverse ETL. Yet, much of the time we…

The post-modern data stack is coming. Are we ready? — If you don’t like change, data engineering is not for you. Little in this space has escaped reinvention. The most prominent, recent examples are Snowflake and Databricks disrupting the concept of the database and ushering in the modern data stack era. As part of this movement, Fivetran and dbt fundamentally…