The Broken Data Pipeline Before Christmas
A cautionary tale for data teams this holiday season
What happens when a freshness anomaly threatens to ruin Christmas? Turns out, even Santa Claus and his elves aren’t immune to broken data pipelines. With a little help from a classic holiday poem, here’s the story of how data downtime nearly derailed Santa’s workshop and the wise elf who saved the day.
’Twas three nights before Christmas, and in the North Pole
The elves were all frantic, they feared for their roles;
Their prep meeting with Santa was a total disaster,
“This gift list looks wrong!” He chanted louder and faster;
The elves Slacked the data team, “Santa’s freaking out!
Our gift list is wonky. What’s he talking about?”
Papa Elf responded, with utmost haste:
“Are you still pulling from list_2020? That table’s a waste!”
One elf fainted, one wept, one fell out of her chair,
Two spilled their mulled ciders as they read on with despair,
“We’re migrating warehouses, from SledLift to Iceberg.
We haven’t updated list_2020 since June, haven’t you heard?”
Their rosy cheeks burned redder, they were mad as can be;
How come no one told them? So long, data reliability!
list_2020 was their bedrock, the one source of truth;
But the news only got worse as the data team sleuthed.
Wish lists from some nations never made it through ingest;
Long country names caused a problem that was never addressed.
The elves knew the huge challenge they must now undergo,
To make presents in time for the Democratic Republic of Congo.
And they’d built 2 million bikes, the effort burned out the staff;
But now learned true demand was really just half;
A bad merge was the source of this terrible trouble,
For each bike that was wished, the elves had built double!
They sulked and they moaned, but then pulled up their boots;
After all, elves are tough beings with joy-bringing roots;
“We’ve brought sleigh-fulls of happiness for so many years,
Bad data won’t stop us!” They said as they wiped away tears.
They went to Santa to share what was the matter;
Shocked, he dropped his hot chocolate, causing a splatter.
He said “There’s no time to dwell now. We can still salvage this!
But first thing in January, I want a root cause analysis.”
For 48 hours, it was all hands on deck,
Even Rudolph and Dasher took a terabyte to cross-check.
Elves reloaded the wishes and fixed null destinations,
Repacked toys with great care, then did three more validations.
When it was finally time for Santa’s big flight,
They sighed with relief as he flew out of sight;
Some gathered at the tavern, a bleary-eyed quorum,
To sip peppermint schnapps and begin their post-mortem.
“Data’s the future,” they all nodded and knew,
“But it’s no good when it’s faulty or looks all askew.
These last 48 hours, this data downtime,
Was a dreadful experience, a work-life balance crime!”
At that, Papa Elf, the most wise of the bunch,
Looked up and said slowly, “I have a hunch!
There’s a way to know before the data turns wonky,
So next season we don’t look like such a big donkey.”
They clung to his words, like snow to a tree,
And got their very first lesson on ‘data observability.’
“We can monitor our data pipelines, from end-to-end,
So Santa’s list is reliable, and he’ll be forever our friend!”
With quick flicks of his pen, Papa Elf set new OKRs,
To the cheers of his team that “the future is ours!”
So what’s next for Santa in 2021, you ask?
It’s simple: preventing data downtime is their very first task.
If data downtime is something you’ve experienced, I’d love to hear from you! Reach out to Barr Moses and the rest of the Monte Carlo team.
This poem was written by Will Robins, Head of Growth at Monte Carlo.
