Sep 14, 2022
You are absolutely right on both counts.
It is extremely difficult to maintain full lineage and observability without automation. Here is how Fox Networks did it: https://www.montecarlodata.com/blog-data-reliability-at-scale-how-fox-digital-architected-its-modern-data-stack/
Great point about the failures of logistics systems in recent months. Another reason why we should never believe systems are perfect and build accordingly!