For modern data teams, data orchestration is no longer a luxury. It’s a must-have. As organizations scale, data sets multiply, and analytics demands grow, manual data management becomes a bottleneck. The key to solid data management lies in automating and coordinating every stage of your data pipeline. Why Data Orchestration Matters A data pipeline is
At a data meetup I attended recently, I overheard two people having a more-animated-than-expected conversation about data version control. Each had strong feelings about how best to track and manage changes to data files over time. One point they both agreed upon is that Git is a very useful tool, but maybe not optimal for
Not too long ago, I was working with an investment advisor who needed bond reference data. Her firm had recently brought on a bunch of new data scientists (a.k.a. quants in capital markets), and they were struggling to get their existing data engineering team in sync with their new colleagues. A common challenge for organizations
I recently provided corporate actions data to an asset manager who had recently missed a voluntary event due to bad data. In this case, the bad data caused a monetary loss. Depending on your industry or the systems relying on the data, the impact of bad data can be much worse. So let’s explore how
As you might imagine, much of the work I’m doing nowadays is with companies trying to leverage AI. Each company and use case is different, but a common issue is data quality. So let’s take a look at why data quality issues arise, and how to prevent them… Every data team has faced the frustration
It’s a challenge humans have been struggling to overcome since the dawn of time. Building a solution that is fast, but also reliable. In Formula 1 auto racing, teams strive to build the fastest car possible. However, if their car never makes it to the finish line due to technical failures, it doesn’t matter how
I recently had lunch with the founder of a business analytics company whom I mentor. Over some very tasty interior Mexican food, he vented about how his clients do not fully understand how hard it is to get data right every day. If I got a dollar every time I heard this, I’d be having
Agentic AI systems don’t just passively receive commands. They plan, reason, and act autonomously to accomplish goals. But even the most capable autonomous agent is only as smart as the data scaffolding around it. The difference between brittle execution and adaptive autonomy? Complete, accurate, and structured metadata. To operationalize agentic AI, particularly in commercial environments,
There are so many new (cool) things happening with data nowadays, it’s easy sometimes to overlook the basics. So I thought it would be helpful to look at some “best practices” for setting up your data workflows. Effective data workflows are essential for maintaining data quality, promoting collaboration, and driving reliable insights across an organization.
I recently spoke with the head of data and analytics at an investment management firm about how 2025 was going so far for him and his peers in the industry. Maybe not surprisingly, he said it’s been challenging. Setting aside geopolitics and market conditions, he said, throughout the industry, data engineers and data scientists are
