I was chatting with a chief data officer recently who was venting about the challenges his team faces to provide complete, accurate, and timely data while keeping costs down. Sound familiar? Modern data teams face constant pressure to deliver insights faster while controlling compute spend. Choosing the right data transformation strategy can determine how efficiently
Many organizations utilize a monolithic data warehouse. This is a traditional data management architecture where all enterprise data (e.g. sales, marketing, support, finance, etc.) is collected, stored, transformed, and managed within a single, centralized, and tightly coupled system. The advantages of a monolithic data warehouse are that it is easy to manage given its single
Data scientists often face a tough balance between pushing the limits of innovation versus actually shipping models that make it to production. The teams that succeed know how to blend creativity with structure. Here’s how they do it… Start with Operational Constraints Before diving into a new algorithm, ask yourself: What are the latency and
For modern data teams, data orchestration is no longer a luxury. It’s a must-have. As organizations scale, data sets multiply, and analytics demands grow, manual data management becomes a bottleneck. The key to solid data management lies in automating and coordinating every stage of your data pipeline. Why Data Orchestration Matters A data pipeline is
At a data meetup I attended recently, I overheard two people having a more-animated-than-expected conversation about data version control. Each had strong feelings about how best to track and manage changes to data files over time. One point they both agreed upon is that Git is a very useful tool, but maybe not optimal for
Not too long ago, I was working with an investment advisor who needed bond reference data. Her firm had recently brought on a bunch of new data scientists (a.k.a. quants in capital markets), and they were struggling to get their existing data engineering team in sync with their new colleagues. A common challenge for organizations
I recently provided corporate actions data to an asset manager who had recently missed a voluntary event due to bad data. In this case, the bad data caused a monetary loss. Depending on your industry or the systems relying on the data, the impact of bad data can be much worse. So let’s explore how
As you might imagine, much of the work I’m doing nowadays is with companies trying to leverage AI. Each company and use case is different, but a common issue is data quality. So let’s take a look at why data quality issues arise, and how to prevent them… Every data team has faced the frustration
It’s a challenge humans have been struggling to overcome since the dawn of time. Building a solution that is fast, but also reliable. In Formula 1 auto racing, teams strive to build the fastest car possible. However, if their car never makes it to the finish line due to technical failures, it doesn’t matter how
I recently had lunch with the founder of a business analytics company whom I mentor. Over some very tasty interior Mexican food, he vented about how his clients do not fully understand how hard it is to get data right every day. If I got a dollar every time I heard this, I’d be having
