As with past crises, COVID-19 is likely causing you to consider how you can reduce your data costs. Now is a great time to act on ways you can reduce costs while not disrupting your operations.
Several factors contribute to data costs:
• Managing relationships with multiple data providers
• Handling numerous data subscriptions
• Fees for data
• Data compliance
• Data engineering and IT
Below, I discuss these challenges and offer a few thoughts on what you can do to alleviate costs.
In addition, James Marshall, Product Manager for Clear Capital, weighs in with some of his own valuable insights and strategies. Clear Capital is a leader in real estate property valuation management and data solutions.
5 Ways to Reduce Data Costs
1. Understand Data Contracts, Services, and User Consumption
It is critical to know when your existing data contracts are due to renew, and when you can cancel your subscription without a penalty.
Keep track of which data services are being used by whom within your organization. If there is a data service that is redundant or underutilized, cancel it!
Reconcile all data invoices and allocate costs accordingly.
Finally, provide detailed reports to decision makers and end-users for data expense allocations and budgeting.
2. Validate Data Needs and Usage
Regularly review data spending with end-users. This makes it easier to identify data services that are redundant or could be downgraded.
For GUI data services with per-seat charges, encourage end-users to share terminals.
Clear Capital’s James Marshall offers this sound advice for data managers balancing the needs of end-users with a company mandate to cut costs:
”When initially procuring a data set, it’s easy to say, ‘We need all of the data you offer,’ because you never know how you’re going to utilize it. After a couple years of usage, it becomes more apparent what data is necessary based on your usage patterns. Upon renewal of a contract, it’s worthwhile to discuss if there are options to pare down the data delivery to only the essentials to take advantage of lower pricing.”
3. Optimize Your Data Sourcing Process
A standardized process for sourcing new data can help reduce your overall data spend.
First, create a schedule of when data services renew to give yourself time to consider less-expensive alternatives.
Use templates so end-users can fully document requirements, and potential data providers can answer critical questions about the sources of raw data, how much history they will provide, frequency of updates, file formats, delivery methods, etc.
For data samples, have pre-defined parameters, so you can get the same sample from each potential provider for easier apples-to-apples comparisons. Designate who will review the data samples and what criteria they will use. Such criteria must align with requirements and the use case.
Set a deadline for completing the review. Too many data projects get delayed because the individuals involved don’t have the time/urgency to complete due diligence tasks. In some cases, a longer data trial or proof of concept (POC) may be required to effectively evaluate a data set.
When possible, don’t be afraid to include upstart providers in the process that are likely more flexible on pricing and contract terms.
James at Clear Capital adds a good point about “data sustainability”:
”Data consumers often forget to ask how the data provider is set up to ensure consistent delivery for several years into the future. This necessitates learning a prospective vendor’s data gathering techniques, their sources, and the compliance of their contracts behind the information being sold (e.g., screen scraping versus direct license agreements with data sources).”
James also advocates not basing your decision solely on price:
”You should dive deep into the standardization and quality practices of a prospective data vendor to potentially save hundreds of thousands of dollars in operating expenses in the future. It’s the old adage that you’ll eventually pay for an expert — either now, or in the future.”
Finally, ask data providers to be flexible. If you only need seven fields in the file, ask for just those fields (don’t pay for the entire file). If the end-user will be analyzing the data using Excel, ask the data provider to send the file in Excel format.
4. Ensure the Most Favorable Commercial Terms When Choosing a Data Provider
Take the time to conduct regular vendor meetings so you know what data is available and what is in the pipeline. This enables you to have a substitute source cued up for each data provider. If you want/need to switch, it will be less painful.
Data fees are important (of course), but don’t forget the fine print (e.g., payment terms, renewal terms, duration of subscription, data sharing restrictions, etc.). Avoid “purge clauses” that demand you delete all of the data you’ve already paid for if you ever decide not to renew the contract.
5. Understand Your Data On-Boarding Process
Implementing a new data service, or switching providers for an existing service, is often a pain point (I’ll have more about data on-boarding in a future post).
First, determine who exactly will be managing the on-boarding project. Then nail down the following:
• What is his/her availability?
• How soon can they start the process?
• How many hours will the on-boarding project realistically require?
If you lack internal resources, don’t be afraid to ask the data provider for assistance. They may already have loaders you can use. Perhaps a different file format or delivery method would expedite the on-boarding.
When it comes to getting your data engineers involved as soon as possible, Clear Capital’s James Marshall offers this insight:
”When a new customer signs up with Clear Capital, we ask for a meeting that includes the data engineers who will be interacting with the data. This allows them to have a direct explanation of the best practices for working with the specific dataset in question. The benefit is a much smoother process when working with the data, and limits the number of dead-ends a data engineer can run into based on the nuance of the product/service.”
James also offers some practical advice on the importance of being mindful of data storage costs:
”At Clear Capital, we are self-aware data hoarders. We are getting better at maximizing our storage options based on the needs of the products. We’ve found that it’s worth the investment to spend time architecting data access that utilizes a balance of fast and slow storage determined by product usage patterns. For example, if one percent of our end-users are requesting 90 percent of the information available, it makes sense to use an on-demand storage solution that keeps 90 percent of the data on a cheaper platform, such as S3, while keeping the most popular 10 percent at our end-users’ fingertips.”
These are just a few of things you can do to reduce your data costs, now and with an eye toward the future.
I’ll be expanding these ideas in an upcoming white paper. Sign up here to receive the white paper.
Also, you can connect with us on:
Stay Healthy. Stay Safe.
Tom Myers is the founder of Data In Harmony (DIH), a data consultant and provider. DIH help firms find the data they need, validate & clean data, integrate data, and monetize their data. DIH also provides a wide variety of financial and alternative data, as well as data engineering tools.