Cloud centers of excellence are hailed as the best approach for businesses looking to centralize their cloud computing practices. According to Gartner, centers of excellence or CoE are enterprise architecture functions that help companies “set cloud policy, guide provider selection, and assist with solution architecture and workload placement, with the goals of improving outcomes and managing risks.”
As an experienced systems integrator with decades of experience, Hakkoda has created a series of centers of excellence in data migration, modernization and strategy for clients across industries. As a result, both Hakkoda’s data experts and our clients can leverage specialized knowledge and resources to ensure the successful execution of their data initiatives. The CoE plays a vital role in driving efficiency, reducing risks, and maximizing the value of data assets throughout the organization.
Many automated tools comprise Hakkoda’s migration center of excellence, including accelerators like Automated Query Performance Tester (AQPT), Ampr, and Tundra, among others. Learn how a large healthcare organization leveraged Hakkoda’s migration center of excellence to reduce costs and migration timelines while increasing efficiency.
Tackling Large-scale Data Migrations
At a major healthcare organization, during a Teradata to Snowflake migration that involved more than 12 databases and two petabytes of total data, the Hakkoda team of engineers was asked to drive success end-to-end throughout the organization’s migration. This type of large migration can take months, if not years for organizations to run on their own, and has many levels of transfer and optimization.
Hakkoda was initially engaged to run a 1 month POC that answered 3 questions:
- Performance: Can Snowflake handle the queries and dashboards that are struggling on Teradata?
- Value: Will Snowflake save the large healthcare organization money vs. Teradata?
- Satisfaction: What would the engagement look like once we proved the Value and Performance of Snowflake?
In the POC, Hakkoda focused on migrating the largest table of data, delivering a high value/high usage dashboard and looking at problematic (expensive) queries. The POC was a success, showing that Snowflake is the best tool for the job.
After the POC, Hakkoda worked with the large healthcare organization’s data teams to structure the full migration engagement. Three work streams were created to each migrate a number of the databases in parallel over an 8 month period. Each workstream would ensure DDL conversion and migration of existing data was done correctly. Pipelines were connected to Snowflake for new/changed data, data was validated to ensure quality, and all necessary BI dashboards were rebuilt to work with Snowflake.
The real power of Snowflake was quickly apparent as the team was able to leverage different warehouse sizes for different workloads. The Hakkoda HCLS team worked to not only optimize queries, but to optimize warehouse sizing and clustering as well. We started with a small warehouse to minimize cost and tuned it upward from there to find the sweet spot.
The Power of a Migration Center of Excellence
After the POC, the Hakkoda Healthcare and Life Sciences (HCLS) team worked with the large healthcare organization’s data teams to structure a full migration engagement. To do this, they created three work streams responsible for migrating a number of the databases in parallel. Once the data was shifted, much of the real work began for the team.
Normally, it is extremely labor intensive to manually test all existing queries to ensure configuration changes did not negatively impact the platform. It is critical to know how queries perform in your new Snowflake environment. Part of Hakkoda’s Migration Center of Excellence includes a host of automated tools built to address specific client needs and to rapidly accelerate the migration process while reducing manual effort needed from client data teams.
One of these includes a scalable accelerator to run through and automatically execute a batch of queries. Whether you have a couple of queries or thousands of queries, Automated Query Performance Tester (AQPT) can handle the job.
Hakkoda’s Approach to a Migration Center of Excellence
Without a tool like the Automated Query Performance Testing framework (AQPT) as part of the migration center of excellence, teams are forced to manually test hundreds of queries. In the case of the large healthcare organization, the initial testing period took three data engineers four days to run the queries and compile the associated metrics. Using Hakkoda’s AQPT, however, the healthcare organization was able to achieve a higher level of testing in only one day and by leveraging a single resource. That’s 16x faster testing.
Overall, the engagement was a massive success, resulting in a 60% cost reduction on Snowflake versus Teradata. Aside from cost, AQPT clearly improved query performance and in some cases presented up to 85% improvements in terms of speed.
Finally, because all alerts and monitoring were put in place to proactively monitor performance, experts could easily run applications and dashboards directly on Snowflake. This allowed a quick decommissioning of the legacy platforms.
Continuously Innovating with Hakkoda
Hakkoda did not stop with AQPT. As the team continued to support Snowflake end users, we saw an opportunity to make use of Snowflake’s innovative technology. Snowflake is great for data storage but it is much more than that. At this major healthcare organization, the team is now working on using Snowflake’s native capabilities like Snowpark to produce and productionalize data science models.
Make sure to fully leverage the power of AQPT as part of your Snowflake migration. This invaluable tool not only saves time but also offers essential insights, easy performance comparisons, and a user-friendly experience. AQPT enables you to fine-tune your queries for optimal performance on Snowflake, allowing you to maximize the benefits of this advanced data analytics platform.
By embracing AQPT, your organization can supercharge your Snowflake migration and unleash the full potential of your data analysis. Prepare to enjoy enhanced efficiency, increased productivity, and remarkable insights with Hakkoda’s Automated Query Performance Tester.