Challenge
Performance testing plays an important role in determining the success of a large data migration by allowing stakeholders to understand how their queries are performing in Snowflake compared to previous systems. This testing also provides data engineers and teams with the important diagnostic data they need to remove bottlenecks and optimize performance within the Snowflake environment.
Unfortunately, this process has historically required data teams to pull performance metrics manually, which is both time-consuming and an inefficient use of resources.
Solution
Hakkoda developed our Automated Query Performance Tester (AQPT) as a scalable accelerator for testing SQL queries in Snowflake against their pre-migration counterparts. AQPT can be used to run and automatically execute large query batches within Snowflake while generating insights on the following performance metrics:
- Compile Times
- Execution Times
- Warehouse and Cluster Size and Override
- Bytes Spilled to Local and Remove Storage
By automating the performance testing process, AQPT allows our clients to make more efficient use of data talent and other resources by freeing them up to focus on other critical tasks. For one of Hakkoda’s large healthcare clients, the use of AQPT resulted in 16 times faster testing and a 66% reduction in resource allocation.