As organizations navigate the path from AI experimentation to complex agentic systems and external-facing experiences, one question becomes increasingly important: Can you trust what your AI is telling customers?
For one global healthcare and consumer products company, that question took on critical importance as it prepared to launch its first GenAI-powered website search experience.
The goal was straightforward: create a smarter search experience that would improve product discovery and customer engagement. But with terabytes of closely regulated healthcare content in play, success couldn’t be measured in terms of speed or innovation alone. Every AI-generated response needed to be accurate, relevant, properly cited, and compliant before reaching customers.
In a word, the organization needed complete confidence that its third-party AI search solution was ready for production. Anything short of that goal would be tantamount to failure.
The Challenge: Evaluating AI at Scale
Launching a customer-facing AI application requires more than functional testing. Teams must evaluate how consistently models generate responses, whether citations are accurate, if hallucinations occur, and whether outputs meet organizational standards for quality and compliance.
In this case, the organization needed to evaluate nearly 1,700 query-and-response pairs generated by its website search platform. Without an established AI evaluation framework or internal auditing capability, however, reviewing every response manually would have required both a significant time sink and considerable subject matter expertise.
The challenge wasn’t simply finding errors. It was identifying the responses that posed risk and guaranteeing similar responses were eliminated before reaching production.
Designing a Trusted AI Evaluation Framework
To help ensure the GenAI search experience was ready for production, the organization engaged Hakkoda, an IBM Company, to design and operationalize an AI evaluation framework that could assess response quality, identify potential compliance risks, and reduce the burden of manual review.
The engagement began with rapid prototyping in Jupyter notebooks, where the Hakkoda team used Claude to develop heuristic checks alongside an LLM-as-judge methodology capable of evaluating hallucinations, citation fidelity, and response relevance.
When Snowflake introduced CoCo (formerly Cortex Code), however, the Hakkoda team saw an opportunity to take the solution further. Rather than managing disconnected notebooks and scripts, Hakkoda rebuilt the entire evaluation framework natively in Snowflake, transforming an early prototype into a governed, repeatable workflow that could scale alongside the client’s AI initiatives.
Operationalizing AI Governance with Snowflake CoCo
Using Snowflake CoCo, Hakkoda rapidly developed a structured, multi-stage evaluation pipeline directly inside Snowflake.
CoCo generated much of the Python and SQL required to build the workflow, allowing the team to focus on what mattered most: designing robust evaluation logic, governance rules, and risk thresholds instead of spending valuable time on implementation.
The resulting solution combined several complementary techniques:
- Rule-based heuristic detection for predefined compliance and quality issues
- LLM-as-judge grading to evaluate hallucinations, citation fidelity, and response relevance
- Query and response similarity analysis to validate consistency
- Structured evaluation tables that tracked results across multiple evaluation runs
Because the workflow was built natively within Snowflake, the client gained a scalable, repeatable evaluation process that can continue supporting future GenAI initiatives.
Reducing Manual Review by More Than 95%
One of the most significant outcomes was that this Snowflake-native solution didn’t simply identify problematic responses. Instead, it introduced significant improvements to review efficiency.
Rather than asking subject matter experts to evaluate all 1,700 query-and-response pairs, the framework intelligently prioritized the results based on predefined risk thresholds.
By the end of the engagement, less than 60 responses of that 1,700 were escalated for detailed SME review. That allowed experts to spend their time validating the responses that mattered most while giving business stakeholders clear, evidence-based reporting on launch readiness.
Delivering AI Confidence at Scale
Thanks to Hakkoda’s CoCo-powered evaluation framework, the organization is starting its next chapter with the total confidence needed to move forward with its customer-facing GenAI search experience.
By combining AI-powered evaluation, governed workflows, and Snowflake expertise, Hakkoda helped validate that the vendor’s solution met the organization’s standards for accuracy, relevance, and compliance before production deployment. The engagement also resulted in a reusable workflow entirely on Snowflake that the business can carry into its future AI initiatives.
As enterprises continue deploying generative AI into customer-facing and regulated environments, evaluation is becoming a critical capability. Launching an AI solution is one thing; sustaining long-term trust in it is another.
At Hakkoda, we believe successful AI starts with trusted data, responsible governance, and rigorous evaluation. By combining AI-powered delivery with deep expertise in modern platforms like Snowflake, we help organizations accelerate innovation while ensuring every deployment is built on a foundation of complete confidence.
Let’s talk about how we can help you take your AI ambitions and turn them into trusted, production-ready solutions today.
