Betterment Raises Chatbot BSAT, Containment, and Automated Resolution Rate with QA

Chatbots often serve as the first point of interaction between brands and customers. When their performance is subpar, it can negatively impact customer satisfaction. Natalie Langdale, Chatbot Strategy Manager at Betterment, understands the importance of getting chatbot performance right. In this article, we share Betterment's approach to Chatbot QA and the techniques used to boost chatbot accuracy, efficiency, and overall experience through MaestroQA.

Challenge: High Volume, Compliance, and Limited Visibility in Reporting

Betterment’s rapid growth presented several challenges when it came to maintaining chatbot performance and customer satisfaction:

• Managing high ticket volumes: With a growing customer base, the number of inquiries increased significantly. Betterment needed their chatbot to handle basic queries while allowing human agents to focus on complex issues.
• Ensuring compliance in a regulated industry: As a financial services company, Betterment had to routinely review and check that their chatbot didn’t provide inaccurate or misleading advice—a serious concern in such a regulated industry.
• Limited visibility in reporting: While Betterment’s chatbot platform, Ada, provided high-level metrics such as containment rate and automated resolution rate, the data was not granular enough to identify the root causes of issues or knowledge gaps in chatbot responses.
• High-level metrics that don’t show the full picture: The chatbot tool’s basic stats failed to give a complete understanding of where the bot was falling short, making it difficult to refine performance and help maintain consistent, accurate responses.

We established a really rigorous QA process for our bots. A key part of getting approval from our legal, risk, and compliance teams was proving that we had a systematic way to review and improve our generative bot’s outputs, which was entirely enabled via our Maestro integration."

Natalie Langdale

Chatbot Strategy Manager

Solution: A QA Program That Delivers Granular Insights

Betterment implemented a robust Chatbot QA program with MaestroQA, allowing them to address both the performance and compliance challenges while gaining deeper insights into chatbot interactions.

1. Building a Comprehensive QA Rubric Betterment developed a tailored QA rubric to evaluate their chatbot on four critical areas:‍

• Accuracy: Ensuring that the chatbot understands the user’s inquiry and responds with the correct information.
• Completeness: Verifying that every inquiry is fully addressed.
• Compliance: Ensuring that the bot adheres to strict financial regulations and does not provide misleading or incorrect advice.
• Clarity: Maintaining a conversational tone that aligns with Betterment’s brand while being clear and concise.

2. Using MaestroQA to Dig Deeper: While Ada provided general metrics, MaestroQA allowed Betterment to go beyond surface-level data. The platform enabled detailed analysis of chatbot conversations, helping identify specific gaps in knowledge and opportunities for improvement. By reviewing individual interactions, the team was able to improve chatbot responses in real time.
3. Regulatory Compliance with RAG (Retrieval-Augmented Generation): To avoid compliance risks, Betterment implemented RAG technology. This model supports the bot’s ability to cite verified sources, reducing the likelihood of generating incorrect or inappropriate responses. The integration of MaestroQA further strengthened this by providing detailed feedback on the bot’s performance.

Results: Greater Performance and Compliance, Enhanced Visibility

With a focus on both chatbot performance and compliance, Betterment achieved substantial improvements:

• Improved Visibility into Performance: The combination of MaestroQA and Ada allowed Betterment to move beyond high-level metrics, providing detailed insights that enabled continuous improvement of chatbot interactions.‍
• Better Compliance Management: RAG technology and regular checks helped maintain the chatbot’s adherence to legal standards, reducing compliance risks and increasing trust in chatbot responses.‍
• Accurate and Relevant Responses: Detailed analysis of conversations through MaestroQA allowed the team to fine-tune responses and close knowledge gaps, significantly improving the quality of chatbot interactions.

Since implementing Chatbot QA, Betterment has seen the following improvements to their chatbot performance:

• BSAT: +5%
• Containment Rate: +2%
• Automated Resolution Rate: +2%

Impact: Holistic Chatbot QA for Consistent Improvement

Betterment’s Chatbot QA program supports their bots in meeting rigorous standards of accuracy, efficiency, and compliance. By integrating both bot and human CSAT data, Betterment can monitor the entire customer journey and help maintain a seamless, high-quality experience across all touchpoints.

As Betterment continues to grow, their focus on granular insights and continuous QA ensures that their chatbots evolve in line with customer expectations, providing reliable and compliant support at scale.

Ready to refine your chatbot performance? Natalie Langdale shares detailed insights on how Betterment uses MaestroQA to boost chatbot accuracy, efficiency, and compliance. Watch the full webinar and discover how to take your chatbot strategy to the next level.