Writing the Auto QA Playbook & Transforming Customer Support

As the customer support industry evolves into an experience center, agents are no longer simply support representatives. They also serve as brand ambassadors and are critical to customer retention. However, the introduction of AI tools is once again transforming the CX landscape. This blog draws heavily on insights from a recent webinar with Vasu Prathipati, CEO & Co-Founder of MaestroQA. We will explore how an Automated Quality Assurance (Auto QA) program can speed up QA processes to revolutionize customer support, improve key performance indicators such as CSAT and AHT, and ultimately boost your bottom line. By sharing Vasu’s valuable frameworks, techniques, and tips, we aim to equip you with the knowledge needed to navigate this transformation and stay ahead of the competition. 

‍

So, read on to learn more about how to write your own Auto QA to transform your customer support experience and drive sustainable business growth.

What is Auto QA?

First, let’s define auto QA. 

Automated QA (Quality Assurance) for call centers provides automated quality metrics on 100% of your support interactions. 

By automating the QA process, companies can customize and run precise QA metrics to surface actionable insights from all your customer interactions -- voice, chat, text, and screen capture.

Shifting from Manual Call Scoring to Automated Scoring

‍

‍

‍

‍

‍

‍

‍

Today’s CX leaders want their quality assurance teams to execute a vision (see image above) that involves a gradual shift from manual QA to pairing Auto QA with targeted deep dives. “We’re seeing our customers come to us with this vision of how they want quality teams to operate going forward. Quality assurance teams want to do this. They see this as a huge opportunity to unleash their creativity. The reality, however, is that they come into this role of quality with the promise of being an analyst and coming up with insights, but they end up feeling and spending a lot of their time doing more data collection. The word auditor comes to mind.

"The hope is that there's a possibility with new technologies and new companies like MaestroQA that we can help quality teams transform into the promise that they've always set out for themselves, which is becoming a data analyst or a customer experience data analyst in particular.

"So, that’s the shift we’re working on with our customers, and that’s exciting because by allowing Auto QA workflows to handle tasks such as identifying low-scoring DSAT tickets to perform targeted QA or automated transcription to analyze 100% of interactions to get a more accurate picture of what’s causing negative feedback from customers, Quality Assurance teams can increase their strategic value to the organization significantly.” They can now shift their focus on “conducting targeted deep dives to determine where training gaps exist, uncover actionable insights, identify process and policy improvements, and enhance overall quality like never before.” With three simple steps, CX teams can elevate their Quality Assurance strategy to drive business success.

‍

Step 1: Map Out Support Organizational Structure

A customer service organizational structure refers to the hierarchical framework and arrangement of roles within a customer service department or team. It outlines how positions and responsibilities are organized, defining reporting lines and communication channels. 

This structure typically includes roles such as customer service representatives, team leads or supervisors, managers, and may extend to include specialized teams like technical support or escalation departments. The structure aims to streamline operations, facilitate efficient customer interactions and support, and ensure effective coordination and collaboration among team members. It helps distribute tasks, establish accountability, and promote smooth communication to deliver high-quality customer service consistently.

“When CX teams want to transition to this vision of shifting their QA teams to a more data analyst role,” shared Prathipati, “it’s important that they have a clear understanding of their current state. I recommend mapping out your support work structure. A support work structure is where there are not only just Tier 1, Tier 2, and Tier 3, but there are also different lines of business.” For instance, “an Airbnb may have a line of business for renters and another line of business for hosts. But they also may have a line of business for the people who host Airbnb experiences. Each of these lines of business has tiers like Tier 1, Tier 2, and Tier 3.”

‍

‍

‍

. 

‍

‍

Step 2: List Current KPIs Used

‍

This next step is crucial in enabling you to identify and prioritize the metrics like CSAT, AHT (average handle time), or TTR (total time to resolution) that matter most to each line of business and within each tier. “You don’t want to include something about SLAs (service-level agreements) unrelated to agents here, but if it's an SLA related to agent performance, amazing. But if it's first response time, that's a reflection on volume and staffing; you don’t want to include that here,” detailed Prathipati. “We often see some metrics that are not relevant to all the groups, and that's an important thing to note.”

‍

Here’s an example: 

‍

‍

‍

Lyn Miller-Bright, Director of Workforce Management and Quality Assurance at Zendesk, agrees that not all metrics are of equal importance. “While some metrics, like FCR, may be crucial for Tier 1 because they handle a wide range of tasks, they may not be as critical for Tier 2, which deals with a narrower range of responsibilities.” In other words, different tiers prioritize different metrics based on their specific roles and objectives.

“For the sake of this exercise,” said Prathipati,  you want to focus on the metrics impacting your frontline agents (the people speaking to customers) so you can make better decisions about how you manage your teams, your BPOs and how you do root cause analysis on how you coach. This exercise is ideally something you can do if you have a data person on your team or a more data-savvy person, or a metrics-oriented person.”

‍

‍Step 3: Identifying Trusted Metrics vs. Misleading Metrics

‍

In our final step, you want to understand the trust level that your teams have in metrics such as CSAT since customer satisfaction scores are a combination of support, service, product, and/or other market feedback. And like any voluntary-response survey, they’re subject to problems like sampling and response bias. 

‍

‍

‍

‍

Take this all too common, real-world example:

‍

A customer leaves a low CSAT score immediately after a lengthy call with a customer service agent. The reviewing manager looks at that low score, assumes the agent needs additional coaching, and sends them training resources to work on their soft skills.

The truth of the matter is, without additional context, that low CSAT score could have been the result of several things: an ineffective company appeasement policy, an outstanding product issue, or even prior service frustrations. In this case, CSAT didn’t reflect how (well) your agent handled the situation but reflected the customer’s larger company dissatisfaction that the agent would never be able to solve by themselves.

‍

Therefore, CSAT can be misleading.

‍

“We use the labeling term ‘misleading’ as an indication that the team feels that the metric doesn’t accurately reflect reality or that they have no trust in that metric.”

For example, furthered Prathipati, “When we complete this worksheet with our customers, we may discover that the total time to resolution metric may not be applicable to a particular tier. This is due to the fact that when an issue is escalated, it can often sit in the queue for an extended period of time, which can skew the average and make it an inaccurate measure of an agent's performance. Therefore, it may not be effective in coaching agents on what they need to improve.” 

Completing this exercise can bridge the gap between crucial decisions that leadership teams have to make, such as choosing “who to promote, reward, or let go, and the actual ground reality. By exposing the misleading indicators, you can make more informed decisions that can impact teams, culture, and customers positively,” said Prathipati. The ability to make better-informed decisions that align with the real-world situation ultimately leads to better outcomes for your organization.

‍

‍

‍

‍

‍

One-dimensional metrics often lead to misinterpretations, which can have a significant impact on customer experience. “MaestroQA aims to bridge this metrics gap in a new and innovative way that empowers quality teams to be more strategic and influential in their role. With MaestroQA, people-powered quality teams can become more impactful and make a meaningful difference in how customers perceive the quality of their experience.”

‍

The Two-Step Prioritization Technique: Shift the Vision into a Reality

‍

Now that you have identified the challenges that misleading metrics can bring by doing steps one through three and drilled down to identify your problem statement, “it’s important to remember,” said Prathipati, “that you can’t tackle all the challenges that you uncover at once.” He recommended that you follow a two-step prioritization technique:

‍

Prioritize #1:  Pick one team within a single tier to focus on

‍

This team might have the most misaligned metrics, or they represent the biggest areas of customer dissatisfaction and require immediate attention to improve customer experience.

‍

Prioritize #2: Within that team, determine which metrics are truly important

‍

During this step, the worksheet will help you identify your metric alignment process, which metrics should be kept, and which ones can be discarded. According to Prathipati, it's also a great opportunity to discuss your ideal scenario. For instance, your team may aspire to access metrics like customer request escalations, sentiment escalation in conversations, chat-to-email transfer rates, or the number of agents tickets are handed off to. These discussions not only provide insights into what's happening at the ground level but also open up a world of opportunities, such as identifying Auto QA partners like MaestroQA, that fit into this vision.

‍

‍

‍

‍

‍

This shift in vision and the ability to execute that vision has inspired some quality teams to resonate more with team names such as Performance Excellence Team or Quality as a Service Team.

“We’re not saying everyone needs to change their quality assurance team’s name,” said Prathipati, “but we're trying to highlight this real shift in mindset and how Auto QA from Maestro can create value for teams, companies, and customers, plus, help understand agent performance and manage BPOs. With MaestroQA, teams can see which BPOs are hitting performance expectations and which are not. Again, this type of real, actionable data helps make better decisions about people and their livelihood and, ultimately, customer experience.”

‍

Speaking about quality teams…

‍

The Role Of Team Leads And Quality Teams In Change Management

‍

It's crucial to factor in change management. This means taking into account the role of a quality person in the future, as well as the evolving responsibilities of the quality team. 

‍

Here's how a real-world example of how a MaestroQA customer has made adjustments in their change management.

‍

Team Leads use MaestroQA to monitor the performance of agents by analyzing aligned metrics and incorporating them into their coaching sessions. Using the MaestroQA dashboard, Team Leads can extensively review tickets based on these newly identified metrics and take a hands-on approach to coaching, providing personalized one-on-one sessions based on the metrics to ensure their team is meeting their targets.

The quality team, on the other hand, uses MaestroQA to look at things from a slightly higher level, such as how the BPOs are performing. They also look at how multiple teams are performing across a particular line of business. 

MaestroQA allows this team to look at patterns at the BPO level or across multiple BPOs to create ideas for training, new processes, or policies. “They can determine if they want metrics they don’t have today that allow them to monitor and inspect based on what they are seeing in customer interactions.,” said Prathipati. “The QA team shifts to a more targeted role where they can be hyper-focused on an insight level. “That’s why it’s important to flush through this and adjust your change management. Technology helps, but the way teams operate today is different.”

‍

Pick the Right Auto QA Partner 

In the previous sections; we've covered the essential steps needed to identify misaligned metrics, create a clear problem statement, and explore the benefits of redefining the roles of your QA teams. The ultimate objective is to develop an Auto QA playbook and put it into action. Now, the crucial question is, how can you choose a trusted partner to help you bring things to fruition and doesn’t overpromise on AI? 

‍

‍

‍

‍

“This is such a massive shift, so finding a partner who is going to keep it real, who is an expert in the Auto QA space, and can help you execute the vision that you have defined, this is the type of partner you want to seek out,” said Prathipati. 

‍

If you would like to learn more about what MaestroQA can do for your business, please request a demo today.

‍

‍

‍

‍

‍