Making Conversational AI Journey successful with Bot Metrics Store!

By: Satesh Kumar

Many founders overlook the importance of merchandising as well as couponing in their retail outlets.  At $12-15 per store visit, the cost of hiring a merchandising firm can quickly add up, especially since most merchandising firms have monthly minimums. 

In addition, working with a traditional clearing house in the couponing space can require $10,000-$20,000 in capital.  How does a small brand with a small marketing spend drive sales in store? 

As a founder, I encourage you to personally visit as many of your stores as possible.  I think it’s important to get a feel for your stores – ie. if your product is located in the right location in the store, educate staff on your products, and offer them samples, and of course drop off coupons next to your product on the shelf. 

Did you know that shoppers often ask staff to learn about new products?  Plus, the staff love meeting the person or team behind the brand and hearing your story. 

When you’re not on promotion, coupons on shelf help drive attention to your products as well as encourage trial.* 

Instead of using a traditional clearinghouse, we partnered with Perinovus, to purchase the coupon bar codes and then we instructed the retailers to mail the coupons to us for reimbursement. We also ask stores to mail the physical used coupons to prevent fraud. 

Essentially, we are acting as our own clearinghouse.  Once we receive the coupons and the invoices in the mail, we cut the retailer a check.  To streamline this process, we do this once a month.  While we realize this may not work if we were in 5,000+ stores, this works for us for now.

Making Conversational AI Journey successful with Bot Metrics Store!Satesh Kumar

Introducing Bot Metrics Store

A Bot Metrics Store provides a guidance on the key business and technical (aka subject areas) that are to be focused and an illustrative collection of KPIs. This store provides the business and data analysts with valuable insights in customer experience and operational efficiencies powered Conversational AI programs.

Making Conversational AI Journey successful with Bot Metrics Store!

It is to be noted that all the subject areas and metrics discussed henceforth are applicable to both Concierge/Master as well as Child bots depending on the functionalities implemented.

Business Insights

Bots are key digital agents who are in most cases the first level contact point of customers, hence it is important to measure the pulse of the customer interacting with them and the efficiency with which the bots operate.

Customer Experience

Engagement and Retention: This analytics area helps to understand the characteristics of the conversations and stickiness of the bot. Provides insights into the following areas: User adoption and engagement, Session (chat, voice) trends across different time periods, channels, comparison of different session metrics and the impact it has on customer satisfaction.

Another notable aspect of this analytics area is that the Conversational Lineage analysis that is equivalent to customer behavioral analysis in the Conversational AI world.

It provides information on the conversational path traversed by the customers enabling business to understand the most/least used conversational paths and customer journeys. It leads to understanding paths with high-dropout rate and planning the best course of response action.

Feedback Analysis: This analytics area provides insights into customer satisfaction levels during the session. Additionally with deep dive possibility based on the type of interaction – Bot Only, Bot + Agent

Some stats: “Nearly 40% of internet users worldwide prefer interacting with chatbots than virtual agents, and with major industries including retail and healthcare turning to digital technology” – Business Insider[2]

Business Value

Cost to Serve: This financial metric helps understand the operational cost and efficiency drivers. They are measured by understanding how bot handles interactions independently and the agent re-direct scenarios also known as safety-net.

Some of the key questions addressed by the analytics area are: What were the intents that resulted in a re-direct scenario? What are the correct (expected) and wrong (unexpected) re-directs? And associated operational cost factors.

Some stats: Even if a person makes a “call me” command to connect with a live agent after initiating a chatbot interaction, a contact center pays about 33 percent less for outbound calls than inbound calls"[3]

Automation Efficiency: Truly, enterprise bots go beyond merely answering standard FAQ queries by enabling users to execute conversational commerce, business process (e.g. ordering, cancellation, profile changes) from within the application console (chat window or voice medium) thereby making them “One Stop” application for all customer queries, commerce and support.

This analytics area provides insights on bot based enablement of business processes and their efficiency. For example some of the metrics are automation rate, failure reasons.

Some stats: “Between 2019 and 2020, preference for using a Conversational Marketing solution to schedule a meeting, purchase an item, or add oneself to a mailing list more than doubled.” [4]

Technical Insights

Continuous assessment, improvement are key to engineer the bot towards becoming truly conversational and is a journey that requires rigorous monitoring, measuring and re-engineering.  


Effectiveness from a bot perspective focuses on making the conversations meaningful. This can be measured by success of intent understanding from customer messages, initiation of right action (e.g. process) or response from Knowledge Base (KB) or Intent database.

This analytics area provides insights on Intent Vs. Customer message mapping, Non-Intents analysis, Q&A KB response mapping. This is a key focus area for Bot/Language trainers.

“A report that clearly indicates the user query, bot’s intent recognition capability and associated confidence score provides valuable insights to IT team for re-engineering and re-training” 

Usage Metrics

The number of intents or KBs in an enterprise bot may vary from few tens (for a use case) to few hundreds. Hence it is important for the IT teams to get insights into the usage patterns of the Natural Language Understanding (NLU) and KB engines since significant amount of time is spent in maintaining these components.

This analytics area provides insights on the top - Intent categories, KBs accessed, FAQs most/least used and corresponding reasons.

Your bots have a story to tell – Insights to Action!

The true value of any analytics solution goes beyond reporting metrics and relies on the insights it provides for the stakeholders to take business action.

Making Conversational AI Journey successful with Bot Metrics Store!

Closing Comments

Bot Metrics Store enables organizations to improve customer experience, business process refinement and drive operational excellence. Critical success factors are:

Make analytics your Day 1 strategy: Bot Metrics Store must be an inclusive part of the bot building initiative to ensure that all conversations are captured holistically and must not be seen as a siloed or a “Post bot deployment” implementation. Bot Foundation and Bot Metrics Store implementation must go hand in hand

Look at the “Whole 9 Yards”: The analytics foundation must look beyond capturing basic metrics for operations purpose (statistics) and focus on metrics that will create a difference to business, customers and also act as a feedback to engineer the bot itself.

A good starting point is a shift in the mindset to identify metrics that will enable to “discover possibilities” rather than merely “measuring performance”

Voice of Customer: Bots have the unique advantage of capturing customer emotions, interactions at run/real time leading to numerous possibilities in understanding the customer personality, behavioral patterns better. Analytics on conversational data will be the "Voice of Customer"

Pivot to future of Personalization: Progressively move the Bot Metrics Store from being descriptive to prescriptive by leveraging advanced analytics techniques. For Instance, Hyper Personalization through implementation of Next-Best-Action (NBA) enabled bots will lead to personalized conversations.  

Special Edition