At Master of Code, we developed a Chatbot Analysis Framework for businesses to review and evaluate their conversational AI solutions in terms of the user experience and the solution’s maturity and complexity. It reviews user-facing elements that are critical to the overall customer experience. With this framework, analyzers were able to identify the solution’s strengths, any gaps in the user experience, and opportunities for new features.
To start, we picked an exciting sector to dive into: Financial Services. These banking chatbots are high volume and serve the needs of customers seeking finance-related support and services which makes way for the opportunity to develop or enhance contextual and transactional experiences. Given how personalized the banking web experiences are today, we can now translate them into a more conversational one.
Table of Contents
Evaluating Canadian Banking Chatbots Based on Top Success Metrics
We reviewed 6 banks using our Chatbot Analysis Framework and scored their Conversational AI Solution. Each line item in the table was rated during analysis by giving:
- A full point if the chatbot met the feature’s best practice standards.
- A partial point if the solution displayed only part of the described feature.
- A zero if none of the features were currently present in the solution at all.
These scores were then added up out of a total score of 20 and reviewed against the chatbot evaluation metrics table below.
Let’s take a deeper look at the results of each of the 6 banks.
TD Bank Chatbot
First, we looked at TD Bank’s chatbot, Clari, which exists on the TD app only and can answer basic FAQs and account questions from customers.
TD’s Banking Chatbot Strengths
- Provides bot prescription
- Utilizes structured content capabilities
- Meets accessibility standards
TD Bank’s Clari does well to identify itself as a Conversational AI assistant and has an option to educate users on what it can do as a bot, which helps set expectations for the end-users up front.
Clari utilizes the app channel capabilities, such as structured content, buttons, and has app-specific UI blocks where users can tap the call-to-action button to bring up a pop-up window with more information and a link to a web landing page. Another best practice Clari demonstrates is that its UI meets the standards of the Website Content Accessibility Guidelines (WCAG).
TD’s Banking Chatbot Opportunities
- Offer both unauthenticated and authenticated experiences
- Implement transactional use cases
- Provide more personalized responses
- Live agent handover
- Shortening long tailed actions on web into faster conversational flows
Clari provides simple answers to common financial support questions but is connected to the customer’s account, as you have to be logged into the app to talk to the chatbot. There’s opportunity here to include transactional use cases and make the conversational experience more contextual by pulling that data into the banking chatbot. Clari also has a few intents that leverages customer data to add personalization (i.e. recent transaction questions), but why stop there? As an example of a personalized customer experience, adding customers’ names to the dialogue would not only make the experience more personal but strengthen trust as well.
Additionally, having the option of live agent handover would greatly benefit the user experience. Currently, Clari only offers customers the option to call if they need live support assistance. Introducing the ability to transfer to a customer support agent within the bot will boost customer CSAT, and reduce the friction of having to switch channels.
One of the main use cases Clari has is viewing and breaking down recent transactions. While this is something banking customers like to do often, this can be easily accessed on the dashboard screen when they open the app. Therefore, it isn’t clear what pain point this use case is solving as it’s just as, if not more efficient on the web & app user flows. Reviewing customer feedback and utterances would be helpful in figuring out how to improve this experience even further.
Our Framework Results: Minimal use of automation.
Start a discussion on improvements
RBC Bank Chatbot
Next, we reviewed RBC Bank’s chatbot, Ask NOMI. It’s only available on the RBC mobile app and can help support frequently asked questions, including inquiries about a recent transaction, canceling an e-transfer to an individual, or helping customers navigate the banking app.
RBC’s Banking Chatbot Strengths
- Setting customer expectations
- Consistent bot persona
- Leverages app capabilities
- Meets accessibility standards
Similar to TD Bank’s chatbot Clari, Ask NOMI helps set customers expectations by introducing itself and providing menu options once the chat is initiated. We found that Ask NOMI also has a trustworthy and personable tone of voice that’s consistent throughout the bot, which is beneficial for customer engagement and retention.
Ask NOMI does well to leverage app capabilities such as quick replies, structured content, and rich links and images (i.e. to show recent transactions, etc). It responds to both text and voice queries and handles free form natural language understanding (NLU) in 12+ languages. It provides concise content that is easily digestible, reducing any heavy cognitive load for customers and meets the WCAG standards.
RBC’s Banking Chatbot Opportunities
- Pull existing tools into the chatbot
- Turn wayfinding use cases into transactional ones
- Live agent handoff
- Include global commands for users
While Ask NOMI utilizes customer data to provide insights such as recent transactions, there is an opportunity for it to turn its wayfinding use cases into transactional ones allowing users to complete more tasks in a shorter amount of time. Ask NOMI is part of a larger feature, simply called NOMI, that includes other tools called NOMI Insights, NOMI Find & Save, and NOMI Budgets. These tools could be pulled into the bot where customers can access everything in one place while still engaging in a conversational experience.
Ask NOMI seems to have no live agent handoff via chat. Instead it provides a link to a page for escalation via the Advice Centre which is only accessible by phone. Like TD Bank’s Clari, we recommend implementing live agent escalation for banking chatbots because it will reduce friction for customers who need additional help and open another channel for support. Customers who are not able to talk or use the phone due to a disability or other reasons are currently blocked from this experience.
Lastly, it was observed that Ask NOMI’s NLU wasn’t able to comprehend phrases like “main menu” or “go back to main menu”. For customers who want to see what else the bot can help with or simply exit a flow, this global command is very important to have in chatbots according to best practice standards.
Our Framework Results: Good use of automation.
Start a discussion on improvements
Scotiabank Chatbot
The Scotiabank Chatbot is an FAQ-type virtual assistant available within their mobile app experience that answers questions about everyday banking and connects you with a customer care advisor if you want to chat with a person. It also leverages the app for transactional capabilities (i.e directing the customer to go to a section within the app to make a bill payment).
Scotiabank’s Banking Chatbot Strengths
- Excellent welcome message with personalization & prescription
- Leverages app structured content capabilities
- Content is easily digestible
- Includes live agent handoff
The conversational experience starts with a short introduction, where the bot introduces itself as a ‘chatbot’, presents multiple options on what the bot can handle, and refers to the customer by name in the introduction. This is an example of a good welcome message for a banking chatbot. It’s short and concise, inviting the customer by providing personalization and setting expectations. Additionally, the bot continues to keep a friendly tone throughout the experience.
This banking chatbot does well to utilize rich links, structured carousel menus, and buttons to help guide the customer on what they can do next. The content in the bot is easily digestible as well, keeping the cognitive load to a minimum.
The Scotiabank Chatbot appears to have live agent handoff, but while reviewing only one instance was found where it offered to “chat with a person”. Additionally, it informs the customers of other live support options after hours.
Scotiabank’s Banking Chatbot Opportunities
- Utilize APIs to have contextual & transactional experiences inside the bot
- Review and update training data & content
- Improve UI to meet accessibility standards
Apart from the personalization in the welcome message, we could not find other areas in the bot that give indications of APIs or other integrations. As mentioned above, the chatbot leverages the app for transactional capabilities however it removes the customer from the experience and relocates them to a page in the app. There’s a big opportunity here to enhance the flows by using customer data to create more contextual and transactional conversations (within the bot) as the customer is already authenticated.
Furthermore, we identified that the chatbot seemed to not recognize common financial terms like “mortgage payment” and “interest rates”. It would be worth reviewing such keywords to understand the utterance volume to see if new intents or use cases need to be implemented. If they’re low volume, updating the dialog to help set expectations that such content isn’t available in the bot or offering alternative options would be a recommended experience.
In regards to accessibility standards, there’s an opportunity to increase the font size and improve the contrast of the buttons to make them more legible for those with visual impairments.
Our Framework Results: Good use of automation.
Start a discussion on improvements
Banking Chatbots Evaluation Based on Top Success Metrics Conclusion
After putting these banking chatbots to the test with our Chatbot Analysis Framework, we found that they shared some strengths such as meeting accessibility standards, leveraging the appropriate channel capabilities, and having a consistent bot persona or tone of voice. However, they had differing results for opportunities which included leveraging authentication, adding more transactional use cases, implementing live agent handoff, and providing better prescription. If you’d also like to put your chatbot to the test, run these chatbot evaluation metrics and learn how you can optimize your virtual assistant to demonstrate a strong use of automation, download our Bot Analysis Framework or get in touch with us.
Stay tuned for part 2 where we’ll review another three banking virtual assistant using our Chatbot Analysis Framework with most important chatbot evaluation metrics, share more insight about the top Canadian banking chatbots, and discuss valuable use cases for financial services bots.