To YODA say hello, Proximus's internal support chatbot

How can we restore a chatbot to its former glory?​

‍

As one of the first chatbots launched in Belgium back in 2019, YODA needed a retune. His natural language processing (NLP) model was confused, so he either didn’t understand the question to begin with or responded with the wrong answer. This is a more common problem than bot owners think because project owners change, updates aren’t documented, content is forgotten about, duplicate intents are created, etc.

‍

Fortunately, YODA’s structural bones were great! The content hierarchy, basic flows and breadth of intents were on point. All the confusion centered in the NLP model, which is basically his brain. Since YODA is a massive chatbot, we quickly realized this wasn’t going to be an optimization project that could fit in a two-week sprint. Together with the Proximus team, we broke the project up into four phases:

1. Content
2. NLP model
3. New features
4. Documentation

Using our NLP expertise and bot-building best practices, we worked on fixing the broken to restore YODA to his wise self.

Setting up a multi-phased chatbot optimization project​

‍

Content: Updating YODA’s responses and knowledge to match the correct information​

Starting with the content gave us the opportunity to get to know YODA and familiarize ourselves with the different topics, intents and flows. Together with the HR Team, we reviewed all the existing bot copy and highlighted what needed to be removed, added or updated. For example, Proximus’s mobility policy changed, so we needed to adapt the existing bot responses to match the new policy. There were also new employee benefits such as ecocheques, which meant that we needed to create new intents for those new benefits.

These types of content updates are relatively easy to make, but it’s something that needs to be done regularly as information and policies change. Otherwise, the content becomes messy and outdated, and the changes pile up in a gigantic backlog.

NLP model: Cleaning up YODA’s confusion ​

Cleaning up the NLP model is the most intensive phase of any chatbot optimization project. We began with a master list of all the intents currently in YODA. Then we went through each intent one by one to review the name and expressions. We implemented naming conventions to make it easier to recognize and find intents. We then checked the expressions to make sure that they matched the intent, removed any that didn’t match, and added expressions to make sure there were +40 per intent. Some intents were very similar to another, so we merged them to eliminate any confusion, and deleted any duplicate intents. We also review existing conversations to add real user expressions to the matching intent or make a note of an intent that’s not yet covered by YODA’s knowledge so that we could add it to the backlog and include it in a future release. Then it was testing, testing and more testing.

Offloading to a live chat is a common practice that we use in customer support implementations, but quite rare in HR implementations. It’s normal that a bot can’t answer everything, so then we need to design the experiences when a bot can’t answer to be as frictionless as possible for the user. Based on our experience with customer support chatbots, we proposed introducing a live chat handover. When YODA can’t answer a question, he proposes to send the user to one of his human colleagues for further assistance. If the user agrees, YODA alerts one of his HR colleagues and shares the conversation history, so they can easily pick up the conversation where YODA left off. The user keeps the chat window open and in a few minutes, a human colleague joins that same chat to help solve their problem. YODA’s main objective is to automate everything that can be automated so that his human colleagues can focus on these more complex conversations. By integrating with Interact, a live agent handover solution, we make it possible for human colleagues to jump in and support YODA when he needs help solving an employee's request.

Documentation: Making sure it’s clear how YODA works, so this problem doesn’t happen again​

Documentation is crucial for a successful bot in the long run. We like to create a Miro board to have a visualization of the flows, copy and technical logic in one place. We also generate an NLP gatekeeper document so that we have a searchable overview of all the intents, expressions and bot responses linked.

We use the Miro as our single source of truth because it’s easy to zoom in on certain flows and visualize how one flow connects to another. The Miro board also allows for collaboration with non-technical project stakeholders. Its easy-to-use interface means that anyone can comment, add notes, propose changes, etc.

‍

An NLP gatekeeper document links the intents, expressions and bot responses in an easy-to-search Excel file. In bot platforms, this information is usually fragmented in different windows, so it’s difficult to have an overview of all the questions and answers.

‍

Save time, YODA does​

Since YODA’s relaunch in 2022, the annual volume of HR tickets went from +30,000 to ~14,000. He handles +6,600 conversations a month and only offloads 1.17% of those conversations to human colleague. On average he saves the HR team ~41 days of work per month!

In an internal survey, 50% of Proximus’s employees indicated that they prefer talking to YODA than creating a support ticket or calling the HR helpline. The HR Team expects this percentage to increase as existing employees become more familiar with YODA and younger employees join the company.

‍