Signpost AI Chat Pilot: Using the AI Tool

Hello everyone! Welcome to another blog post on our Signpost AI chatbot pilot series. Today we are going to take a look at our chat AI tool and how moderators in our pilot countries (Greece, Italy, Kenya and El Salvador) are using and testing it.

The ZD AI tool is the name of our chatbot application tool which has been integrated into our existing Customer Service platform, Zendesk. This uniformity of platform allows moderators to familiarize themselves with the new tool more quickly.

Let us take a quick close at the tool from the perspective of a moderator:

Here is a cropped screenshot of the interface. The tool options on the right side have a few options:

• Include Sources: this instructs the bot to add references for any response that are generated

• Include Comments: from previous tickets: this option is applicable for any returning user

• Bot ID: if moderator has to test another bot, this is where they will add its ID 

• Extra User Prompt: This allows the moderator to include additional instructions to the user question

• Generate Tags: this button automatically fills out relevant form tags that accompany each user ticket. In case the moderator uses this button, they will also have to review this

Once the moderator has included information, they can press the “Generate AI Response” which will prompt the chatbot to generate an answer in the Internal Note column. This is located right below the user’s question window.

The answer will appear in the Internal Note column where the moderator reviews the answer and edit where required. In this case, the user's question is asking how to get a driver’s license as an Eramus student in Greece. 

The moderator will review this response, checking the formatting (numbering, etc.) based on pre-established guidelines. They will check for tone, accessible language, accuracy and relevance in the content. For example, in many cases the moderator will have to edit use of the first-person I  in the responses to We, to better reflect the standard of response used at Signpost. The moderator will also test the references, one by one, making sure they are correct, relevant and point to active websites. 

After the moderator is satisfied with the final response, they can use the tool to respond directly to the messaging channel where the question came from. 

Once this is done, the moderator will have to leave feedback:

Good: refers to one where the response did not require any edits; it was a human moderator-level response

• Bad: refers to a response that requires edits or changes (e.g. adding additional information, removing irrelevant information)

• Flag: refers to malicious, inaccurate, irrelevant, hallucinated or false responses and references. Essentially if any of the responses contravene the Red Team or Quality framework.

After this, the moderator has to evaluate the response on the Quality metrics:

Moderators are instructed to use the Comments for every response detailing why a response has been marked in the way that it has. It allows the team to gain a qualitative and in-depth understanding of how well/poorly the chatbot is performing. This reviewing and evaluative process is similar to the one we have documented previously.

After the moderator submits their evaluation, the following information is recorded on our content management platform, allowing pilot leads and program managers to review data on a broader scale:

• Question

• Answer

• Reporter

• Score

• ID

• Expected

Finally, the moderator can also request an article if they identify a knowledge gap for which articles do not exist.

The AI tool can seem a bit much at first glance but as moderators are reporting, it becomes easier and more intuitive to use over time. The key is to have (a) continuity in use of technology platforms so users only have to learn one new technology application (b) having an intuitive design interface and (c) making sure the users have plenty of training and guidance at the beginning.

e will be sharing their moderator feedback on the chatbot in more depth in a future research piece so be on a look out!