Signpost AI Chat Pilot: A Roadmap
Welcome back, AI enthusiasts! So far we have talked about our Signpost Chat, its development, its quality, red-team testing and other aspects. In today’s post, we are going to take an overview of the Signpost Chat pilot and how we have structured it.
Signpost AI is conducting a 6 month pilot in four countries: Greece, Italy, Kenya and El Salvador. The choice of these countries was simple. The Protection Officers (POs) who conducted early quality testing of the AI agents in development were based out of these countries. Given their experience with testing and evaluations, they are serving as country pilot leads.
The objectives of this pilot are:
To ensure that the chatbot functionality aligns with specific program needs
Optimizing chatbot performance through iterative testing and refinement
Evaluate chatbot performance and identify areas for improvement
Create a scalable and sustainable strategy for future implementation
The timeline for the pilot is as follows:
Phase 1: Deployment and Rapid Iteration: Launching Signpost AI chatbot in all pilot countries and begin the process of gathering feedback for rapid iteration. We are currently in this phase with moderators in all countries beginning to use the AI agent. The launches were staggered because each country needed time to coordinate their logistics and staffing. This phase is planned to last one month.
Phase 2: Testing and Refinement: The goal of this phase, which will last 2 months, is to conduct in-depth testing, collecting feedback and making adjustments through rapid iteration. This means testing different features and functionalities while and gathering use feedback on AI AI agent accuracy, quality of response and ease of use
Phase 3: Impact Assessment and Feature Enhancement: This phase is where the team will thoroughly evaluate the chatbot’s performance and impact. Based on the results of the evaluation, we will then implement enhancements. This phase is meant to last two months
Phase 4: Scaling and Sustainability Planning: Based on the evaluations, and learnings from the previous phases, in this one month phase, Signpost will develop a plan for scaling and long-term sustainability. Feasibility checks will be conducted to assess how the chatbot will be deployed in additional contexts as well as surveying potential impact of external factors.
We have also done associated scenario planning and corresponding mitigation strategies related to technical, moderator, programmatic and external challenges.
The pilot test is ongoing. As the moderators test and evaluate the Signpost AI chatbot and the team iterates on this feedback, we will continue to publish how we are coming along. We are excited to invite you on our journey as we develop and proof-test AI for the humanitarian context.
