Training AI for Trust: Insights from Cuéntanos El Salvador's Chatbot Journey
In July 2024, I joined the Signpost team as a Protection Officer (PO) to explore how a chatbot behaves and how to train it to provide trustworthy responses in a humanitarian context. My initial focus was testing the quality of responses generated by different bot models (Claude, GPT, and Gemini) to determine which offered the most precise and reliable answers.
After evaluating these models, GPT emerged as the most promising. It consistently outperformed the others based on our three main quality metrics: trauma-informed language, client-centered information, and safety/do-no-harm. We launched an AI pilot program with Cuéntanos El Salvador—a platform providing information and psychosocial support to individuals experiencing emotional crises.
Pilot Program with Cuéntanos El Salvador
Following the model selection, we launched a pilot program with Cuéntanos El Salvador, a platform offering vital information and psychosocial support to individuals in emotional crisis. Training sessions for the Cuéntanos team began on October 7th, covering AI fundamentals, roles and responsibilities, and the integration of the chatbot into their existing Zendesk platform.
Addressing Initial Concerns
During the first training session, several key questions and concerns arose:
Safety: Was it safe to deploy a chatbot with the vulnerable population Cuéntanos serves?
Job Security: Would AI eventually replace human staff?
Localization: Could the chatbot effectively adapt to local language and cultural nuances?
These were valid questions, as AI is often misunderstood to operate autonomously. These concerns, rooted in common misconceptions about AI autonomy, were addressed through practical experience during the pilot. Key observations included:
Human Oversight is Essential: All AI-generated responses required regular review by human moderators.
Training Improves Performance: Targeted prompting and iterative training significantly enhanced the chatbot's accuracy and appropriateness.
Feedback is Crucial: Continuous feedback from moderators proved invaluable in refining the chatbot's performance.
Progress and Achievements
The collaborative approach with the Cuéntanos team has been instrumental in tailoring the chatbot to their specific needs. Key achievements include:
Accurate Responses to Simple Queries: The chatbot effectively handles routine inquiries.
Understanding and Responding to Local Language: The bot has learned to understand and respond appropriately to Salvadoran slang.
Supporting Moderators: The chatbot frees up moderators to focus on more complex cases, including the provision of Psychological First Aid (PFA).
A notable success was the chatbot's automated response system for a gender-based violence prevention campaign on Facebook, demonstrating its capacity to handle sensitive topics with increasing proficiency.
Challenges and Lessons Learned
Despite the positive progress, the chatbot faced challenges emphasizing the need for ongoing human involvement:
Complex Queries: The chatbot sometimes struggled with lengthy or complex questions.
Tone and Language: Initial responses occasionally used overly directive language (e.g., "we recommend you"), necessitating adjustments for a more empathetic and supportive tone.
Hallucinations: Instances of inaccurate or fabricated information ("hallucinations") underscored the critical need for continuous monitoring and refinement.
These challenges reinforced the fundamental principle that AI is a tool requiring constant human guidance, particularly in the sensitive context of humanitarian work.
A Collaborative Approach
The Cuéntanos El Salvador pilot program demonstrates the potential of AI to augment humanitarian efforts. It also definitively highlights the indispensable role of human oversight and collaboration. By working together, we have successfully trained the chatbot to provide meaningful support, adapt to the local context, and alleviate moderator workload, enabling them to focus on more critical interventions.
The future of AI in humanitarian work depends on this balanced partnership between technology and human expertise. By embracing this collaborative approach, we can build safer, more responsive systems that effectively serve those in need.
