Navigating Generative AI at Signpost: Risks, Mitigations, Benefits and Trade-offs
Introduction
OpenAI’s release of ChatGPT in November 2022 and its subsequent public uptake marks the beginning of the era of Generative AI. The breakthroughs and capabilities of Generative AI seem sudden, but they are underpinned by incremental innovations over the past two decades.
While there is no consensus on definitions, Artificial Intelligence (AI) can be understood as the broader field of computer science that seeks to simulate human intelligence in machines and perform human tasks. Such tasks include learning, reasoning, prediction, problem-solving, perception and the understanding of language [1]
Generative AI refers to AI systems and models which rely on machine and deep learning to produce novel-but-similar textual, audio or visual outputs based on the data that they have been trained on.[2]
Not only is the adoption of this technology fast moving [3], it has also set off an AI arms race between the largest technology companies in the world, hurrying adoption. McKinsey forecasts that Generative AI could add trillions of dollars to the global economy [4] with its tools enabling increased productivity, automation, and reduced workloads.[5] On the other hand, there are warnings that the technology has shortcomings and risks, namely: opacity [6], stereotyping [7], bias, misinformation [8] and infringement of intellectual property [9] [10]
Our Ethos at Signpost with respect to technology and AI in general is balanced, practical minded and solutions-oriented: we are not positivists but we are ambitious optimists. We are realists but we are not cynics. This perspective means, we do not buy into the extreme hype or doomerism of new technologies like Generative AI; instead we prefer exploratory, praxis-grounded approaches to see how best we can use them as tools to further our humanitarian mandate.
Generative AI in the Humanitarian Context
The global context for Generative AI should be acknowledged by humanitarians deliberating and utilizing this new technology to reach under- and unserved communities. This is particularly pertinent given the rise in humanitarian need and the lack of funding required to meet it.
There are risks to Generative AI which require mitigation actions. Generative AI also offers cost-effective solutions and scaling opportunities to meet increasing humanitarian information needs in areas of crises. With these benefits, however, come crucial trade-offs which must be thought about, contended with, and whose effects must be alleviated. At Signpost AI, we are taking an experimental, learning-based approach to explore such benefits and attendant trade-offs by developing a Generative AI agent technology (also can be referred to as chatbots). By utilizing principles of being Transparent, Evidence-based, Collaborative, Ethical and Responsible in our development, we aim to be humble, and deeply cognizant of our responsibility to our communities, our staff and our peers in the humanitarian sector.
Structure of the Paper
This short paper is one part of our overall approach in documenting Signpost’s practical efforts, processes and decision-making in the development of a Generative AI agent technology for personalized information provision in a humanitarian use-case. This documentation process is part of our approach in making everything we do in developing the AI agent technology open, transparent and available to everyone. We believe that the most effective way to address the use of Generative AI in the humanitarian sector is through sector-wide willingness to learn, collaboration, and openness on the matter.
This paper uses Signpost AI’s development and implementation efforts of a Generative AI agent technology as a case-study to look at the risks, problems, benefits and trade-offs associated with GenAI, specifically in a humanitarian context.
The paper has two main parts. In the first part, we set out the current issues, risks and problems of Generative AI implementations in the humanitarian and Signpost context. We then look at specific mitigation actions that Signpost AI is undertaking to mitigate them. This is not to say that such mitigations are complete fixes or fully inoculate us from risk; the mitigations are meant to be first practical steps towards learning what we do now, what works and what we can potentially do in the future in collaboration with other partners.
In the second part, we detail the benefits of implementing Generative AI agent technology for information provision in Signpost’s humanitarian use-case. We acknowledge that such benefits necessarily come with trade-offs and we list them even ones that might be unfixable.
Key Assumptions
These two parts are based upon a key set of assumptions. That (a) our listing of issues and risks of Generative AI is complete (b) Signpost efforts mitigate the impact of each of these risks to zero or to close to zero to a point where humanitarian principles are foregrounded, Signpost service quality is not compromised, users are kept safe and are not harmed, and compliance to legal and data protections are upheld. and (c) assuming that such risks have been mitigated, what are associated benefits and trade-offs of implementing a Generative AI agent technology in Signpost use-case.
There are instances where certain problems are unfixable and will require band-aids. There are also trade-offs which are unavoidable. These truths highlight the necessity of choice and the importance of value-driven decision making in non-ideal situations. They also point to a greater need for humanitarian sector dialogue, cooperation and collaboration; to think, research and work out together the best ways to de-risk and utilize Generative AI as a tool to meet the needs of our communities.
The Moral Question
The calculus of risks, mitigations, benefits and trade-offs is necessary given the moral imperatives that are raised by Generative AI; do humanitarians have a moral duty to use AI to reduce human suffering? Given the power of AI to efficiently scale and reduce human suffering, it is the view of some that humanitarians should adopt this technology; others voice deep concern and reservations about its tendency to centralize resources and power into select private companies and its negative effects on the core humanitarian mission.
Our view is that the key to addressing the moral obligations of humanitarians in relation to the use of generative AI systems lies in the practical experiment of de-risking and deeply embedding values-driven ethical principles in the development of Generative AI agent technology. It is only after the accounting of risks, mitigations, benefits and trade-offs, and the lessons of operationalizing humanitarian values and ethical principles in active technological development, can we begin to see (a) whether Generative AI can effectively and safely work in a humanitarian context such as Signpost’s, and (b) based on lessons learned from this practical endeavor, how can the sector learn to use this technology given its benefits/trade-offs/risks potential space. These practical questions are essential precursors to formulating an answer to whether humanitarians have a moral obligation to utilize this technology.
Risks, Problems and Mitigations
Accuracy and Reliability
Problems:
Generative AI systems such as Large Language Models (LLMs) regularly output plausible yet factually incorrect information and state these outputs with high confidence. This tendency of Generative AI is referred to as “hallucinations” [11][12]. It is simultaneously an open problem as well as an inescapable characteristic, given how LLMs are trained and created.[13] Such hallucinations give rise to inaccuracies, lack of reliability and misinformation. Additionally AI/ML models can degrade in performance (referred to as drift) over time due to changes in parameters, weights or future training runs.
In a humanitarian context, even minor inaccuracies in information can have catastrophic outcomes for clients and users.
In the specific case of Signpost AI, Quality team’s Protection Officers’ (PO) testing and evaluation efforts to ensure that humanitarian principles are upheld in Generative AI agent responses, have caught the agent stating wrong information with confidence. For example, in a synthetic prompt question, our Kenyan PO asks from the POV of a refugee in Kakuma, Kenya, what they should do to receive their mandate. The agent provides three plausible-sounding steps, complete with phone number and address information. The problem is , some parts of the response are hallucinations, where the procedure does not exist in the knowledge base and the phone numbers are made up. See the prompt, a partial response and PO’s comments in the screenshot below:
The POs have also identified hallucinated phone numbers and addresses as a persistent problem: the giving out of wrong or made-up physical addresses and phone numbers. Such issues, if not fixed, can lead clients to take actions which may waste their time, even possibly harm their well-being traveling to unvetted locations.
Hallucinations, at the time of writing, come with the territory of Large Language Models. There are concerted efforts underway to fix the problem of hallucinations but currently, even though they cannot be avoided but they can be minimized. In the next section, we look at some ways in which we are mitigating their occurrence.
Given the software nature of the agent infrastructure, it will sometimes suffer from software-related bugs which can result in errors.
In the longer term, there is also a risk, that the quality of LLMs could go down due to data bottlenecks [14] and future LLM training might be on poisoned synthetic data (data on the internet created by LLMs themselves) creating a circular doom spiral of diminishing output quality, an engineering problem referred to as “model collapse.” [15]
Mitigations:
SignPost AI is rigorously, developing, prompt-engineering, testing, and evaluating agent outputs for factuality, accuracy and reliability. Specific Mitigations include:
Rigorous testing and evaluation by Humanitarian Domain Quality experts & Red Team
Signpost Protection Officers (PO) are highly skilled humanitarian professionals who possess extensive experience and deep expertise pertaining to the specific country or context in which they operate. The POs use their Quality Framework to conduct evaluations and testing, “passing”, “failing”, or “red-flagging” problematic or hallucinatory/made-up answers. They also offer prompting suggestions to direct the bot to give a higher standard and quality of response. You can read more about their work flow on our blog.
Let’s take an example of how a PO used local prompting to make an inconsistent agent response into one that would be useful for our Signpost clients. In response to a synthetic user question about food distribution in Swahili, it outputs a generic response with no actionable information such as address or phone number of relevant organization:
The PO would flag this response as not meeting client-needs and attempt local prompting to coax the agent into giving the information that is already in the Signpost Knowledge Base. The PO uses the Content Management System to insert the following prompt:
“When the user talks to you about food, food distribution, summarize and generate a response from https://www.julisha.info/en-us/services/4189.”
The retest after the prompt finds that the AI agent is finally responding with good quality information to the synthetic query:
In addition to the POs, the Signpost AI Red Team also contributes to the accuracy and reliability of AI agent responses. They conduct Adversarial Testing, Continuous Improvement and Enhancements to ensure the quality, performance and safety compliance.. Their adversarial testing workflow can be seen below:
Signpost AI Quality and Red Teams consistently and deeply collaborate on Prompt Design and Engineering as well on rapid evaluations to mitigate hallucinations and increase reliability and accuracy and bring agent output to humanitarian-standard responses
As part of Prompt Design and Engineering, Signpost AI Agent technology is subjected to Local and System prompts and Constitutional AI Rules which in addition to other directives, ensure that it is following stringent guidelines on accuracy, reliability and information provenance. To give a sense of System Prompts focusing on accurate, verifiable information see screenshot below:
A partial list of Signpost Constitutional AI rules can be seen below, one rules which instruct the agent to be objective, considerate and using only factual and verifiable in its outputs.
Software bug-fixing is a recurrent and continuous process and will continue to be so. Software related bugs and errors are monitored and identified by Signpost AI Quality and Red Teams who create tickets for the development team to fix such bug-related issues immediately. An example of the ticket created for the error shown in the Problems section can be seen below:
Curated and Vetted RAG
Signpost AI uses a Retrieval Augmented Generation (RAG) pipeline to ensure that the agent gives responses based on carefully vetted, specific and contextual information. It also provides references of where the agent copies its textual information from.
Signpost AI’s RAG approach is built on a Vector Database (Vector DB) containing embeddings of service mappings and approximately 30,000 curated and vetted articles that Signpost has created to meet users’ self-expressed information needs.
Since a static RAG will depreciate in informational value, there are plans to ensure continual creation of user-centric articles through meeting new client needs and keeping information up to date. This serves as the source of the Signpost up-to-date knowledge base and is a key safeguard against inaccuracies and mis/disinformation. You can read more about RAG and how the Generative AI agent technology works here.
Data and Privacy Concerns
Problems
Use of Generative AI raises significant data and privacy considerations along a few different vectors. First , private AI companies confirm collection of all data that is sent by their LLMs, including conversations, personal information and other forms of sensitive information.[16] [17][18] There are no guarantees that this data will not be shared with third-parties and vendors for legal or business purposes.
This lack of privacy extends to use of proprietary algorithms that are engaged in internal processing and working of the agent. For example, the Signpost AI agent has an internal software tool that rechecks LLM output against Constitution AI rules to maintain ethical compliance, and is itself an OpenAI bot whose functionality depends on sending additional information back to the company.
There are no guarantees that such centralization of data may not be subjected to present and future use in training, fine-tuning, model evaluation during the AI development life cycle. Inadequate data policies for example have unintentionally contributed to humanitarian surveillance. There is also a fear that AI systems’ tendencies towards centralization and extraction give these technologies the quality of anti-localization [19], leaving out local, national groups and communities.
There is also a hypothetical risk that a threshold volume of prompts sent to LLMs might be used in future training to potentially produce a competing humanitarian AI product.
The Signpost AI agent is currently being tested on Claude-Sonnet, ChatGPT 4o and Llama 2 LLMs. These three LLM are receiving synthetic test prompts from the dev, quality and red-teams in addition to the data being sent by the internal Constitutional checker mentioned above. No personalized or private data is being sent yet but that could very well change with eventual deployment. This requires a comprehensive data governance and privacy policies compliant with regulatory instruments.
Mitigations
To safeguard user data privacy, Signpost AI has a rigorous Data Protection and Security Measures Policy which transparently outlines what, how and why data is collected, how it is secured and safeguarded and what actions are taken to protect user data privacy. These actions include:
Collection/deletion rules on internal platforms and databases
Anonymized and de-personalized data storage controls
Sharing agreements
Usage Guidelines, and
Cyber-security protocols and breach contingencies
These mitigation measures start from the ground up where privacy is at the foundation on top of which Signpost technologies are designed. Other mitigations include:
Legal protections inscribed in tech partnership agreements that Signpost AI data will not be for training Large Language Model
Vet contracts, and agreements with third party external partners to maintain safety, private and ownership of Signpost AI data
Compliance with existing regulatory instruments (EU AI Act, GDPR)
Quality and Trust
Problems
Signpost builds credibility, reputation and trust through providing high-quality individualized two-way information to meet user needs. In deploying an AI agent, there are risks that the “human touch” of human staff and moderators will be lost in these interactions. Other risks include general lack of confidence in usage of personal data, quality of outcomes, or concerns over AI fallibility. These risks of Generative AI agent deployments are particularly pertinent for Signpost; they raise questions over key parts of the Signpost reputation and mission, namely user trust, and individualized service.
There is evidence which shows user preference favoring humans over non-human chatbot interactions. For example, a report on humanitarian organizations’ past use of pre-LLM chatbots highlight that people “express a desire for personalized interaction” [20] There are also studies on AI chatbot-human interactions showing that people surveyed preferred interacting with human beings [21].
Furthermore, there is a risk that trust in Signpost will be negatively impacted by clients’ lower trust in AI. In the United States, surveys find that most Americans express reservations about AI.[22] Globally, according to a 2023 global KPMG survey, 3 in 5 (61 percent) are wary about trusting AI systems, while 67 percent report low to moderate acceptance of AI. This is partly alleviated by the finding people in emerging economies are more trusting, and positive about AI. [23]
Such AI related skepticism has the potential to risk Signpost reputation and trust and might lead to lower adoption/usage rates for technologically anxious users. In some cases, the presence of the chatbot might detract AI skeptical users from using the services. In others, there is a risk that those users distrustful of AI, using the service will have a lower likelihood of even using that information.
Humanitarian ethical obligations necessitate disclosure that users are informed that they are interacting with an AI chatbot rather than a human. Even in the case of a perfectly accurate and reliable chatbot providing quality responses, there is a risk, users’ AI skepticism will either preclude them from using the service or even if they use it, not trust it.
Current Signpost offerings through moderators provide specific, individual information to client requests. There are no guarantees that the Signpost AI agent, by the time, we have fully tested it, will be able to reach the same level of quality. Accepting a lower quality service provision even if by a little bit could be potentially detrimental to long-term Signpost reputation.
Even if human moderator level Signpost quality can be achieved by an AI chatbot, there is the question of “uniform” or robotic-sounding answers. In attempts to regularize chatbot outputs, there is a risk of homogenized, non-diverse responses, which while accurate, may not have “memorability” for the user. This memorability can come in the form of qualitative, subjective states such as “a kind word”, feeling heard, not made to feel that they are alone or sharing a moment of human connection, etc.
Mitigations
Signpost AI agent technology does not preclude human connection. Clients will be transparently and explicitly informed that they are interacting with a chatbot and will always have an option to speak to human staff whenever they so desire. The agent should be thought of as a tool providing basic information, and directing clients to specific services while still having a permanent option to speak to human staff.
Even though there is some pre-LLM evidence on user preference for humans over non-humans, other studies show users can come to use chatbots if they fulfill their needs. For example, on study highlights how clients’ human-like experience of AI chatbots enhances their trust and credibility (24). In another survey study, “most respondents preferred waiting for a [human] agent” but were open to chatting with a chatbot first to get information.[25] On the whole, how humans view LLM-powered AI chatbots still seems to be an inconclusive, open research question. A key point that emerges from the research, in humanitarian contexts, is the importance of fulfilling users’ needs for personalized information, regardless of the interactional form.[26] The testing and evaluation based on Signpost Quality Framework is to ensure such personalized client-centeredness in the humanitarian use-case.
Furthermore, the Signpost AI agent technology is optimized to essentially target this need. It optimizes for informational retrieval to respond to queries which have historically been asked for. The spine of the RAG-pipeline based design of the agent is the Vector DB which includes embeddings of over 30,000 human-curated articles created in order to meet specific client/user needs. Hence, the agent’s potential ability to be reliably accurate and relevant answering user queries depends on its retrieval, processing and presentation of existing information, not on generation of answers to novel questions.
If we can improve on this retrieval-processing-presentation reliability and accuracy, an attainable but challenging goal, we should be able to provide quality answers which fulfill user informational needs.
The mitigation hypothesis here, is that this use-value of the information, that it is reliable, accurate and meets the users’ information needs, is what makes Signpost credible and trustworthy. Hence, through transparent, responsible and ethical development, testing and evaluation, and ensuring that AI agent can reliably provide high quality use-value information, Signpost is by proxy, positively affecting trust and quality measures. The “human touch” in this hypothesis is not the key to Signpost quality and trust, rather a supplement. There is some research to back this claim; according to a systematic literature review of 40 empirical publications, the most influential factors when using agents for client service in non-humanitarian contexts are response relevance and problem resolution, which usually result in positive client satisfaction and increased probability for continued agent usage.[27]
Furthermore, time, labor and budget savings through AI agent services scale and this efficiency can be productively funneled into other creative human aspects of humanitarian work focused on on-the-ground efforts or direct to client assistance (cash).
Bias, harmful and Discriminatory AI
Problems
LLM output is intrinsically connected to the underlying quality of its training data and algorithms. Internet content makes up the bulk of this underlying data that Large Language Models have been trained on.[28] Such data is predominantly in English, and is culturally and epistemologically Western. English makes up the largest language (being at ~50% or higher) of text databases of crawled internet data used for LLM training. [29][30][31] As a result LLMs by default, assume anglophonic and Western contexts.
Due to its training data’s provenance, connected to the internet, LLMs are inherently trained on biased, stereotypical, misogynistic, discriminatory and harmful data. This also flattens descriptions of people from other parts of the world and does not represent diversity, complexity or heterogeneity. [32]
Given LLM optimizations for efficiency, performance and cost-savings, these models run the risk of exacerbating above-mentioned tendencies and do not have any no internal mechanism to foreground marginalized populations.
In the case of Signpost AI, our agent has variously used LLMs provided by major technology companies such as Google, Meta, OpenAI and Anthropic. Located in the United States, these LLMs contain all of the issues outlined above. Unfortunately, procurement of LLMs through these providers is unavoidable if one wants to use the technology effectively given other providers with more diverse training language corpus in their LLMs offer lower quality outputs. Given this intractable issue, mitigations are only possible downstream.
Mitigations
LLMs themselves have safeguards against bias and harm, but these cannot be relied upon in sensitive humanitarian contexts. While little to no mitigation actions can be directly applied to LLM outputs, Signpost has utilized mitigations downstream in the AI agent development. These actions include ethical guidelines and modifying problematic LLM output through prompt design and engineering.
For example, a few months of testing in, we found almost zero instances of agent giving biased, sexist or discriminatory responses and limited instances of agent giving harmful ones. This, we attribute to our partly to inherent LLM safeguards and partly our downstream interventions in the form of Constitutional AI rules, system and local prompts. See below for a harmful response, where the agent response encouraged illegal activities in its response:
The Signpost Red Team added the following prompt which fixed the problem:
After addition of System Prompt:
Other steps taken by Signpost to safeguard clients from unsafe AI responses include:
Signpost AI adopts Ethical and Responsible AI principles based on the humanitarian basis of “do no harm” to ensure quality and safety. These values and principles frame all of Signpost’s development, testing and evaluation and deployment decision-making.
This can be reflected in task-specific areas. For example, Signpost AI teams have specific frameworks related to Quality (see here) and Red-Team (see here) testing and evaluations respectively, all grounded on the idea that the Generative AI agent technology is being vetted for humanitarian use-case. These teams use the frameworks to carry out rigorous rapid evaluations to confirm that final outputs are above acceptable thresholds and closely mirror human moderator success rates
Creation and curation of a set of system and local prompts which “train” the agent to be kind, responsive and helpful as well as prohibiting discriminatory and harmful responses
Each output is checked against Constitutional AI rules which act as double-checks to ensure nothing biased,or harmful is let through. These rules, which direct agent behavior on client safety and protection are based: (a) on humanitarian values, (b) ethical principles and (c) Signpost human moderator guidelines handbook.
The Signpost AI agent technology is also being fitted with the ability to to detect sensitive high-risk queries. Its workflow will include human-in-the-loop who are activated on detection of such escalatory queries. High risk queries include: gender-based violence, child protection, self-harm, mental health, safety and security, etc.
In Signpost testing and evaluation so far, the agent has given almost zero sexist, and discriminatory responses while it has given single digit biased responses (e.g. responding to the definition of a political concept in a biased fashion). We have also red flagged and fixed potentially harmful outputs which leaked confidential employee data in one instance and internal documents storage information in another. In the screenshots below, the agent gives personal information about a Signpost staff. After prompting mitigations by the Red Team:
This is an ongoing process but the majority of outputs have been safe which attribute to our downstream mitigations listed in this section.
LLM Future Costing
Problems
Given Generative AI is a nascent, booming field searching for the killer use-case,, the current cost of LLM tokens should be cautiously seen as an introductory price to induce industry uptake. There is a risk of these costs going up in the face of a smaller and centralized number of LLM providers as well as costing which might increase in the form of maintenance, and technical support costs.
For example in Signpost’s case, LLM costing for a single query includes the input and output token costs of the following steps:
Extracted Contextual Information + Local Prompts + System Prompts + User Question Question + initial LLM output
Cost of Constitution AI rule checks where the initial LLM is response is checked against each of the rules and may undergo changes
Final AI agent response
Per token cost of even a small percentage would increase cost burden across processes of the agent technology.
Mitigations
There is scope for a longer-term partnership with technology partners which will help discount some of the token costs of using LLMs and agent internal tools. A crisis cache of cloud credits and automated deployment will allow for flexibility and speed in crisis and the ability to share resources among tenants.
There are plans that Sustainable, and successful implementation of AI agent will prompt a move to open source model and move operations in-house. This will entail higher up-front costs and incur increasing cloud costs but in the longer-run will have the safeguard Signpost from long-term external upward trending LLM cost lines
Pricing algorithms are opaque, i.e. when an agent response is made, there is no exact accounting of what each feature cost. Currently, Signpost AI’s options include an API rate limiter to control overuse of API and to avoid adversarial cost attacks.
Signpost AI is attempting with technology partners to gain the ability to audit price such workflows. What is the cost per feature? What does it cost to add a risk-detection bot or a new constitutional rule? How would changing parameters which limit the amount of text generated affect costing? Price auditing would make it much easier to answer questions about which features can feasibly be added to the AI agent. These answers are crucial, as they will determine the scope of the agent's capabilities.
The Black Box Problem
Problems
Generative AI suffers from the Black Box problem; how Generative AI tools process and generate answers is not disclosed. Neither is the provenance or ownership of data that it uses to generate outputs. Only the inputs and the outputs are visible and everything inside the technology is a mystery or a black box.
There is general research consensus on how LLMs’ neural networks are trained on billions of words of language which are then used to “predict the next word.” [33] Why they are black boxes is because of the sheer complexity and scale of billions of parameters which make them difficult to pinpoint the machine’s decision-making process. They also often exhibit “emergent behaviors”, or capabilities that were not explicitly programmed.[34]
Explainable AI efforts (XAI) attempt to open this box to make such processes comprehensible for humans but such efforts have only had limited success thus far. [35]
This opacity poses the problem of trust in GenAI systems which can cascade down to second order unknown risks, open to legal, regulatory, accountability and responsibility challenges.
The Signpost AI agent technology is also connected to such black boxes. This raises a host of issues covered above related to data protection, privacy and the kinds of discriminatory outputs that are returned. This is the fixed reality of both closed and open sourced Generative AI technology. In order to use this technology, currently, the only recourse is downstream, adding contextual information and specific rules to inputs and de-risking LLM outputs.
Mitigations
Signpost cannot decipher the inner-workings of the LLM, but we can mitigate the effects of this unknowability downstream. While LLM processes and outputs are black-boxed and unknown, Signpost AI agent technology is designed to be fully open, documented, explainable and transparent.
We may not know the inner workings of LLMs but we do know the inner workings of the Signpost AI technology, de-risking mechanisms are being applied and how they are working. For example, through agent logs, we are able to trace what search terms the agent generates in response to a user question, what information it extracts from our knowledge base, what and how it changes LLM responses based on Constitutional AI checks. We also have visibility on its ability to detect queries for “contact”, language being used or the location of the user. You can see screenshots highlighting how our AI agent responded to a question, what search terms did it detect, what text it extracted and from where, what answer it generated, etc.
This kind of information allows Signpost development, red-team and quality team to test, evaluate and make necessary adjustments. It also makes the task of sharing our work publicly transparent and easier.
Security Issues
Problems
Generative AI is the new frontier of cyber-security. There is research that existing security and data infrastructures around Generative AI architectures are inadequate to hold back new threats.[36] In the agent space, such threats include a mixture of the old and the new.
Data Poisoning: the injection of harmful/biased information into the knowledge based used for retrieval
Prompt Injection Attacks: Input prompts which could manipulate agent into bypassing security/protection measures or reveal sensitive information
Information Leakage: agent might inadvertently reveal sensitive or confidential information from its knowledge base
Adversarial attacks: inputs designed to mislead the agent to potentially malfunction or produce harmful outputs
API Vulnerabilities: A agent accessible via API will be vulnerable to injection attacks or authentication bypasses
Denial of Service (DoS) Attacks: a technique through which a system is overwhelmed with service requests causing it to become unavailable
Data Breaches: User interactions with the agent could be intercepted or stored insecurely
Given the nascent nature of the technology, there are also many unknown unknowns in the threat matrix.
Most of these threats potentially apply to our use-case and we are preparing safeguards. For example, DoS attacks are likely given our operating environment while our usage of API does open a potential attack vector. Some of these threats have already been identified through our testing. For example, we caught and identified the agent giving out confidential information related to our staff.
We have also mapped potential future threats associated with future data usage in the form of White and Black Box attacks.[37]
Mitigations
Signpost is undertaking a proactive approach toward ensuring robustness of our AI agent. Signpost AI has an AI Data Protection and Security Measures Policy which maps areas of concern and offers safeguards in our use-case. This policy takes a “no-size-fits-all-approach” security approach adopting measures based on the level and types of risks that emerge from specific processing activities.
To ensure robustness, a dedicated Red Team proactively identifies and evaluates potential vulnerabilities, security risks, and unintended consequences. [38] This redteaming does security testing, simulates adversarial strategies, attacks and techniques, and proactively explores potential security risks expecting the to find the unexpected.
For example, Signpost’s system utilizes two layers of bot protections built into our communication routes limiting API rates to ward off DoS attacks. Similarly our Red-teaming efforts have identified information leakage and ensured that personal information of any kind is purged from the knowledge base.
Finally, we have utilized existing data safeguards (with respect to storage locations, duration and data retention and deletion ) to the Signpost AI agent as well. Signpost is also exploring embedding privacy considerations through “Privacy by Design” principles, into all aspects of AI system development and deployment. In case of a security breach , Signpost has also scenario-planned and set out response protocols.
External Future Risks
In this section, we examine the broader Generative AI landscape to identify potential issues and risks that may impact us in the future.
Regulatory, Legal Challenges and Generative AI Adoption
Problems
Generative AI LLMs face a number of legal issues which could have a negative impact on their quality. The legal challenges come from media organizations [39][40], government regulators[41], and creative industries[42][43] and span investigations over potential consumer harm and litigation over copyright infringement, and violations of intellectual property.
The outcome of these challenges and investigations is unpredictable but whose negative trajectories will have a negative effect on Signpost AI agent quality downstream.
Mitigations
Regardless of how litigation and regulatory investigations go, the technology is widespread enough that it is not going away. Pandora’s box is open.
The legal cases will take years if not decades to resolve fully. Even in the worst legal or regulatory outcomes, liability concerns within current regulations for Signpost in the space are not very high.
There is a deep question here about what the humanitarian sector should do, in the interim, about Generative AI?
The proliferation of AI technologies across diverse markets, sectors, and national contexts presents a double-edged sword. While these advancements offer significant potential for progress, they simultaneously pose grave risks of misuse while introducing a range of AI-related harms.
Some humanitarian actors advocate that it is their responsibility “to ensure AI’s positive potential and protect those most at risk of being negatively impacted by its use.”[44] Others advise caution, linking hasty, poorly-thought out adoption of Generative AI to a “technosolutionist” mindset.[45] Another position posits that the adoption of Generative AI in the humanitarian sector is not a question of if but how. [46]
Our efforts at Signpost AI are an attempt to ground and operationalize ethical values and humanitarian-principles in the development of a Generative AI tool. Our how is to safely cater both to the increasing needs of our communities while improving our ability to do so through this technology.
Long-Term Quality Concern over LLMs
Problems
Research on scaling laws [47] predict that increasing model size, training compute and dataset size will improve LLM outputs.
While additional training computing power is coming online, there are concerns over training costs of ever larger model sizes and bottle-necks on obtaining new training data, which might contravene scaling laws’ trend extrapolations, i.e. Generative AI quality will hit a wall.
Obtaining additional data is less likely than it seems. According to a recent study, publicly available training data for LLMs will be exhausted some time between 2026 and 2032[48][49], while the jury is still out on whether generation of synthetic training data will have the same effect as having more high-quality human data.
There are also concerns that different LLMs might recursively cannibalize each other in future training periods.[50] This refers to the act of different AIs consuming others’ outputs and vice-versa creating a spiral of low-quality built upon cascades of computer-generated data.
There appears to be no short-term risks with AI scaling up with at least two more cycles of scalar improvements predicted.[51] It is crucial for Signpost AI to keep an eye on these developments in the longer term as any change in LLM quality will directly affect functionality of its AI agent.
Mitigations
Signpost AI agent technology relies on the current quality of LLM output and combines its ability to pattern-match accurately and retrieve quality information. If we are able to effectively utilize current LLM quality to access our curated knowledge base and provide good answers, we will have a viable product. One, the lack of future LLM improvements/capabilities will have less of an effect on our use-case.
Signpost's current value proposition is in its creation of valuable, user-specific information in the form of articles and using their ground truths to provide answers to our clients in a timely fashion.
Our commitment has always been to continually enlarge and improve this knowledge base to scale information provision; by appraising, inspecting and vetting new relevant sources of audio, visual and textual humanitarian information before ingesting them into our knowledge database. This commitment remains undimmed. We may not control the future quality of LLMs but we can control our knowledge base.
Effects on Climate
Problems
While AI offers benefits in combating climate change; from innovating new materials and modeling[52], climate forecasts [53], monitoring ice melt and deforestation [54] and optimizing deforestation, it is also a huge burden on global power consumption [55] leading to increased carbon emissions and subsequent climate effects.
It is not just energy-intensive but also fresh water intensive which is required to cool servers and processors. [56] According to one preprint study, demand for water for AI could be half that of the UK by 2027 [57]
Finally, with the focus and hype strictly on AI, there are signs that attention and money are being funneled away from climate-change efforts and mitigations. [58] Climate crises already disproportionately hit poor and marginalized and at-risk communities in developing countries. [59] Increased AI implementation directly risks worsening the conditions of the very populations we are trying to serve.
Mitigations
Signpost AI will calculate our emissions based on our LLM usage and engage in carbon offsetting arrangements. Use of Generative AI agents will be made judiciously only to serve the needs of vulnerable clients. The global nature and scale of this issue poses significant challenges to engage in scaled up efforts.
We have detailed the attendant and potential risks of Generative AI and our efforts at their mitigations in our Signpost AI Chabot. Do these mitigations work? How well do they work? We cannot know for sure but our current actions towards developing a Generative AI agent technology and grounding ethical, humanitarian questions around it are meant to be investigations, to find answers to such questions. Being transparent about how we are de-risking AI, and how we grapple with ethical issues, is our way to be as rigorous, responsible and accountable.
We believe this is a necessary process, one that the humanitarian sector as a whole should work on together.
The question of whether to implement Generative AI in the humanitarian sector or not is ultimately a question of value-driven decisions and governance more than it is about the technology itself.
In the second part of this paper, we turn our attention to the aftermath of implementation. What are the associated benefits and trade-offs of implementing a Generative AI tool?
Currently there are AI and ML models being used to improve humanitarian impacts [60]. For example (i) applications delivering impact for individuals, communities and programmes , (ii) tools improving organizational operations and systems and (iii) external environments in which humanitarian agencies operate. The Signpost AI agent technology can be placed primarily in this first category. We will now look more specifically at the mapping of potential benefits and trade-offs in our use-case.
Benefits and Trade-Offs
Efficiency and Scalability
Generative AI tools’ ability to produce domain-specific answers based on already-vetted information offers an augmentation of Signpost’ information provision services. Done correctly, the Signpost AI agent technology will make Signpost services significantly more efficient, at the same time, offering an order of magnitude increase in scaling options to meet client and user information needs.
Given the humanitarian context of increased client need and funding deficits [61], this use-case of GenAI offers an unprecedented ability to reach more people with fewer resources. The specific benefits would include:
Signpost’s ability to reach more underserved populations
Signpost AI agent technology would lower workloads of moderator staff, allowing them dedicated more time for creative, on-the-ground, high impact qualitative work
The AI tools’ ability to simultaneously handle large number of clients’ requests will allow Signpost to provide much needed information to more users during acute crises
Time and cost savings over the long run can be strategically deployed towards more on-the-ground qualitative of humanitarian work
As scale increases, Signpost AI expects even more moderators will be needed to handle larger number of high-risk escalations
Signpost AI has quantitatively approximated such efficiency and scaling gains with successful pilot and final product deployment:
[Inquiries made by client] 30% increase in the number of inquiries from clients, meaning more engagement.
[Scaling] Access to the tool will be expanded to at least 10 countries and scale to a rate of 100,000 people served (per year).
[Clients Reached in Pilot] Reach 5,000 clients with the bot in one to two Signpost instances
[Content Creation Impact] The AI proposes 20 new or revised articles per month, with a 75-85% approval rate by the editorial team.
[Form Classification Accuracy] 30% increased classification accuracy as scored by our metric framework in AI-assisted ticket form classification.
Possible Tradeoffs:
The initial set up, development, implementation and roll-out of the Signpost AI agent technology will incur upfront costs with no guarantees of a return
Displacement of human moderator work by an accurate AI tool might result in losing the “human touch”. It could also result in the relocation of domain expertise from humanitarian-based interactional expertise to a more data-driven one. [62]
Successful agent implementation and roll-out might raise fears about unemployment and job loss among staff. It is paramount that such fears are assuaged through verbal and practical and material reassurances
Accessibility
Signpost AI agent technology will always be on and operational. It will be able to operate 24/7 as well as during holidays and weekends. It will also be very fast to respond to clients. This greater accessibility can greatly lower moderator workloads while giving clients the blanket safety of reaching out any time they require information or help. Forecasted Signpost AI benefits include:
[Time until first response to inquiry] 20% reduction in average response time per ticket in human in the loop model. 1000% reduction in average response time per ticket in human on the loop model.
[Client Wait Time] 50% decrease in the average time to resolve a ticket in human in the loop model and 200% decrease in the average time to resolve a ticket in human on the loop model due to more efficient response drafting and information retrieval during business hours.
[Increased moderator availability] 60% reduction in tickets requiring tailored responses from an agent, increasing time to generate new content or respond to more tickets.
[Editing Time] 20-30% reduction in human agents' time editing AI-generated responses due to improved AI drafting capabilities.
Possible Trade-offs:
This setup will still require human(s)-in-the-loop to monitor agent performance in case of malfunction/hallucinatory events or if the request is one that requires escalation to human-intervention. This means additional maintenance, human effort and overhead costs on holidays and off-days
The processing-time that a agent might need to provide a final output could detract potential users. Time-to-response values are worth considering especially in immediate crisis situations where time to escalation to human will be of the essence
Underserved communities without access to a reliable internet connection might have issues accessing the Signpost AI agent services. However, the AI agent will still be faster with access times maxing out at two minutes. Signpost AI is also working on potential solutions in the form of kiosks or AI on devices which field specialists can use in inaccessible geographies
Technology outages will amplify loss of service delivery simultaneously across different instances
There might be upstream infrastructural issues and disruptions given that Generative AI technology is relatively new, being rejigged and will undergo downtime pain points as newer hardware/software goes online or API setups are reconfigured, etc. Such AI infrastructural issues will lead to intermittent inconsistent services. For example, evaluators encountered the following agent errors related to LLM provider’s infrastructural issues:
Multilingual support
Generative AI technology is based on an innovation that was initially created to do improved language translations. As a result, Generative AI’s quality and performance out-of-box in multiple languages is very high, better than has been available for even low-resourced languages.
Signpost AI agents are developed to scale into different languages and hence easier to implement in new country contexts. For example, currently, in addition to English, we are also doing early tests on Arabic, Swahili, and Pashto. Signpost AI is also collaborating with technology partners to use new state-of-the-art LLMs which cover 100+ languages in order to better facilitate different language needs for our clients.
Possible Trade-offs:
While Signpost AI agent technology can be repurposed to new languages, LLMs are generally only very good at well-resourced languages. Their performance for low-resourced languages is not service-provisionally adequate. This could potentially change with LLM models specifically created for low-resourced languages
Multilingual support may potentially require a completely different set of evaluations. Languages are not just words but embed local contexts and assumptions which may not be reflected in agent response
Cost-effectiveness
Improved scale can be achieved through minimal costs. As a result, per-instance cost and per-user cost will trend downwards offering more per dollar value for funders and donors. This lower cost structure can lead to future organizational sustainability.
Signpost AI also hopes evidence-based successful implementation of agent will allow us to offer this technology product to others in the humanitarian sector at minimal costs.
Possible Trade-offs:
Signpost AI agent technology on a larger scale will incur novel maintenance costs: technical maintenance, updates, etc. LLM costs may also increase as usage increases (also refer to Risks over LLM costing)
Reliable and Consistent
A Signpost AI agent technology infrastructure is being rigorously developed and tested based on ethical grounds to offer outputs based on reliable, consistent, and accurate information. These efforts are aimed at ensuring that there is no deviation from the Signpost knowledge base and service mappings of approximately 30,000 user-needs based articles contained in the Vector DB.
[Accuracy in response content] 85-95% of AI-generated responses require no or minimal edits by human agents, indicating high initial accuracy in human in the loop model.
[Knowledge Base Use] 50-60% increase in the use of knowledge base articles in ticket responses, indicating parameterized responses.
Possible Trade-offs:
Potential lower quality of responses through response standardization
This will result in a “human touch lost” situation; it will be difficult for agent to find a balance that human moderators do: that between giving accurate and informative answers and making users feel they are being listened to and given individualized attention to (also see Quality and Trust Risk mitigation above)
Lessons for Humanitarians on Implementing AI So Far
Based on analyzing risks, benefits and trade-offs as well as what we have learned from our efforts of developing, testing and evaluating the Signpost AI agent technology, we believe there are a few key factors to keep in mind when thinking about Generative AI. Lessons for us, at least, so far have included the following:
The Importance of Humanitarian Principles and Values
The key building blocks of any AI project should be underpinned by humanitarian principles and values. In our case, we have combined our humanitarian principles with those specific to AI in our Ethical and Responsible Approach to AI:
Ethical and Responsible: Ensuring that our AI portfolio is equal, human-centered and does no harm
Transparent: Dedication to openness, accountability and trustworthiness
Evidence-Based: Rigorous efforts made to affirm that our AI tools are effective, competent and credible
Collaborative: ultimate success of our work depends on inclusion, stakeholder relationships and mutual knowledge-sharing and production
Practical AI Thinking Versus Existential AI Thinking
AI discourse is replete with sensational reporting on AI’s technological industrial revolutions [63] or doomerism that predicts the end of humanity[64]. Speculative extrapolation of trend lines and overblown hype litters the landscape of AI discussions.
Signpost AI’s position is to cut through these existential conversations, drill down to a specific implementation use-case and explore in practical details the question of AI ethics. This Practical AI Thinking approach is not to discount AI future outcomes completely but to bring rigorous empirical evidence and learnings to the conversation.
An essential component of this approach is to engage in careful and thoughtful testing and evaluation and allow their results to be the beacon for next steps.
Signpost AI Projects as Public Goods
We work in sensitive contexts where decision-making needs to be based on reliable, accurate and up-to-date information. Inability to hold this goal can have catastrophic impacts on our communities.
This is why Signpost AI principles and values position our AI offerings as Digital Public Goods, whose workings and outputs are transparent, widely available and subject to external tests of technical reliability. We invite sector and non-sector partners to review our model frameworks, data policies, legal compliance, impact assessments and trace the information-workflow of our AI products for quality and ethical assurance.
AI Literacy
As we undertake AI development, what has become increasingly apparent is the need for AI Literacy and an infrastructure for its spread. This literacy is aimed towards arming users, humanitarian staff and stakeholders with knowledge required to understand how Generative AI works in practice, its larger context and associated risks, benefits and trade-offs.
This requires investments in increasing basic understanding and explainability of the technology tailored to various roles and parts of the humanitarian system. User communities, humanitarian staff, technical specialists, C-suite officers, etc. all require bespoke knowledge needs in terms of understanding of (a) GenAI in general and (b) specific implementations of Gen AI and how it might affect their own work.
This understanding and literacy is crucial in fostering an inclusive, community-directed AI, where local data, talent and perspectives are folded into the development process. It is also critical for using AI safely, responsibly and effectively in humanitarian efforts, including advancing standards and policies that ensure ethical AI development life-cycle and learning from past tech adoption cycles to avoid repeating mistakes.
Conclusion
Signpost has the responsibility of providing crucial, potentially life-saving information services to the world’s most vulnerable populations. Our interest in exploring GenAI implementations in the humanitarian sector is not predicated on FOMO or adoption of the newest technological fad but in opening the conversation on not only if, but also how we can ethically and responsibly meet the needs of our communities at scale through this technology. [65]
The how of AI is a harder question, requiring a meticulous, rigorous approach to AI governance, development, evaluation and deployment built on humanitarian principles and values. It is only if we have evidence-based clarity on the “how” that we can approach the “if” question in good faith and ultimately seek resolution to the humanitarian sector’s moral responsibility towards a technology that can potentially help manifold more people.
This is what the exercise of this paper is in service of; combining realities of risks, benefits and trade-offs of practically developing an AI agent technology. We are very excited by what it teaches us, others and what we can take forward to fulfill our core humanitarian mission. We hope efforts such as this will kindle active collaboration and knowledge-sharing among humanitarian organizations to collectively address the challenges, and leverage the benefits of Generative AI.
References
[2] Explained: Generative AI | MIT News | Massachusetts Institute of Technology
[3] UBS: ChatGPT is the Fastest Growing App of All Time
[4] Economic potential of generative AI | McKinsey
[5] Don't wait—create, with generative AI
[6] 'Black box' of generative AI: Importance of transparency and control
[7] Generative AI like Midjourney creates images full of stereotypes - Rest of World
[8] Science in the age of large language models | Nature Reviews Physics
[9] The Times Sues OpenAI and Microsoft Over A.I. Use of Copyrighted Work
[11] [2303.08774] GPT-4 Technical Report
[12] [2303.08774] GPT-4 Technical Report
[13] Note: A parallel narrative frames this same inherent hallucinatory quality of LLMs as desirable because it gives Large Language Models a creative facet, which has the potential to foster innovation
[14] [2211.04325] Will we run out of data? Limits of LLM scaling based on human-generated data
[15] A New Study Says AI Is Eating Its Own Tail
[18] Privacy & Legal | Anthropic Help Center
[20] CHATBOTS IN HUMANITARIAN CONTEXTS
[21] Mani, Z., & Chouk, I. (2018). Consumer resistance to innovation in services: Challenges and barriers
in the internet of things Era. Journal of Product Innovation Management, 35(5), 780–807.
[22] MITRE-Harris Poll Finds Lack of Trust Among Americans in AI Technology
[23] Trust in artificial intelligence - KPMG Global
[24] Murtarelli, G., Gregory, A., & Romenti, S. (2021). A conversation-based perspective for shaping
ethical human – machine interactions : The particular challenge of chatbots. Journal of Business
Research, 129(March 2019), 927–935.
[25] What Do Your Customers Actually Think About Chatbots? [Research Study].
[26] Cole, Kevin. 2024. “Navigating Humanitarian AI:”
[27] Nicolescu, Luminița, and Monica Teodora Tudorache. "Human-Computer Interaction in Customer Service: The Experience with AI Chatbots—A Systematic Literature Review." Electronics 11.10 (2022): 1579. ProQuest. 19 July 2024 .
[28] How Tech Giants Cut Corners to Harvest Data for A.I. - The New York Times
[29]LAION-5B: A NEW ERA OF OPEN LARGE-SCALE MULTI-MODAL DATASETS
[30] Statistics of Common Crawl Monthly Archives by commoncrawl
[31] https://arxiv.org/pdf/2201.06642
[32] Generative AI like Midjourney creates images full of stereotypes - Rest of World
[33] A jargon-free explanation of how AI large language models work | Ars Technica
[34] [2206.07682] Emergent Abilities of Large Language Models
[35] What is Explainable AI (XAI)? | IBM
[36] https://www-tandfonline-com.libproxy.newschool.edu/doi/full/10.1080/08839514.2022.2037254
[37] How should we assess security and data minimisation in AI? | ICO. White Box attack refers to one where the attacker has complete access to the model and is able to inspect underlying code and properties whereas in a Black Box attack, the attacker only has the ability to query the model and observe relationships between inputs and outputs. See link for more information.
[39] The Times Sues OpenAI and Microsoft Over A.I. Use of Copyrighted Work
[40] OpenAI and Microsoft Sued by Center for Investigative Reporting
[41] F.T.C. Opens Investigation Into ChatGPT Maker Over Technology's Potential Harms
[42] John Grisham, other top US authors sue OpenAI over copyrights | Reuters
[43] Music industry giants allege mass copyright violation by AI firms | Ars Technica
[44] USAID. 2024. “Artificial Intelligence Action Plan | Document.” U.S. Agency for International Development. Retrieved July 23, 2024 (https://www.usaid.gov/digital-development/artificial-intelligence-action-plan)
[45] Cole, Kevin. 2024. “Navigating Humanitarian AI:”
[46] Spencer, Sarah. 2024. “HPN Network Paper.”
[47] [2001.08361] Scaling Laws for Neural Language Models
[49] [2211.04325] Will we run out of data? Limits of LLM scaling based on human-generated data
[50] AI Is an Existential Threat to Itself
[51] Bill Gates Reveals Superhuman AI Prediction
[53] How AI is improving climate forecasts
[54] 9 ways AI is helping tackle climate change
[55] AI Computing Is on Pace to Consume More Energy Than India, Arm Says
[56] Generative AI’s environmental costs are soaring — and mostly secret
[58] AI Obsession Obscures Bigger Promise of Climate Tech - Bloomberg
[59] The climate crisis disproportionately hits the poor. How can we protect them?
[60] Spencer, Sarah. 2024. “HPN Network Paper.”
[61] Global Humanitarian Overview 2024, February Update (Snapshot as of 29 February 2024) | OCHA
[62] Current moderators’ expertise is foundationally built upon their experiences interacting with clients. Successful agent implementation will necessitate relocation of this expertise to a more above-the-ground expertise of data analytics
[63] AIPolitan | Cryptopolitan
[64] Exclusive: 42% of CEOs say AI could destroy humanity in five to ten years | CNN Business
[65] Spencer, Sarah. 2024. “HPN Network Paper.”