Investigating “Society in the Loop”
Introduction
After the long “AI Winter” of the 1980s and 1990s [1], the theoretical and practical advances in AI technologies such as Deep Neural Network architectures [2], Reinforcement learning [3] and Transformers [4], have resulted in an accelerating ubiquity of AI technologies in everyday life [5]
This AI proliferation has yielded substantial societal benefits such as efficiency gains in supply chain management, augmenting the scientific method [6], more reliable medical diagnosis and drug discovery [7], improved customer service, and climate change mitigation [8].
AI, like most technologies, is a double-edged sword; along with these benefits come risks, harms and questions of regulatory and governance mechanisms for complex algorithmic systems and autonomous machines. A large portion of these questions emerge due to such systems being black-boxed [9], whose inner workings are hidden to stakeholders. AI in the form of recommendation systems have broadly been found to create filter bubbles [10], which can isolate individuals and amplify particular viewpoints. There have been arguments made that such systems also perpetuate injustice because of bias built either in their design or in their training data [11]. In the case of predictive policing or rating creditworthiness, such AI algorithms have been found to reinforce inequality through feedback loops [12].
In response to these harms and problems, states, organizations and civil society actors have started thinking about AI governance. Given the impetus that has been given to AI adoption through Generative AI, there have been plenty of discussions on AI governance, AI Ethics [13], Data Privacy and Protection [14], and AI regulations.
In this research, we will explore a conceptual framework put forward by Iyad Rahwan called Society-in-the-loop (SITL) to think about AI regulation and data-driven systems [15]. This concept emerges from the idea of Human in the Loop (HITL) and delves into how to think about AI more holistically, by outlining mechanisms for negotiating values of various stakeholders affected by AI systems.
In this piece, we will first explore details on what is Human-in-loop (HITL) and Human-on-the-loop (HOTL), how they are used and how they differ. This will be followed by an explanation of Rahwan’s SITL. We will then evaluate what SPAI and the humanitarian sector can learn from this concept.
Human in the loop (HITL)
Human-in-the-loop (HITL) is a branch of AI which brings AI and human agents together in a machine learning model/system. In this configuration, humans are often involved in initialization, exception control, optimization, supervision and maintenance processes. This concept has been studied in the field of supervisory control for decades. [16] [17] [18]
This idea of humans being essential components in automated control processes made its way in the field of Human-Computer Interaction (HCI) where it has been integrated into mixed-initiative interfaces. This refers to situations where the autonomous system can make decisions based on when and how to engage the human [19].
More recently, this idea has been borrowed from HCI in thinking about AI and ML systems. For example, HITL thinking has been applied in humans labelling data for ML algorithms [20], and human-robot interactions. [21]
While HITL thinking has improved accuracy of AI systems, it is also a powerful tool for regulating AI system behavior [22].
In recent AI HITL systems, humans are often actively involved in decision-making or operational steps. The system is unable to complete certain tasks without explicit human input or approval. HITL systems are excellent in sensitive use-cases where the environments require a high degree of accountability and ethical scrutiny or where AI systems can be improved through iterative and constant human inputs. Examples can include content moderation where AI flagged posts are reviewed by humans [23], and medical diagnoses where doctors verify AI-generated predictions . Another example comes from Signpost AI pilot in the humanitarian context, where SPAI moderators generate copy with an AI tool, then meticulously verify its veracity, tone, formatting, and accuracy before sending a complete response to a client request. [24]
HITL’s major functionalities include the (a) ability to identify misbehavior by the system and take corrective action (to both improve the system and eliminate out errors) (b) provide an accountable entity in case the system misbehaves [25] This is particularly important in cases where the behavior of an AI system may cause harm or have detrimental effects for humans.
HITL is a useful design paradigm for constructing AI systems subject to oversight. As we will see later in this article, this oversight might not be as comprehensive as might be required in sensitive contexts.
What is Human on the Loop (HOTL)
In HOTL systems, humans oversee and supervise an automated system but are not directly involved in real-time decision-making. Their role is more supervisory, stepping in only when necessary. HOTL systems are designed such that they are empowered to make and enact their own decisions [26] [27]
In such automated environments, the role of humans is often to (a) monitor for errors or anomalies, (b) initialize, set parameters or define objectives before deployment or intervene (c) intervent in rare, exceptional or emergency cases with overrides or manual adjustments. Examples of this include full autonomous vehicles where humans can intervene if the system fails , automated financial trading platforms monitored by analysts or Autonomous weapon systems with human oversight for critical engagements.
HOTL systems are often useful in more routine task environments with often predictable outcomes where speed and efficiency are prized and where human intervention would slow operations and in scenarios where the AI system has been trained to operate with a very high level of confidence.
HITL and HOTL do not exist as exclusive choices where only one paradigm might be applied. They can co-exist within the same system. Some self-adaptive systems (such as computer-vision systems in Unmanned Aerial Systems or drones) [28] are capable of responding to changes in the system and environment by enacting adaptive behavior such as switching modes of operations or by reconfiguring parameters within the mode. A special form of adaptation occurs as a result of uncertainty in the computer vision model where the system temporarily switches from HOTL behavior to HITL behavior. This often occurs because CV is perceived to have become too unreliable for performing task and so control must be given over to the human [29].
Society in the Loop (SITL)
Whereas HITL and HOTL AIs are about embedding the judgement of individual humans or groups in the optimizations of AI systems with narrow impact, Society in the Loop (SITL) is about embedding values of society as a whole, in the algorithmic governance of societal outcomes that have broad implications. [30]. The differentiating factor is the scope of both the input and the out of AI systems; SITL becomes relevant when this scope is very broad.
According to Rahwan, SITL raises a fundamentally different question to that of HITL: how to balance the competing interests of various stakeholders, including interests of those who govern through algorithmics. Using the construct of a social contract, he equates
SITL = HITL + Social Contract
Using Jean-Jacques Rousseau’s definition, Rahwan explains the social contract as one that provides the efficiency and stability of sovereign states, but which also ensures that the sovereign implements the general will of the people [31] and is held accountable for violations of fundamental rights. Rahwan argues, a similar contract is necessary for the use and governance of algorithms, an algorithmic social contract.
Within HITL and HOTL systems, the human controller ensures that AI systems fulfil uncontested and common goals of its stakeholders. For example, a human pilot overseeing an airplane autopilot to increase passenger safety. In an SITL system, various societal stakeholders must identify the fundamental rights that the AI must respect, the ethical values that should guide the AI’s operation, as well as the cost and benefit tradeoffs the AI can make between various stakeholder groups. [32] .
This means that there should be broad agreement over tradeoffs, for example, between security and privacy or between different notions of fairness. This also means a social agreement on which parties reap which benefits and pay which costs. For example, how improvements in safety made possible by autonomous driving cars are distributed between passengers, pedestrians and stakeholders. [33]
A visualization of Society-in-the-loop by Rahwan can be seen below:
Figure 1. Society-in-the-Loop (SITL). From: Rahwan, Iyad (2017) “Society-in-the-Loop Programming the Algorithmic Social Contract”. Page 5
Implementing SITL requires a few ingredients:
Knowing what types of behaviors people expect from AI and to enable policymakers and public to articulate these expectations to machines (goals, ethics, norms, social contract) to machines
To close the loop requires new metrics and methodologies for evaluating AI behavior against human values. These new tools can monitor the algorithmic social contract between humans and algorithms. According to Rahwan, this requires government regulation and industry standards that represent the expectations of the public as well as provide oversight [34]
Challenges to SITL
There are a few challenges to the implementation of SITL:
Articulating Societal Values: there is a gap between how the humanities-based experts (legal scholars, legislators, ethicists, etc.) reveal moral quandaries, issues and potential violations and how these are articulated by designers and engineers. The insights of the former are hard to operationalize within AI systems and are generally watered down to meet pragmatic aspects of project and product management.
Quantifying Externalities and Negotiating Tradeoffs: The negative externalities - costs incurred by third parties not involved in the decision - of algorithms are not always straightforward to quantify, especially over longer time frames. Even if such quantification is made, the necessary trade-offs and their quantification is itself a complex and arduous process where unintended consequences may not be properly mapped.
Verifying Compliance with Societal Values: the inability of computer scientists and engineers to adequately quantify behaviors of AI systems make it difficult to scrutinize this behavior against set expectations and metrics. [35] This in turn, makes the act of evaluation, and compliance difficult.
SITL for Signpost AI?
Signpost AI's (SPAI) retrieval-augmented generation (RAG) AI chat tool has predominantly been evaluated through the lens of Human-in-the-Loop (HITL) methodology. This focus is particularly relevant given that the tool has been developed to facilitate information dissemination for vulnerable populations, including refugees, migrants, and asylum seekers. Such a paradigm is suitable as it emphasizes micro-level considerations pertaining to the accuracy, tone, appropriateness, and format of the responses generated by the chatbot for its users.
The current iteration of the Signpost AI (SPAI) chatbot is unable to independently complete the information delivery lifecycle with the necessary accuracy and appropriateness, necessitating active human involvement for editing, decision-making, and operational execution to fulfill the final stages of information provision. Given its focus on accountability and ethical considerations, the Human-in-the-Loop (HITL) approach is particularly well-suited for this context.
Additionally, the potential for Human-on-the-Loop (HOTL) methodologies has been examined at SPAI. This approach envisions a future version of the SPAI chatbot that would autonomously provide information-related responses, with human oversight limited to high-risk inquiries or emergencies.
Considering the current capabilities of large language model (LLM) technology, and in the absence of significant advancements that mitigate issues such as hallucinations, the most pragmatic vision for the SPAI chatbot appears to involve a hybrid model that integrates both HITL and HOTL paradigms in its final deployment.
In the context of the Signpost AI chatbot operating within the humanitarian sector, the incorporation of a Society-in-the-Loop (SITL) model merits careful consideration. This author posits that exploring a SITL framework is particularly valuable given the unique challenges and ethical imperatives inherent in humanitarian work.
In a sense, Signpost AI (SPAI) already incorporates certain elements of the Society-in-the-Loop (SITL) model. The Social Contract illustrated in the SITL diagram is reflected, albeit in a limited manner, through Signpost’s operational process, which generates knowledge articles based on user inquiries. These questions serve as cues for article generation which are then ingested into the vector database, forming the knowledge base upon which the SPAI chatbot predominantly relies upon for its responses. Additionally, SPAI employs an inclusive design methodology to develop its technological tools and applications.
SPAI’s red-teaming and quality testing methodologies also introduce the “norms, values, expectations” of the humanitarian organization (IRC and Signpost), the sector, and moderators. This process includes incorporating into the RAG-based chatbot local and system prompts derived from moderator handbooks, protection officers’ insights, and the ethical guidelines of the organizations. These elements can be viewed as relevant mediating components of society. What has been described can be considered the negotiating factors of the social contract that inform the SPAI chatbot.
In theory, this aligns the SPAI chatbot with the SITL model over a longer time frame; however, in practice, it overlooks several key aspects.
For instance, SPAI’s inclusive design has limited inclusive elements; it currently lacks genuine two-way communication with users prior to the creation of articles or technologies. Their needs, wants, and expectations are broadly approximated in the existing system. True stakeholder buy-in as discussed within SITL requires more grounded interactional mechanisms.
The term “society” is also somewhat limited, as there is no actual negotiation occurring; rather, there is a retooling of established standards, regulations, and ethical and security guidelines, where conflicting interests and values are often resolved through expediency, pragmatism, and donor pressures. SPAI as an organization, lacks the resources and infrastructure necessary to facilitate negotiation and interaction among a diverse array of stakeholders.
Furthermore, red-teaming, quality evaluations, and retrieval-augmented generation (RAG) serve as filters over a large language model (LLM). SPAI's efforts are downstream and do not address all the inherent problems associated with black-boxed, often proprietary, for-profit LLMs. As such, these initiatives may fall short in completely resolving upstream issues. Verifying compliance will only be limited to the chatbot and its RAG architecture, it will not cover the opaqueness of LLMs generating first responses.
To establish an appropriate algorithmic social contract for LLMs, a humanitarian sector-wide mobilization will be required, which may not be sufficient compared to other sectors in effecting meaningful change in LLMs. A collaborative effort across multiple sectors including law, technology, government and civil society is necessary to effectively influence regulations, standards, legislation, and public sentiment.
As demonstrated, SITL presents a compelling concept that should not be dismissed solely due to the challenges of its operationalization. It can serve as a guiding principle for the future use of AI, not only at SPAI but within the humanitarian sector more broadly. The true value of SITL lies in its ability to illuminate and articulate all components of the algorithmic loop, which are often assumed, obscured, or overlooked.
There is merit in applying SITL thinking to explicitly frame all parts of an AI system’s loop, allowing us to fully grasp present and future challenges when it comes to effective governance.
Technological advancements, particularly in machine learning and generative AI, are progressing more rapidly than regulation, governance, and related policy adaptations can keep pace up with. In light of this, it is increasingly prudent to establish multi-stakeholder coalitions to work on stable institutional frameworks, mechanisms, and tools that can provide consistent guidance amid ongoing technological changes.
References
AI Winter: The Highs and Lows of Artificial Intelligence - History of Data Science
LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learn-ing. Nature, 521(7553):436–444.
Littman, M. L. (2015). Reinforcement learning improves behaviour from evaluative feedback. Nature, 521(7553):445–451.
Levy, S. (2010). The AI revolution is on. Wired.
Scientific discovery in the age of artificial intelligence | Nature
Sweeney, L. (2013). Discrimination in online ad delivery. Queue, 11(3):10.
'Black box' of generative AI: Importance of transparency and control
Caliskan, A., Bryson, J. J., and Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186
Boyd, D. and Crawford, K. (2012). Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon. Information, communication & society, 15(5):662–679]
Governance of artificial intelligence | Policy and Society | Oxford Academic
Signpost Data Privacy and Protection in the Age of AI. https://www.signpostai.org/airesearchhub/data-privacy-and-protection-in-the-age-of-ai
Society-in-the-Loop. Programming the Algorithmic Social… | by Iyad Rahwan | MIT MEDIA LAB | Medium
What is 'human-in-the-loop'? And why is it more important than ever?
Allen, J., Guinn, C. I., and Horvtz, E. (1999). Mixed-initiative interaction. IEEE Intelligent Systems and their Applications, 14(5):14–23.
Sheridan, T. B. (2006). Supervisory control. Handbook of Human Factors and Ergonomics, Third Edition, pages 1025–1052
Horvitz, E. (1999). Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 159–166. ACM
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S.,Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M.,et al. (2015). Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252
Cakmak, M., Chao, C., and Thomaz, A. L. (2010). Designing interactions for robot active learners. IEEE Transactions on Autonomous Mental Development, 2(2):108–118.
Society-in-the-loop: programming the algorithmic social contract | Ethics and Information Technology
SPAI: AI Chatbot Pilot: Using the AI Tool. https://www.signpostai.org/blog/signpost-ai-chatbot-pilot-using-the-ai-tool
[1707.07232] Society-in-the-Loop: Programming the Algorithmic Social Contract
Adaptive Autonomy in Human-on-the-Loop Vision-Based Robotics Systems | IEEE Conference Publication
J. E. Fischer, C. Greenhalgh, W. Jiang, S. D. Ramchurn, F. Wu, and T. Rodden, “In-the-loop or on-the-loop? interactional arrangements to support team coordination with a planning agent,” Concurrency and Computation: Practice and Experience, p. e4082, 2017.
Adaptive Autonomy in Human-on-the-Loop Vision-Based Robotics Systems | IEEE Conference Publication
Ibid.
Society-in-the-loop: programming the algorithmic social contract | Ethics and Information Technology
Rousseau, J.-J. (1762). The Social Contract
Society-in-the-loop: programming the algorithmic social contract | Ethics and Information Technology
Ibid.
Ibid.
Ibid.
SPAI: Mapping AI Design Principles. https://www.signpostai.org/airesearchhub/mapping-ai-design-principles
SPAI: Red Team Metrics, Scope of Work and Workflows https://www.signpostai.org/airesearchhub/documenting-signpost-ai-red-team-metrics-scope-of-work-and-workflows
SPAI Quality Experts Evaluate the Chatbot! https://www.signpostai.org/blog/hjb1trsrqtbza8iklrcz97ngdvz49t
