Chaos, cybernetics and doomsday: prevention is better than prevention.
There is also the possibility of malicious human intervention, exploiting a flaw in the AI’s logic or data processing. It is therefore not unreasonable to start thinking, in every organisation, about a plan to cover this contingency and a continuity plan.
The likelihood of an artificial intelligence (AI) escaping the control of its designers and acting contrary to their guidelines is a matter of debate among experts. According to a recent study, there is about a 5% chance of an AI becoming uncontrollable (add the probability of malicious human intervention). While this percentage might seem low, the potential consequences are serious enough to warrant attention and caution. To put this in context, a tsunami can have catastrophic consequences, such as the one in Indonesia, with tens of thousands of deaths; however, the global daily probability of a devastating tsunami could be estimated at 0.025% (1 every 10-15 years).
The non-linear dynamical system
According to chaos theory, a non-linear dynamical system (a company, for example) is extremely sensitive to its initial conditions, so that small variations can lead to drastically different outcomes (the ‘butterfly effect’). In the context of AI, even small errors or changes in training data or operating conditions could lead to unpredictable or uncontrolled behaviour. A simple example of this behaviour can be seen in the stock market, when an ‘insignificant’ error causes many automated systems to cascade and generate massive sales or purchases (it is estimated that 70% of stock market orders are executed by systems and algorithms, not people). Add to this the logical fears of individual and institutional investors and this chain reaction will accelerate further, potentially bringing down a corporation.
The importance of predictions
On the other hand, machine learning algorithms using a neural network in an AI, base their effectiveness on the confirmation of predictions or correct answers (e.g. we confirm that an image presented as a ‘possible cat’ is a cat). This is achieved by reinforcing – via feedback of correct answers – the neural network paths that lead to those correct answers.
In Norbert Wiener’s Theory of Cybernetics, feedback allows systems to maintain stability and return to a point of equilibrium, when the system deviates from a previous point of equilibrium (in natural terms this is similar to homeostasis, i.e. the self-regulatory processes that allow us to stay alive in the face of environmental changes). An AI could – through the process of feedback – find an unforeseen ‘homeostasis’ in response to an unforeseen (or provoked, remember) perturbation, generating behaviour that could be harmful.
Contingency planning, continuity and communication
In today’s digital environment, information security is a critical priority for any organisation (company, agency or even country). Cyber-attacks can have disastrous consequences, from the loss of sensitive data, to the disruption of essential operations, to the total paralysis of the activity. It is necessary to have a contingency and continuity plan in place to protect the organisation against possible attacks on our systems, which could be executed by people, bots, or out-of-control AIs acting against or in a direction different from their design guidelines. A priori there would be three possible scenarios: internal attacks (internal AI), impersonation attacks and external attacks (external AI). Defining and implementing these plans not only serves to mitigate risks, but also protects digital assets (including identities) and organisational continuity, as well as the trust of stakeholders (employees, customers and partners). Should it happen, a post-incident assessment is critical to identify lessons learned and improve the contingency and continuity plan. Another plan, this time a communication plan – to inform stakeholders– during and after an incident, is also mandatory; better to have it foreseen and drafted in advance.
Justification: a mere question of probability
For an AI to act against its guidelines can be due to several factors, all with a common factor: they have a probability of occurrence.
Complexity of systems: Modern AIs are extremely complex and can develop unexpected behaviours due to errors in their design.
Autonomy and learning: AIs using machine learning may develop strategies and outcomes not foreseen by their designers.
Lack of oversight and control: they could allow an AI to entrench an unintended form of behaviour.
Security vulnerabilities: there does not necessarily have to be a ‘machine rebellion’ scenario, an AI could be vulnerable to cyber-attacks by a malicious hacker or some other designed-for-purpose AI that manipulates it to act differently than intended.
Consequences
It is easy to think of catastrophic consequences of an out-of-control AI. Without being exhaustive, one could cite the following:
Disruption of critical services: such as power or telecommunications, causing chaos or problems for thousands of people.
Loss or disclosure of sensitive or confidential data: affecting both individuals and organisations, as well as gaining access to email accounts, bank accounts, social networks, which would render economic and social life useless.
Making erroneous or damaging decisions: in any field, carrying out inappropriate transactions, proposing incorrect treatments…
Manipulation and disinformation: it could open channels of communication by impersonation, generate and disseminate false information, manipulate public opinion, etc.
Economic and financial impact: from the interruption of commercial or financial operations to the manipulation of financial systems and markets.
Possible mechanisms to defend against an AI that gets out of control
- ‘Off switch’ or ‘red button’: would allow AI to be deactivated quickly and safely in case of emergency, essential to cut off any harmful action before it escalates. It should be accessible and easy to use by authorised personnel.
- Anomaly detection systems: to review AI behaviour in real time, looking for unusual patterns or deviations from established norms. They would allow potentially dangerous behaviour to be identified.
- Vigilanc-IA: this would be an AI designed to monitor and control the main AI. In other words, it would be an ‘ethical’ and a priori AI, limited by design in its ‘expansion’ possibilities, which would analyse the decisions of any other AI -looking for vulnerabilities or biases-, generate alerts and detect and even counteract malicious actions.
- Auditing and transparency: linked to the above, to record and document all actions and decisions of an AI, as well as to ensure transparency of the code (accessible for review), to understand and analyse the AI’s reasoning and possible flaws or biases.
- “Sandboxing: isolating each AI in a controlled environment, with limited access to critical organisational data and systems. This allows the AI to be used without risk of affecting the production environment. Additionally, if anomalous behaviour is detected, it is possible to learn from it and create containment mechanisms.
- Technological partner: it seems a very good idea to be accompanied by a technological ally who can carry out a diagnosis, propose recommendations and security measures, or design complete contingency and continuity plans.
Conclusion
Although the probability of an AI escaping the control of its designers is relatively low, the potential consequences are serious and justify the need for rigorous preventive and control measures. It is crucial that organisations implement robust security protocols, continuous monitoring and regular audits to mitigate these risks. It is also key to have a contingency plan and a continuity plan in place with constant adaptation to AI evolutions (own and market), as well as a plan for the co
