Governing Lethal Behavior
Posted by Jeriaska on May 21st, 2008
At AGI-08: The First Conference on Artificial General Intelligence, Ronald Arkin presented the second of three talks on the theory and formalisms for the implementation of an ethical control and reasoning system potentially suitable for constraining lethal actions in an autonomous robotic system. These controls are implemented so that they fall within the bounds prescribed by the Laws of War and Rules of Engagement, based upon extensions to existing deliberative/reactive autonomous robotic architectures. Part two of the series focused on Formalization of Ethical Control.
The following transcript of Ronald Arkin’s AGI-08 presentation “Governing Lethal Behavior: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture – Formalization for Ethical Control” has been corrected and approved by the speaker. Video is also available.
Governing Lethal Behavior
Part One of this talk will be delivered in two weeks at the Human Robot Interaction Conference. Part Three has already been delivered at the Technology in Wartime Conference at Stanford, so I have been going backwards in time. If you really want to understand this, you are not going to get it from this talk because I don’t have enough time to do it. There is a 120-page technical report available on our website. I will be able to tell you a little bit of the why, a little bit of the what, and almost nothing of the how, in terms of what I am accomplishing here in this project, which is funded by the Army Research Organization. It has gotten a lot of traction in the press of late for either good or bad reasons.
One of the things you are going to have to be aware of is that autonomous systems are working their way out toward the battlefield. There are a number of already fielded systems or potentially very short-term fielded systems which are going to have impacts in the real world very soon. There is a South Korean robotic platform, which has been designed by SamsungTechwin. I don’t know who does your HDTV, but they have a system which is capable of autonomously identifying and engaging targets, deployable in the demilitarized zone in Korea.
The makers of your favorite home vacuum cleaner, Roomba, also makes the PackBot, which has been equipped with tasers. Non lethal weapons are just less lethal weapons, by the way. There is no such thing as a non lethal weapon. They also, I believe, have a weaponized version. The SWORDS by Foster-Miller is already in Afghanistan and Iraq. It has not fired a shot yet, to my knowledge, but it is a fully equipped machine gun, grenade launching robot. The MQ-9 Reaper, appropriately named, carries ten times the ordnance of the Predator, which you have already seen in action.
I am amazed by the number of systems that are rolling out on a continuous basis—this is part of the quick “why.” These systems, I firmly believe, are going to sneak their way up in terms of autonomy in their ability to use and designate targets. There is a whole bunch of reasons for this, which have to do with battlefield tempo—read the papers. I don’t have time to tell you about it. But there are many systems already. Even anti-tank mines or anti-personnel mines can be considered at some level an autonomous robot, because they sense and they actuate in a perhaps not particularly desirable way.
I have spent 25 years dealing with the military on autonomous systems, and we have come a long way, for better or for worse, in terms of the ability to field these systems relatively soon. The argument is—Is the human in the loop? In the air force, some people always say there will be a human in the loop. I’m not exactly sure what that means. This autonomy is sneaking up gradually, even according to the military charts. The air force has used the phrase “a human on the loop.” What does that mean? What’s next—the humans out of the loop to some extent.
Even in some cases, the cognitive abilities of humans to be able to process when that light comes on, should I fire or not? As evidenced, for example, by the Patriot missile batteries in the Kuwaiti engagement that we had many years ago. Anecdotally, there had never been a case where someone did not push the button when the light came on. There is just not enough time to think about it, from a cognitive perspective.
It appears, for all practical purposes, that this is going to happen. The underlying thesis of this work is that not only can we potentially use these systems in the battlefield, we have to examine how we are going to use them. It is my contention that ultimately we can do better than human beings in the battlefield. I will try and convince you that it is a very low bar, and that is quite unfortunate, to be honest with you, in this set of circumstances. Also, I make no claims that the systems will be perfectly ethical in the battlefield. My claim is they will perform more ethically than humans currently do. That is the benchmark that we are doing. In so doing, we will be able to save non-combatant lives, among others, hopefully without eroding mission effectiveness.
This is an amazing report. It is online. It was the first ethical report done by the Surgeon General in the history of the military, published in 2005, from returning soldiers from Operation Iraqi Freedom. It examined and created a benchmark for the ethical performance of our soldiers. 45% of soldiers and 60% of marines did not agree that they would report a fellow soldier or marine if he had injured or killed an innocent non-combatant. They are not going to ask the question, “Did you kill or injure an innocent non-combatant?” It goes on and on. Just reading the summary section, which this is reproduced from is very disheartening.
I have great respect for our colleagues in uniform, and they are doing the best that they can. But I used to think that these violations were just an isolated point. They are not. These violations seem to be endemic, due especially to the ever-increasing tempo of warfare. As such, something needs to be done to enhance the ways in which we can co duct warfare. If we are stupid enough to continue to kill ourselves in the battlefield, we should do it ethically.
We have spent hundreds of years, and I don’t have time to talk about this, engaging from St. Augustine on in just war theory, jus ad bello and jus ad bellum, in the ways in which we are allowed to kill each other. I had lectures at one of the first workshops in Roboethics to be told the right caliber of bullet to shoot somebody with. You shoot them with big, fat, slow bullets and not high speed small bullets. The reason for that is to reduce unnecessary suffering and humanity. There are all sorts of things that you probably don’t really want to know about. But if you do, read the technical report.
There are a variety of reasons why I would like to see our systems using this. Again, forgive me for rushing through this—this talk is usually an hour. There is no inherent right to self-defense for an autonomous system in the battlefield. They can do things in a way which could be potentially self-sacrificing. They may work in pairs, for example, to disambiguate. Sometimes the issue of cost comes up in that, but you can bypass that to some extent by recognizing that even intelligent hand grenades, which could have been rolled potentially through the doors of Haditha, might be able to reserve the decision until additional information had arrived.
I have worked extensively in the building of emotions into these robotic systems for Sony, as well as other applications, including the military. The question is the appropriate use of emotional baggage. The issue is, we would like to use guilt and remorse, oddly enough, as an adaptive effective function in the case that violations of military conventions arise. With the advent of network-centric warfare—the coming of the global information grid, and the like, all these new technologies are going to be able to deliver far more information than any person could possibly assimilate in a reasonable amount of time, but it is my contention that a robot potentially could.
Also, the third-party view, where their mere presence of being able to observe in the battlefield the performance not only of themselves but others as well, monitoring for ethical infractions will have a very interesting effect perhaps on squad cohesion, perhaps on the aspects of the ways in which tactics are conducted, and a variety of other things as well. There are lots of arguments from philosophers and others who think that from the worst case it just cannot be done, but this is the goal: to incorporate the rules of engagement. It is not appropriate, in my estimation, for a robot to independently develop its own moral sense of agency in the battlefield. You don’t hand a solider a gun and say, “Go out there and try and figure it out. Do what you think is best.” These laws have been appropriately encoded since around 1836, and the rules of engagement are developed by lawyers. How can we encode those appropriately to do that?
These are the techniques that we would like to use, making sure that we have an obligation to kill. The principle of double intention ensures that minimization of collateral damage, including civilian property and civilian live, is used. New tactics, using escalating force, so that you do not necessarily shoot first and ask questions later. Indeed, the underlying principle, strangely enough is to first do no harm under these sets of circumstances, and to assume that the system is dealing with non-combatants until it reaches a certain threshold of confidence.
These are the underlying principles of just war. There are actually pseudo-coded algorithms in that tech report for being able to do this. We are building the system as we speak, and the four mechanisms by which we are doing it, one is called the “Ethical Governor.”
The Ethical Governor deals with the encoding and it throttles the overt behavior of the system using formalisms that were described in my textbook a number of years ago, to transform that into a form which yields, at the very least, permissible behavior. Recognizing the use of obligations and the avoidance of violating prohibitions, ethical behavioral control deals specifically with the underlying mechanism for designing these systems. The ethical adapter–as I mentioned before, these systems will not be perfect–the question is, How do they automatically recognize when something has gone wrong, either through after-action review or through the use of affect when they are in real-time when they are in the battlefield? And the responsibility advisor, which is a crucial apect for justification of potential deleterious actions.
You guys in AGI may argue that the robots should be given its own moral agency, can be given its own moral agency, and you can blame the robot. I’m not at that point yet. It is going to take a lot of convincing to get to that particular point. I have given this talk to philosophers and the like, and I will be again later this summer.
There is the hybrid deliberative architecture. Those are the four different components. I know that does not make a whole lot of sense at this point. I cannot talk about the formalisms that are used here, although that is what this paper contains, so please review that if you would like. You might want to read Part One first, which is available on the internet.
We are planning on testing this in four different scenarios based on real-world situations in some cases. The first one deals with 190 Taliban, which were in a cemetery. The second one deals with a violation of the Geneva Convention in my estimation of an Apache helicopter video. The bottom line for this is that we need to as a community recognize our responsibility in managing the technology. You are creating an AGI, we need to understand it and be able to manage it in effective ways. There is also a survey of public opinion on that. In conclusion, there is more information.
