Extending the Soar Cognitive Architecture
Posted by Jeriaska on January 28th, 2009
One approach in pursuit of general intelligent agents has been to concentrate on the underlying cognitive architecture, of which Soar is a prime example. In the past, Soar has relied on a minimal number of architectural modules together with purely symbolic representations of knowledge. At the AGI-08 conference John Laird, Tishman Professor of Engineering at the University of Michigan, presented the cognitive architecture approach to general intelligence and the traditional, symbolic Soar architecture. This overview was followed by major additions to Soar, including non-symbolic representations, new learning mechanisms, and long-term memories
The following transcript of John Laird’s AGI-08 conference presentation entitled “Extending the Soar Cognitive Architecture” has not been approved by the speaker. Video is also available.
Extending the Soar Cognitive Architecture
I have been working on Soar for 25 years or more. The original system of Soar we developed in the early ’80s. Over the next ten years we made refinements to it, but at that time, ’95 or so, we had procedural memory encoded as rules, chunkiness learning mechanism, shared symbolic short-term memory, perception and action. At that point Soar somewhat stabilized.
Over the next ten years or so, we did a lot of work investigating what that architecture could do. One of the biggest things we did was create a virtual pilot that flew all U.S. missions and large-scale distributed simulations. This ended up being the largest Soar system ever developed and is currently at about 8,000o rules. It involved both using natural language to understand commands from others and also being able to generate language. It could fly itself or take commands from humans.
We also did research in learning from examples, learning from instructions, learning from what is called behavior cloning. This is similar to learning from imitation. However, about five years ago we got to the point where I felt we were running out of what we could do with the architecture as it were. I also ended up starting to think more about what capabilities could humans have that we did not have in the system already.
Fortunately, I am inspired by popular culture, and I even saw the movie Memento, which demonstrates what a complete cognitive cripple you are without episodic memory. Every AI system I knew, in terms of an AI agent, had that same lack of ability, so how do we expect our AI systems to behave well when they are missing a key capability? That led me to say that it was time to rethink the architecture and see what we needed to change, and what needed to be added.
Over the last three to five years, we have been making some extensions to Soar. The first one is, we added learning from external and internal rewards, which was reinforcement learning. That had become very popular in AI, and we saw how that would naturally fit into Soar. That was added in as a way to tune rules suggesting which operator to select. We then also based on our experience with collaborating with people in that community saw how important semantic knowledge and long-term semantic memory was, so we added long-term semantic memory. Returning back to watching movies and being inspired by those, we added episodic knowledge.
We also were starting to look at emotion. A lot of people think of emotion as something that you are trying to avoid. If you are doing a science fiction movie, emotion is something that you want to keep out of your AI system (or out of your Vulcan–whichever one you have). But it turns out that if you take emotion out of people, if you give them some sort of brain damage where they do not have emotion, it has a very negative effect on their performance. We went to try and understand what that effect was.
We ended up locking onto what is called appraisal theory, which is where people are continually appraising a situation relative to their goals, and other things such as novelty. We felt that was going to be a very important part of creating a generally intelligent system. We added, as part of our theory of emotion, what we call an appraisal detector, which actually hooks up to reinforcement learning and influences internal reward. We became familiar with work by Stephen Kosslyn on mental imagery, and became convinced how important mental imagery is to cognitive behavior–that it is not just pure symbolic all the time; there are really good computational reasons to create images and do processing over those images, and then being able to re-parse that image and notice new properties from that.
If I had a purely symbolic system, they would not be able to solve this problem everyone in this room can solve, which is “take the letter W–turn it upside down, and what do you get?” Everybody in the room should say “an M.” If you don’t have mental imagery, it is very difficult to understand how that would be possible. There are a lot of other reasons to do it, especially when you are going to be doing spacial reasoning. We added a visual imagery component. Not shown in this diagram is that it also has long-term memory associated with it. Finally, we wanted to understand how symbols arise. Where do symbols come from? Statistical regularities in the environment. So we are now studying the use of clustering as hooked up to perception.
Those are the extensions we have made to Soar. This is Soar’s processing cycle. I think in terms of control all the time when I am thinking about how a given system works—not just what is going on in the memory. The cycle is very straightforward. You take an input from the environment, then you try to evaluate that. We also suggest different operators, which is similar to intentions. What is the system thinking about what it can possibly do in this situation. It then can evaluate that situation and make a decision. That is what we call “selecting an operator.” Once it has selected the operator, it might have further evaluations on the operator. It might notice there are some things in the environment that are relevant to the operator. It then goes to apply the operator. That will either lead to internal changes in its short-term memory, or it will lead to actions that are taking place in the external environment.
All these yellow boxes are places where knowledge comes into play. In terms of Soar, the direct available knowledge is associative memory, although it is implemented as rules. The theoretical commitment is to an associative memory that is an asymmetric memory. You match something and you get something back which is different in the retrieval. Although there are a lot of people that do not like rule-based systems, think of this not as a rule-based system but as an associative memory system.
How do you bring associative memories to bear in making decisions? All these are firing rules in parallel, without any conflict resolution. All the decision making happens right here, based on the information you get here. One of the things that can happen is that the knowledge here is not complete or consistent. In Soar, you end up with an impasse. In the impasse, we create a subgoal, and then we can use the same kind of cycle to do more deliberate problem solving in the subgoal for any one of these aspects. If we are trying to select an operator and the system has rules that are inconsistent in some way, it goes into the subgoal and can do more deliberate reasoning. It can be things like a look-ahead search, an analogy, or whatever.
This is the way you integrate in metareasoning and all sorts of different methods into this very lean structure for problem solving. With the extensions we have made to Soar, we now have other sources of knowledge besides just rules and impasse-driven subgoals in order to make decisions. What can happen when we make an application of an operator, one of the things the operator can do is say, “I want to do a query to my episodic memory.” What is in my memory that is similar to this current situation? Or, what is similar in my semantic memory? What are some facts that are relevant to the current situation?
It also can manipulate visual imagery, so it can do commands to the visual imagery system and change what it is imagining. It can take an image, imagine something in that, and re-perceive the situation to detect new characteristics of that. All of those then can play a role in this more deliberative processing that then ends up either helping with selection or application of an operator. Of course, still riding over this are Soar’s existing learning mechanisms such as chunking. Once you have done this many times and have some confidence in it, you will change from doing deliberate problem solving to more automatic problem solving encoded as rules. That is the control side of things.
One of the cool things about this was that although we added all these new capabilities, we did not have to change the basic structure of Soar. If we go to our theoretical commitments, those that stay the same are: we still have the problem space computational model, we still have long-term and short-term memories, we still have associative procedural knowledge, a fixed procedure for deciding, impasse-driven reasoning, incremental experience-based learning and no task-specific knowledge.
What changed were what some people considered key aspects of the theory of Soar, but we were flexible on them. We now have multiple long-term memories, not just rules. We have multiple learning mechanisms, not just chunking. We have modality-specific representation and processing in terms of the image processing, so we are not just symbolic reasoning. We are doing lots of non-symbolic reasoning. A lot of systems do non-symbolic reasoning and they are not exactly hybrid systems. I would not call Soar a hybrid system. Soar does non-symbolic processing in terms of symbol-generation, control, learning control, intrinsic reward (done by appraisal), it has some working memory activation to help with memory retrieval, and does visual imagery, which is non-symbolic reasoning, but it is not doing any modeling of neurons.
Finally, some upcoming challenges are to continue the refinement and integration of these systems, then integrate them with complex perception and motor systems, and adding and learning lots of real world knowledge. I think one of the big challenges for general AI is how do you scale up? You can build small system—anybody can do it—but can you build systems that can still turn over and run fast as you add huge amounts of knowledge?
