DARPA Aims to Create Intelligence
This is that sort of dangerous way of thinking I'm always talking about... "let's create intelligence, assume it won't threaten us whatsoever, and figure out that morality thing later". Yes, DARPA will fail like they always have, but the mentality of trying for "intelligence" without considering the consequences is stupid. I don't care if you believe AGI will take 500 years -- we should approach the issue with the morality factor in mind, regardless.
The Physical Intelligence program aspires to understand intelligence as a physical phenomenon and to make the first demonstration of the principle in electronic and chemical systems. A central tenet is that intelligence spontaneously evolves as a consequence of thermodynamics in open systems. The program plan is organized around three interrelated task areas: (1) creating a theory (a mathematical formalism) and validating it in natural and engineered systems; (2) building the first human-engineered systems that display physical intelligence in the form of abiotic, self-organizing electronic and chemical systems; and (3) developing analytical tools to support the design and understanding of physically intelligent systems.
If successful, the program would launch a revolution of understanding across many fields of human endeavor, demonstrate the first intelligence engineered from first principles, create new classes of electronic, computational, and chemical systems, and create tools to engineer intelligent systems that match the problem/environment in which they will exist. Concepts relevant to the objectives of the Physical Intelligence program can be found in numerous disciplines and areas of research including statistical physics, non-equilibrium thermodynamics, dissipative systems, group theory, collective behavior, complexity theory, consciousness theory, non-linear dynamical systems, complex adaptive systems, systems analysis, multi-scale modeling, control systems, information theory, computation theory, topology, electronics, evolutionary computation, cellular automata, artificial life, origin of life, microbiology, evolutionary biology, evolutionary chemistry, neuropsychology, neurophysiology, brain modeling, organizational behavior, operations research and others.
I already understand that intelligence is a physical phenomenon. How could it be anything else? The universe is entirely physical. Only supernaturalists would argue otherwise. Is DARPA suggesting that intelligence could be aphysical?
This line is also really weird: "A central tenet is that intelligence spontaneously evolves as a consequence of thermodynamics in open systems." Then how come intelligence hasn't evolved in the exatonnes of open systems throughout the solar system, including the Sun and asteroids? And does the word "evolution" mean anything to anyone anymore? Biological evolution is an incredibly precise process, and nothing like what biologically-illiterate scientists mean by the word "evolution" when they carelessly use it.
The attitude presented by DARPA here is similar to that of 95% of government and private projects: we'll just work towards AI, and essentially ignore the risks.
May 8th, 2009 - 19:32
Source of the Darpa quote and notice of workshops:
https://www.fbo.gov/index?s=opportunity&mode=form&id=4539dc1cc3aa10b46cbf655811648137&tab=core&_cview=0&cck=1&au=&ck=
I’m strongly with the 95% here though it’s odd you think that “full torpedoes ahead” is a normal Gov. response. On the contrary fretting over imaginary concerns is far, far more likely to inhibit innovation than the reverse.
Risks are the last thing we want people to be focusing on for two important reasons. First, strong AI is a very long row to hoe and there will be plenty of time for tangential concerns. More importantly the (usually foolishly exaggerated) risk chatter provides great fodder for AI research detractors who are already challenging budgets that are currently inadequate to the task.
May 8th, 2009 - 19:57
The write up on wired is idiotic too.
wired.com/dangerroom/2009/05/darpa-heat-energy-brains-now-make-us-some/
Whoever wrote the stuff about thermodynamics sounds seriously confused. Of course you can use thermodynamics to explain all aspects of the world. Using the words “intelligence”, “evolution” and “thermodynamics” in the same sentence is pure mysticism. There is no purpose to it except to sound good to laymen. It’s the same as people who promulgate the “law of attraction” by invoking “vibrating energy” and quantum mechanics.
May 8th, 2009 - 22:00
Joe, you’re confusing the quote “damn the torpedoes” with “full speed ahead”.
“Full speed ahead” is not the usual government response. But it is with respect to AI, because they don’t understand nor consider the risks of human-equivalent AI.
Maybe AI is a long furrow to plow, but it’s best to start the process with the risks in mind. And if there’s a 10% chance that AI is not as far as we think, it would still be foolish to ignore the risks. Even if the chance is only 0.01%, it would still be unacceptable.
Good thing that concerns over the risk of human-equivalent AI aren’t imaginary in the least.
AI research detractors never argue that AI would be a risk. Instead, they argue that AI is certain to fail. Though I wouldn’t mind giving them fodder to slow down idiotic projects that pursue general intelligence without any consideration of the consequences.
May 9th, 2009 - 01:25
I seem to recall that DARPA has tried this before with something on the order of a 50-million dollar project having something to do with AGI. This was canceled some time ago.
I do not believe that DARPA will cause any harm with this one, it appears to be a benign project, although a little unsettling. My attitude on DARPA is: AI Funding + DARPA Researching + Project Management = Garbage + Money Lost
Sorry DARPA, I don’t think you have what it takes to work on AGI.
May 9th, 2009 - 02:51
I don’t feel this particular project poses a dire threat to our continued existence. The press release indicates that DARPA intend to pull intelligence out of a hat woven from a thicket of confused, unrelated ideas (I mean, topology… wtf? And is there really any such discipline as “consciousness theory”?). I don’t think the people behind this are a tenth of the way to being intellectually prepared for an attack on the problem of intelligence.
May 9th, 2009 - 03:16
I don’t think it’s necessary to be so harsh. They are just trying to generate interest in the project and they may well be more aware of the risks than you realize. Be aware that this one small article doesn’t necessarily do the whole project justice and the media certainly doesn’t speak particularly well on behalf of scientists/facts.
May 9th, 2009 - 05:28
They have a chance to do something. Another contender, possibly the strongest at the moment.
May 9th, 2009 - 06:19
I’m not sure I get the hostility either. This document alone doesn’t seem like reason to get worked up.
As for calling it physical intelligence, I think that’s acceptable. There are ways of understanding aspects of intelligence that are neither wholly physical nor supernatural, such as psychological interpretations. The use of language here seems to be about dispelling confusion through specificity – not creating it.
As with any work of this kind I’m sure those of us with our eye on the intelligence problem would love to hear more about exactly what is proposed. Until we do, however, forming negative judgements seems like jumping the gun.
May 9th, 2009 - 07:34
Felix, that’s the DARPA proposal web page, it’s not an article written by a journalist. Did you even click the link?
Ben, saying you’re creating AI and not making an issue of its motivations is idiotic and deserving of ridicule. It doesn’t matter if we have a 99.9% estimation that they’ll fail.
May 9th, 2009 - 08:20
I think, they will not fail on the long run. The question is only, who will be the first there, more importantly with what motivations linked to that first SAI.
May 9th, 2009 - 16:03
Michael,
Given the socio-political environment from whence DARPA’s funding originates, are you seriously going to predicate your objections on the wording used to request technical input regarding the current state of AI research? Or that they choose to structure that request within existing .mil structural constraints and applications? And, is it your position that should the answer recieved prove to be “No” that DARPA will have preforce “fail”ed?
Might I suggest that you reserve your ire for an actual solicitation for bid offering and see what this solicitation for attendance at a seminar results in? Not everyone assumes that AI=god-like intelligence and a goodly portion of the consequent ethical debate is being ignored and unregarded as a result (and, no, your previous dismissal of the questions doesn’t actually constitute “debate”).
When it comes right down to it, I actually agree with you that DARPA is not going to achieve a physical product from this solicitation – but more because our understanding of the physical structure and mechanisms of human intelligence remains insufficient for us to emulate it successfully as of yet than from any lack of sufficient consideration of the potential for ultimate threat such a result might offer.
My own view on this topic is here and my offer to discuss/debate the general topic of singularity and/or AGI via our respective blog’s remains open. Frankly, I expect that I will learn much more from such an exchange but that what little I might impart in return will have much greater effect on the eventual outcome.
May 9th, 2009 - 19:47
Hi William,
Thanks for your comment. I’m afraid your blog post on the matter only has a paragraph or two of original content. I might be open to discuss/debate the Singularity and AGI, but could you perhaps make a longer blog post on your general position, i.e., whether you consider the motivations of the first human-equivalent AI to be an issue and when you think AI might be developed? I’d also ask that you read one paper, “Basic AI Drives” by Stephen Omohundro.
May 9th, 2009 - 23:22
I just flamed this project, for the inanity of its guiding principle, over at Wired. And see the FAQ:
“Q. May we propose organizational forces other than thermodynamics to use in our implementations?
“A. No.”
I guess I can think of maybe one way to make this work well enough to be dangerous: If you had a scalable general problem solver whose basic principle was inspired by thermodynamics, e.g. an adiabatic quantum computer. If you not only had a “thermodynamic problem solver”, but were able to scale it to the point of performing truly complex tasks, then you would be getting into that danger zone where you have a passive superintelligence capable of being directed towards arbitrary ends.
May 10th, 2009 - 09:24
WIRED looks bad just by posting this project with a basically straight face.
May 10th, 2009 - 09:46
Will all the non-scientists (Michael and cronies) stop posting about topics they are blithly unaware of. As for deserving of ridicule the only ones who are would be Michael and his cronies for speaking in ignorance. I happen to have worked with DARPA so if you have some legitimate critism based on experience then fine otherwise save it. Now everyone the world over join me in ridiculing Michael and his cronies.
May 10th, 2009 - 10:31
DARPA is not a football team, you don’t have to take their side. They launch a lot of projects, both good ones, like their arm prosthetics, and bad ones, like this one. The organizing principle behind this project is a joke, and I eagerly await continuing to make fun of it as more details emerge.
May 10th, 2009 - 14:03
@ Saul:
“I don’t think the people behind this are a tenth of the way to being intellectually prepared for an attack on the problem of intelligence.”
Agreed.
May 10th, 2009 - 14:07
@MoronCrusher
“Will all the non-scientists (Michael and cronies)”
– Sorry, I’m a real scientist and I agree with Michael.
May 10th, 2009 - 18:14
Michael,
I’m not certain either of us is quite ready for this, but here goes nothin’.
May 11th, 2009 - 10:48
Even if the chance is only 0.01%, it would still be unacceptable.
Thanks for all these thoughtful replies – kudos to your blogging style!
You mean the chance of unfriendly AI posing existential risk to humanity? I think .01% and probably even 1% is a spectacularly low and acceptable risk for DARPA or other groups given that we are starting to see competition for the first AI.
I’m not thrilled it may come from a military project, but I’d sure rather it come from USA military than an “unfriendly military AI”. Also, the DARPA project appears to be open enough to suggest their AI goals are remarkably broad and open.
As you know much better than most, funding will play a key role in the development of strong AI. I can think of no projects as deserving as strong AI since major developments there will likely solve most other pressing human problems quickly.
As a leader in the field I hope you seriously consider how focusing policy makers on unlikely existential risks might inhibit funding and the race to strong AI.
May 11th, 2009 - 11:15
Hi Joe!
I mean the chance of this specific project actually reaching strong AI, and my number is actually an exaggeration — I don’t think this project has even a 0.1% chance.
The point I’m trying to make here is if the programmers don’t put a major emphasis on the morality/motivations aspect of programming, I see a very high probability of disaster, as we shouldn’t expect them to get it right by chance. So say I give them a 0.001% chance of successfully building human-equivalent AI capable of recursive self-improvement. Given that, I’d give a 99% or greater chance that the project destroys the world, because of the AI pursuing some utility function that leads to indifference towards human beings. (Assuming they don’t start realizing midway that the morality component won’t take care of itself and basic AI drives can be very dangerous to human welfare.)
My hope is that the learning process will cause any project that is actually approaching strong AI to take steps to ensure that it has human-friendly motivations, but that’s just a hope.
I don’t care where AI comes from as long as it’s friendly to humans and can preserve that friendliness over multiple rounds of self-reprogramming, but it seems like the incentive structure for a military-designed AI project might be unhelpful for the human-friendliness factor. Sure, a USA military AI project might be better than some other military AI projects, but I’d rather strong AI be developed by a company, or better yet, a non-profit, and most of all, be the result of careful paranoid researchers who understand that a system should be built with recursive self-improvement in mind.
Yes, strong AI, if carefully built, could probably solve human problems more quickly than any other approach. But I worry that moral realist theories will cause programmers to just assume that a sufficiently intelligent AI will discover kindness and morality automatically, no special programming required. Right now, that is probably the dominant and intuitive position, so if any randomly selected AI project is successful, I view it as a negative outcome for humanity.
In my view where moral realism causes the majority of AI programmers to be reckless and lazy when it comes to instilling an AI with human-friendly motivations, most strong AI projects should be discouraged. Only those projects that place a very high priority on “morality engineering” (need a better term) and the rejection of moral realism would get my support.
Thankfully, however, it doesn’t seem like this project is a genuine push towards strong AI. It’s hard to figure out what exactly it wants to be, honestly.
May 12th, 2009 - 07:58
“DARPA will fail like they always have”
One of the most stupid things you’ve ever written, Michael. Congratulations.
May 12th, 2009 - 08:50
Fail at AGI, not at everything. Man, you’d think I was insulting someone’s mother when I say anything negative about DARPA. Don’t forget that the main point of DARPA is to help our big group of “defenders” (military) kill other people. Necessary, but nothing to throw your hat in the air about.
January 13th, 2011 - 20:33
Your criticism of this project on grounds that it may be dangerous is spot on, however your criticisms of the science is deeply flawed.
What DARPA means by ‘thermodynamics’ is not the typical thermodynamics, but rather something closer to information theory. The science behind this is sound.
If you want to learn about the actual nature of this project, I suggest you read about how self organized criticality, spontaneous symmetry breaking, pattern formation and neural networks are related to living systems in the context of complexity theory. No doubt you already know some or all of this stuff.
The ultimate goal of this project is to build a neural network out of self assembling nanoscale materials, which at this stage of technological development is actually becoming feasible, and could very well happen faster than you think. After all, we are getting closer to the singularity, linear thinking does not work here.