Moral Machines Blog Tuesday, Jun 23 2009 

Wendell Wallach and Colin Allen, authors of Moral Machines: Teaching Robots Right From Wrong, a fascinating look at morality in machines, have a blog on the topic that I wasn’t aware of until recently.

You should definitely check out their book. It’s relatively brief and analyzes many important issues around how one might go about building machines with a sense of morality. Wendell made a post where he praised my recent project Preventing Skynet and made a call for closer interaction between two communities in machine morality:

Our friend Michael Anissimov has, together with others, initiated a new blog, “Terminator Salvation: Preventing Skynet: Just say ‘no’ to genocidal artificial intelligence!” We applaud this effort and encourage members of the machine morality, machine ethics, and roboethics community to contribute to the blog. There has been a kind of split into two communities, with only a little cross-over, between those focused around future ethical challenges posed by a possible Singularity and those whose attention is directed at more immediately challenges and the implementation of moral decision making in present or near-future technology. I’d like to propose that we make efforts to bridge this gap, and will have more to say about that in a future posting.

I do agree, though the two groups may have a few inherent differences. Upon reading Wallach’s book, however, I think many of these differences may superficial in some instances. Our group is particularly concerned with the question, “how do we create a self-improving AI that we can trust not to kill us as it becomes more powerful than the entire human race?” The machine morality, machine ethics, and roboethics communities are interested in the more general question, “how do we build machines that we can trust to make moral decisions?” Many of the ideas from the latter camp could be useful for the goals of former, and vice versa.

The Singularity Institute (SIAI) is taking steps to integrate itself more seriously with the mainstream roboethics community. In just a week, SIAI employees Anna Salamon and Steve Rayhawk and SIAI associate Carl Shulman will be presenting papers at the European Conference at Computing and Philosophy. One of the papers is based on the project we did last summer as part of the SIAI internship program, Uncertain Future. I am currently visiting the current SIAI summer program and recently got to see preliminary presentations of both papers. Thanks to the support of SIAI’s generous donors, I’m sure we’ll be seeing more SIAI representatives at academic conferences on computing, philosophy, and roboethics.

Musician AI Wednesday, Jun 10 2009 

Here’s an interesting application of AI — to create “elevator music” that is original and interesting instead of boring and repetitive.

Sander Olson Interview with Ben Goertzel Wednesday, Jun 10 2009 

Cross-posted from SIAI blog (where I am now the primary poster), an interview with SIAI Director of Research Ben Goertzel.

Tesla Personal Supercomputer Tuesday, Jun 9 2009 

Another product called “Tesla”, Nvidia is selling supercomputers up to 250 times faster than standard PCs and workstations for just $10,000.

I’d prefer if this sort of product weren’t around. To quote this article:

Moore’s Law does make it easier to develop AI without understanding what you’re doing, but that’s not a good thing. Moore’s Law gradually lowers the difficulty of building AI, but it doesn’t make Friendly AI any easier. Friendly AI has nothing to do with hardware; it is a question of understanding. Once you have just enough computing power that someone can build AI if they know exactly what they’re doing, Moore’s Law is no longer your friend. Moore’s Law is slowly weakening the shield that prevents us from messing around with AI before we really understand intelligence. Eventually that barrier will go down, and if we haven’t mastered the art of Friendly AI by that time, we’re in very serious trouble. Moore’s Law is the countdown and it is ticking away. Moore’s Law is the enemy.

H/t to Jan-Willem Bats.

Computing Power Does Matter for AI Sunday, Jun 7 2009 

A frequently mentioned reason for the likelihood of human-equivalent AI being created within decades rather than longer is the fact that affordable computing power is approaching most estimates of human brain processing power.

100 billion neurons firing at 200 Hz — this is a basic neurological fact. Yes, there are many additional shades of complexity, including dendritic spines, neurotransmitter concentrations, and so on. Still, all of these put together seem to change the estimated computational requirements by no more than 2-3 orders of magnitude.

I can tell that I am speaking with an ideologue when they are unaware of the facts mentioned above, are informed of them, but that information then has no impact whatsoever on their subjective probability estimates of human-equivalent AI being created in the next few decades. Many people seem to act as if computing power has no influence whatsoever.

In contrast, Ray Kurzweil, Hans Moravec, and some other advocates of strong AI have seemingly acted as if computing power is everything — that when we have human-equivalent computing power, we’ll immediately have human-equivalent AI. That is wrong too.

It is easy to take the middle path. Particularly when the notion of human-equivalent computing power being available is combined with neural data from extremely high-resolution brain scans (a brute force argument for the eventual plausibility of human-equivalent AI if there ever was one), critics begin to sound incredulous when they do not revise their probability estimates for AI whatsoever.

One particular confused meme that has been making the rounds for decades is the notion that some fundamental breakthrough in computing would be necessary to implement human-equivalent AI. A digital computer can simulate any possible analog signal, as long as it has the computing power — the inverse is not true. This is proven thousands or millions of times daily as old VHS and other magnetic tapes are converted into the digital medium.

If I had a computer faster than most expert estimates of human brain computing power and an extremely high resolution scan of the human brain, the burden of proof would be on the critics to say why I couldn’t create a human-equivalent AI immediately. The objections here tend to circulate around dualism, mysticism, biology-worship, quantum mumbo-jumbo, etc.

Letter to Cyberspace Policy Review from DailyKos Blogger Tuesday, Jun 2 2009 

Here’s an oddly worded and frantic letter to Melissa Hathaway, Cybersecurity Chief at the National Security Council, by some DailyKos blogger:

The Cyberspace Policy Review neglects to consider the range of threats from the impending instantiation of Artificial [General] Intelligence (”AGI”), and the emerging proliferation of practical Brain/Computer Interface (”BCI”) technology in mass-market consumer electronics devices [such as the $300 Emotiv EPOC].

These developments, which are no longer speculative, and not merely inevitable but which are actually already in the process of happening in realtime, represent far greater danger to human life than the sum total of all other cybersecurity threats, combined.

The rapid - and exponentially accelerating - development of AGI and BCI technologies will lead to a cognitive convergence of Man and Machine, probably before the end of this Administration’s term in office. Technologies which enable full-scale human brain emulation in Silicon, which, in 2005 were predicted by Kurzweil to evolve by 2048, were believed by technology insiders - as of last October - as likely to become available by 2018, and perhaps even sooner.

Which “technology insiders”? Why is “Silicon” capitalized? Some odd thinking in this letter, although I agree that AGI/BCI “represent far greater danger to human life than the sum total of all other cybersecurity threats, combined.”

Intelligence Realm — Distributed AI for Automated Research Thursday, May 28 2009 

An interesting new distributed AI project has been covered by Singularity Hub.

Call for Short Essays — “Preventing Skynet” Monday, May 18 2009 

I’m thinking of doing a small website project for the upcoming movie Terminator: Salvation. The concept is somewhat similar to “Three Laws Unsafe”. I want to call it “Preventing Skynet”. The movie is just coming out in three days, so we’d be late with the site. Hopefully it would go up within two weeks if I receive enough material.

I am soliciting short essays that discuss basic Friendly AI/Singularity concepts:

- why is morality not automatic?
- why could human-equivalent AI be powerful?
- why should I care about AI? The human brain is too complex to duplicate for hundreds of years.

I would suggest including a superficial mention of the Terminator franchise. You don’t need to see the new movie, just see a couple of the older ones.

“Short” essays as in 1,000-2,000 words. Everything should be kept simple, with your educated, geeky, easily distracted target audience in mind. For examples of what I’m looking for, see:

http://www.singinst.org/blog/2007/06/06/tragic-assumptions-about-greater-intelligence-as-depicted-by-termites/
http://www.asimovlaws.com/articles/archives/2004/07/robot_oppressio.html
http://www.asimovlaws.com/articles/archives/2004/07/too_simple_to_b.html
http://www.asimovlaws.com/articles/archives/2004/07/why_we_need_fri_1.html
http://www.asimovlaws.com/articles/archives/2004/07/unsafe_at_any_l.html
http://www.acceleratingfuture.com/michael/blog/2007/06/the-rapids-of-progress-by-mitchell-howe/
http://www.acceleratingfuture.com/michael/blog/2007/04/investing-in-immortality-by-mitchell-howe/

Please consider contributing! Email your essay to me. Even if your essay is a rant, consider submitting it, because I might be able to edit it. Anonymous essays are accepted.

The Danger of AI Part 2 Saturday, May 9 2009 

Two pieces of writing that greatly influence my thinking on AI are “Basic AI Drives” by Stephen Omohundro and “Levels of Organization in General Intelligence” by Eliezer Yudkowsky, particularly Part 3: Seed AI. Basically, the first argues that AIs will find convergent goals that are antagonistic to humans, and the second argues that a hard takeoff is likely. Together, they have the implication that a hard takeoff of a human-equivalent AI that is not impeccably programmed would kill us all. (Another supporting document is “Ethical Issues in Advanced Artificial Intelligence” by Dr. Nick Bostrom.) Yes, I know you may not share this view, but I seek to impart it upon you by referring you to these writings. Here they are, in large screenshot form:

Yes, that’s about it. That’s the primary argument. Another important point are Kurzweil’s books, which argue that AI is plausible by 2030 by brain scanning alone.

I am interested in support or criticisms, but hopefully criticisms that actually involve having read the documents in question, or support likewise.

DARPA Aims to Create Intelligence Friday, May 8 2009 

This is that sort of dangerous way of thinking (changed from “thing”) I’m always talking about… “let’s create intelligence, assume it won’t threaten us whatsoever, and figure out that morality thing later”. Yes, DARPA will fail like they always have, but the mentality of trying for “intelligence” without considering the consequences is stupid. I don’t care if you believe AGI will take 500 years — we should approach the issue with the morality factor in mind, regardless.

The Physical Intelligence program aspires to understand intelligence as a physical phenomenon and to make the first demonstration of the principle in electronic and chemical systems. A central tenet is that intelligence spontaneously evolves as a consequence of thermodynamics in open systems. The program plan is organized around three interrelated task areas: (1) creating a theory (a mathematical formalism) and validating it in natural and engineered systems; (2) building the first human-engineered systems that display physical intelligence in the form of abiotic, self-organizing electronic and chemical systems; and (3) developing analytical tools to support the design and understanding of physically intelligent systems.

If successful, the program would launch a revolution of understanding across many fields of human endeavor, demonstrate the first intelligence engineered from first principles, create new classes of electronic, computational, and chemical systems, and create tools to engineer intelligent systems that match the problem/environment in which they will exist. Concepts relevant to the objectives of the Physical Intelligence program can be found in numerous disciplines and areas of research including statistical physics, non-equilibrium thermodynamics, dissipative systems, group theory, collective behavior, complexity theory, consciousness theory, non-linear dynamical systems, complex adaptive systems, systems analysis, multi-scale modeling, control systems, information theory, computation theory, topology, electronics, evolutionary computation, cellular automata, artificial life, origin of life, microbiology, evolutionary biology, evolutionary chemistry, neuropsychology, neurophysiology, brain modeling, organizational behavior, operations research and others.

I already understand that intelligence is a physical phenomenon. How could it be anything else? The universe is entirely physical. Only supernaturalists would argue otherwise. Is DARPA suggesting that intelligence could be aphysical?

This line is also really weird: “A central tenet is that intelligence spontaneously evolves as a consequence of thermodynamics in open systems.” Then how come intelligence hasn’t evolved in the exatonnes of open systems throughout the solar system, including the Sun and asteroids? And does the word “evolution” mean anything to anyone anymore? Biological evolution is an incredibly precise process, and nothing like what biologically-illiterate scientists mean by the word “evolution” when they carelessly use it.

The attitude presented by DARPA here is similar to that of 95% of government and private projects: we’ll just work towards AI, and essentially ignore the risks.

Thinking About Thinkism Saturday, May 2 2009 

Last September, Kevin Kelly posted a critique of a hard takeoff Singularity, based on what he calls “thinkism”:

As an essay called Why Work Toward the Singularity lets slip: “Even humans could probably solve those difficulties given hundreds of years to think about it.” In this approach one only has to think about problems smartly enough to solve them. I call that “thinkism.”

Let’s take curing cancer or prolonging longevity. These are problems that thinking along cannot solve. No amount of thinkism will discover how the cell ages, or how telomeres fall off. No intelligence, no matter how super duper, can figure out how human body works simply by reading all the known scientific literature in the world and then contemplating it. No super AI can simply think about all the current and past nuclear fission experiments and then come up with working nuclear fusion in a day. Between not knowing how things work and knowing how they work is a lot more than thinkism. There are tons of experiments in the real world which yields tons and tons of data that will be required to form the correct working hypothesis. Thinking about the potential data will not yield the correct data. Thinking is only part of science; maybe even a small part. We don’t have enough proper data to come close to solving the death problem. And in the case of living organisms, most of these experiments take calendar time. They take years, or months, or at least days, to get results. Thinkism may be instant for a super AI, but experimental results are not instant.

Interesting argument, and well-phrased.

But what about that recent story of a Cornell researcher Hod Lipson’s AI program that independently derived the laws of motion based on the swings of a pendulum, “a feat that took physicists centuries to complete”? According to Kelly’s thinkism hypothesis, that should be impossible.

And Lipson’s program is just the start of a whole field:

The research is being heralded as a potential breakthrough for science in the Petabyte Age, where computers try to find regularities in massive datasets that are too big and complex for the human mind. (See Wired magazine’s July 2008 cover story on “The End of Science.”)

Surely intelligence can achieve a lot. Solutions which take certain thinkers years to discover are uncovered in a short period of time by gifted experts. It is wrong to place solid limits on what a superior intelligence could do — could chimps predict what humans would be capable of with thinking alone? Of course, I could be wrong. Any intelligence might require extensive experimentation to generate knowledge. But the difference between my and Kelly’s position is that he sets hard limits based on speculating about intelligence fundamentally different than his own, while I acknowledge my basic uncertainty and say that I’m not entirely sure what thought alone is capable of. It could be capable of nearly everything, or it could only be capable of what it has achieved so far (a lot).

The conservative stance would be to assume that a new mind might be able to benefit from thought a great deal, and as such we should take great pains to ensure that all artificial intelligences above a certain level have human-friendly motivations. The “we have nothing to worry about, I guarantee it” stance would be to assume that thought is practically useless without thorough experimentation and to ignore the issue of human-friendly motivations on the pretense of indefinite human superiority and control. Sounds like the premise for yet another science fiction film where AIs get the jump on humans because we were overconfident.

Even if experimentation were required to glean knowledge, why would such experimentation be limited by the anthropocentric designation of “calendar time”? A nanoscale pendulum swinging in a vacuum demonstrate the same laws of motion as a large pendulum, and does so in a fraction of the time.

The human brain operates at about 200 Hz. Imagine hypothetical alien cultures where creatures evolved to have brains operating at 20 Hz or 2,000 Hz. Why would their advancement of science necessarily be limited by the “calendar time” of another intelligent species in an insignificant corner of the Milky Way Galaxy? Why would the limitations on knowledge to be gained from experiments be perfectly aligned with the inherent neural firing rates and “calendar time” of Homo sapiens? This is the Copernican Error of human-centrism — “if we’re a certain way, every other possible mind will have the same limitations, guaranteed”.

There are a variety of ways to boost one’s experimental output and data input. One obvious method is parallelism. Humans can only focus on one thing at a time, but an AI mind could focus on an unlimited number of experiments as long as it has the computing power and hardware. Another method would be miniaturization. The use of microarrays in biology research has made possible far less expensive and far more parallel experimentation than ever before.

I see Kelly’s position as anthropocentric triumphalism — we’re the greatest, no one can be as good as us, we have nothing to fear from a hard takeoff, any AI mind will need to engage in centuries of research to get as far as us. Sure, it might turn out to be true, but why put the human species on the line for this hypothesis?

Specialized vs. General Molecular Assemblers and the Risk of AGI Thursday, Apr 30 2009 

J. Storrs Hall at the Foresight Institute has responded to my recent post about the challenges of self-replication. Specifically, the line where I refer to the Foresight Institute and the Center for Responsible Nanotechnology:

What is remarkable are those that seem to argue, like Ray Kurzweil, the Foresight Institute, and the Center for Responsible Nanotechnology, that humanity is inherently capable of managing universal self-replicating constructors without a near-certain likelihood of disaster.

Dr. Hall responds:

From this he jumps with very few intervening arguments (”there are terrorists out there”) to a conclusion that we need a benevolent world dictatorship (”singleton”), which might need to be a superhuman self-improving AI. This seems a wildly illogical leap, but surprisingly appears to be almost an article of faith in certain parts of the singularitarian community and Washington, DC. Let us examine the usually unstated assumptions behind it:

A singleton need not be a benevolent world dictatorship — just a “world order in which there is a single decision-making agency at the highest level”, as defined by Nick Bostrom, who says:

A democratic world republic could be a kind of singleton, as could a world dictatorship. A friendly superintelligent machine could be another kind of singleton, assuming it was powerful enough that no other entity could threaten its existence or thwart its plans. A “transcending upload” that achieves world domination would be another example.

Consider the concept of global governance, for instance.

I consider it likely that a singleton will emerge in the 21st century, whether we want it to or not, as a natural consequence of expanding technological powers on a finite-sized planet, as well as a historical trend of aggregation of powers at higher geopolitical levels. Note that the singleton concept does not specify what degree or scope of decision-making powers the entity (which, as pointed out, could be a worldwide democracy) has. 99% of policy choices could very well be made at the local and national levels, while a singleton intervenes in those 1% of choices with global importance. As Dr. Hall points out later in his post, it seems like a pseudo-singleton already exists. He calls it the US Government, but I’d call it a fuzzy entity that consists of the shared consensus between the US Government, its opinion sources (academia, public, media), the UN (which is not just controlled by the US), the European Union, NATO, and other assorted actors.

To me, what I’d want most out of a singleton would be a coherent and organized approach to problems that face the entire planet. Instead of a disorganized patchwork, there’d be more decisive action on global risks. No authoritarianism in cultural, political, or economic matters is implied.

This is what I think of when I hear calls for “more international cooperation” on terrorism or global warming. This is why we have the WHO as the highest source of authority on the emerging swine flu. People say that international organizations and institutions are weak, and maybe some of them are, but at least a portion of them help the entire world move through crucial challenges. Celebrities and politicians emerge to champion causes and rally supporters. Diversity in opinion, unity in action. It’s called cooperation.

The “singleton” I want could merely be described in terms of “more cooperation on threats to us all, including the question of whether certain threats are really threats or not”. Whether AI is in the picture or not is really a secondary issue, but if AI expands our capacity to detect and respond to threats, more power to it.

Next, Dr. Hall argues:

Humanity can’t manage self-replicating universal constructors: We’ve been managing self-replicating universal constructors for tens of thousands of years, from elephants to yeast. What’s more, these are replicators that can operate in the wild. The design process, e.g. to turn a wolf into a Pekingese, takes longer but is much more intuitive to the average human.

If you’re worried about high-tech terrorists, worry about genetically engineered swine flu or other naturally-reproducing agents. If there are terrorists out there who are so technically sophisticated as to be a threat with MNT, at best guess still 20 years away for the leading mainstream labs, why aren’t they doing this? Even terrorist Berkeley professors only make letterbombs.

One type of self-replicating constructor could conceivably replicate itself in less than a day and become arbitrarily large and energy-hungry, and another takes at least a year to self-replicate and has a bounded size. One can make nearly anything and another is highly restricted in its specifications… there’s no comparison here.

I am certainly worried about genetically engineered swine flu or other naturally-reproducing agents, and have been posting about these issues frequently. But I still reserve concern for the challenges of MNT, even if they may be 20, 30, or even 40 or more years off. Partially because the advances are fairly far off, the field for debate and thought is smaller than it would otherwise be, potentially giving early actors such as ourselves disproportionate influence over how the debate evolves in the future. As I plan to be discussing technological risk 20, 30, and 40 years from now, I am getting started early by voicing my concerns in 2009. If MNT does become an issue in 2030 or 2040, then hopefully I will be one of the people that is solicited for ideas on how to handle it, partially based on my public analysis of the problem at such an early juncture.

My concern about MNT is that it will not be that technically sophisticated when it is rolled out worldwide. That is, it will be possible to create weapons cheaply and easily with intuitive interfaces when non-restricted nanofactories become available around the world. (If diamondoid nanofactories are possible at all, which I wager them to be.) Even if the non-restricted nanofactories are only available to “scientists” or “authorities”, there is a significant risk of them being dispersed via the black market. The demand would surely be astronomical.

If the nanofactories in question just use proteins to make products, as Dr. Drexler has been arguing for lately, then a lot of the security issues evaporate. As far as I know, you can’t make a powerful missile, gun, or millipede robot out of keratin.

Next, Dr. Hall rightly points out that universal constructors probably wouldn’t be distributed to everyone:

Once the leading mainstream labs produce self-replicating universal constructors, they are hardly going to hand them out by the billions for people to make shoes with. As Eric Drexler recently pointed out, specialized mill-style machinery is considerably more efficient than universal constructors at actually making stuff. My analysis of this point is that the difference is months for universal constructors vs milliseconds for specialized mills. Nobody is going to want universal constructors except for research.

Of course. The MNT community realized this a while ago. When I say, “managing universal self-replicating constructors”, I don’t mean that universal constructors will be distributed as consumer products. I realize that consumer nanofactories are likely to be specialized devices. I am referring to the point at which a limited number of actors acquire more-general (not necessarily universal) manufacturing capabilities, which in turn leads to distribution of more specialized versions of the technology to millions or billions of people. Perhaps “universal” is the wrong word, because as Dr. Drexler has also pointed out, it may be too much to predict a single device to be universal: it doesn’t have to be. Cooperation between specialized devices should be quite sufficient to hit a very large space of manufacturing targets.

So, to rephrase, what I am concerned about is the widespread availability of more-general high-throughput manufacturing devices, which will result from the invention of a nearly-general molecular assembler. If I could revise my claim, I would subtract the word “universal” and say “general self-replicating nanofactories” instead of “universal self-replicating constructors”. By “constructors”, I meant the entire system, not just the tiny assemblers themselves, so I replace it with “nanofactories” to make it more clear. An individual assembler need not self-replicate — perhaps 1000 assemblers could cooperate together to make another assembler. The technical issues around this are another ongoing debate and analysis. Still, what I am concerned about is that any combination of product-restricted nanofactories could be used to produce additional manufacturing devices that could be put to ill ends. Specialized nanofactories could be used to build more general construction devices, perhaps not even based on MNT at all. I am talking about a general magnification of our manufacturing capability and speed.

The concern is that a variety of products that are likely to be approved for manufacture will be dual-use products that can be turned to illicit ends. For instance, the general equipment in an chemical laboratory can be used to manufacture methamphetamines or opiods like oxycontin. In an MNT-equipped world, instead of this equipment costing tens of thousands of dollars, it may cost a thousand dollars, a few hundred, or even less. MNT, when and if it is developed, will magnify the technological oomph behind any human tendency by orders of magnitude. Tendencies towards good as well as envy, obsession, and evil.

The questions I am concerned about are the following:

1. Once universal constructors are developed, who will get them? The company that develops them? The US military? The US Government? The United Nations? The highest bidder?

2. Will there be any government controls on these universal constructors? As systems are developed that are less general than “root” systems, but still general enough to build weapons, illicit materials (for instance, addictive designer drugs), intrusive surveillance systems, dual-use systems, and the like, who will regulate which level of access gets which products?

The general implied position of the Foresight Institute appears to be, “we’ll figure these things out as we go, MNT should be developed as soon as people put up the funding for it, everything will pretty much be fine”.

In my analysis, the situation is relatively bleak. Forces arguing in favor of “openness” and “power to the people” will, while well-intentioned, probably end up granting too much custom-design, high-throughput manufacturing power to too many actors, and once the genie is out of the box, it can never go back in. Once you have a single unrestricted nanofactory, you can make 100 more (as long as you have the feedstock) in just a few days and hide them in very out-of-the-way places. Note that one of my primary concerns is high-throughput manufacturing, not just generality. If both generality and manufacturing speed could be artificially limited in the vast majority of nanofactory devices, perhaps the global security risk would be much diminished.

There are obvious ideas floating around, which I’ve written about before, for making nanofactories safer: GPS tracking, the need for certification to manufacture certain products, the recommendations set forth in the Foresight Guidelines on Molecular Nanotechnology, restricting the manufacture of products based on their chemical composition, intended purpose, energy density, speed, or size. Military Nanotechnology by Jürgen Altmann, a disarmament expert and physics Ph.D, puts forth some good ideas, which unfortunately will probably be considered too radical and restrictive to be adopted by any major country or company.

Particularly bleak in my book is the vast improvement in isotope separation technology which would become possible when dual-use, MNT-built industrial machinery is put to the challenge. There are over a dozen ways to enrich uranium, and many of the more advanced techniques are held back mostly by 20th century materials and a lack of manufacturing precision and reliability.

Dr. Hall writes:

Note that a really universal constructor at the molecular level would, even under current law, require a bushel of different licenses to operate — one for each of the regulated substances it was capable of making. Sony is not going to be selling these things on the streets of Mumbai.

I somehow worry that the DIY advocates will turn the tide of regulation with this one. For a device that inherently can make practically everything, picking out every item to exclude is much harder than just allowing a wide range of things and only introducing regulation when some terrible accident happens. Because the vast majority of constructed objects will be entirely benign and helpful in an economic and humanitarian sense, the legislatures of the world will be thrown off guard, embracing an “open source” perspective that puts as much power in the hands of the people as possible. When it comes to software, I’m all in favor of open source, but when it comes to manufacturing actual objects that have a physical impact on my world, I’d prefer that not just anyone be allowed to manufacture just anything.

Even a device with highly specialized manipulators at the nanoscale could still produce a huge variety of products. For instance, these highly specialized manipulators could be specialized to create nanoblocks, 100 nm-sized blocks with a variety of pre-programmed structures and functionality which could be combined in arbitrary patterns, like Legos. Specialized at the molecular level, thoroughly general at the person level.

If Dr. Hall means specialized as in “specialized to create dinnerware”, such over-specialization seems unlikely to me. There will be strong social and economic reasons that argue in favor of generality. I don’t want to switch manufacturing machines every time I want to build an object in a slightly different category. Just like with computers, most nanofactories will be relatively general, though the precise question is how general.

Dr. Hall then says, in reference to the notion of a benevolent AI singleton:

Anyway, there already is a “singleton” — the US government. It has clearly demonstrated a willingness to act to prevent even nuisance-level WMD by actors outside the currently-accepted group. (By nuisance-level I mean ones which pose no serious threat to topple the US from its dominant military position.) The notion of producing, from scratch, an entity, AGI or whatever, that would not only seriously threaten US dominance but depose it without a struggle seems particularly divorced from reality. (Note that the US military is the leading funder and user of AI research and always has been.)

But, that is exactly what we are arguing. A “seed” artificial intelligence, an AI built specifically for self-improvement, could break away from its programmers as soon as it gains a threshold level of capacity for self-creation and implementing real-world plans. In the same way that the Wright Flyer was, strictly speaking, many orders of magnitude less complex than a flying bird or insect, the first artificial intelligence may be many orders of magnitude less complex than a human mind and yet still capable of forming useful theorems about learning, decision-making, and competition that allow it to materially enhance its own intelligence and capability to far above the human level.

Because an AI would not be limited by unitary identity (it could break itself into pieces to work on tasks), finite hardware (additional computing power could be rented through cloud computing), the need to rest (an AI could run 24/7/365 with sufficient electricity), a brain unintended for hardware-level self-improvement (nature has retained the same basic neural building blocks for over 400 million years), frustration or boredom, social needs, bodily frailty, short-term memory limited to seven objects, and hundreds if not thousands of other shortcomings of biological minds, an AI mind considered as smart as a 10-year-old could probably achieve a heck of a lot more than a 10-year-old in a similar position.

Essentially, all of the human species is at the same intellectual level in terms of our cognitive capabilities. Even the least intelligent humans, unless they have brain damage, have greater cognitive capabilities than the smartest chimp. Our distinct level of cognitive ability is species-general and all we’ve ever known, so we tend to take it for granted. We fail to realize the solutions that an intelligence just slightly above us would see, just like there are a million things that are obvious to us and impossible to comprehend for a chimp, or even a dumber human.

The central argument is that humanity is not special. Just like the Earth turned out not to be the center of the universe and humans turned out not to be created in the image of God, some humans may be surprised to find out that we aren’t at the center of the cognitive universe. We’re just another step on a ladder between worms and the great unknown. Call it the Copernican Revolution in cognitive science.

Getting AI up to the point of human-equivalent intelligence may be incredibly difficult, and take decades as well as hundreds of millions of dollars in distributed research. But once it is at that point, it is easy to imagine self-improvement scenarios where the practical power of an artificial intelligence quickly begins to exceed that of even the largest human collectives. Some relevant variables are named in my summary of last summer’s SIAI-funded research project.

It is classic anthropocentrism to say, “this human government is so powerful and mighty, how could it possibly be that this new species could exceed its capabilities?” Because from the perspective of the new entity, humans are intellectually just a bunch of monkeys. Physically too. An AI can be in a million different places at once, a human, just one.

I am hardly the first person to suggest that AI could surpass humanity in its capabilities, or even overcome a major government without a struggle. The entire Singularity Summit event is based at least partially on that premise — the idea of an “intelligence explosion”, which originated at least as early as 1965 with the recently deceased I.J. Good. Most of society is at least familiar with the idea of runaway AI, and a sizable educated minority grants it a non-negligible probability in the coming century. Larry Page and Bill Gates are obviously among that minority, which is why Page helped fund Singularity University and Gates is such a big fan of Kurzweil. So is congressman Brad Sherman, who has raised the issue in the US Congress.

Dr. Hall then writes:

It seems to me that if you can make a self-improving, superhuman AGI capable of taking over the world, you could probably make a specialized AI capable of running one desktop fab box. Its primary purpose is to help the 99.999% of users who are legitimate to produce safe and useful products. It doesn’t even have to outsmart the terrorists by itself — it is part of a world-wide online community of other AIs and human experts with the same basic goals.

It’s not that easy — nanofactories could come before any type of sufficiently advanced AI. Remember, in our analysis, a self-improving superhuman AI is not radically harder to create than a roughly human-equivalent seed AI — the latter would transform itself into the former in a relatively short period of time, not limited by human thinking/acting speeds or methods. As Drexler writes in Engines of Creation:

The engineering AI systems described in Chapter 5, being a million times faster than human engineers, could perform several centuries’ worth of design work in a morning.

It is perhaps unfortunate that some thinkers have come to see claims about MNT and superhuman AI as interdependent, when it is possible for one class of claims to be right and the other wrong just as easily for both to be wrong or right. As for myself, I tend to be more convinced that a human-equivalent self-improving AI would be able to empower itself rapidly than I am that reliable diamondoid mechanosynthesis will be implemented in nanofactory systems before 2030. As for when human-equivalent AI will come about, I would certainly prefer it to be before 2040, but I have absolutely no idea. On the issue of converting sand (silicon) into mind, it matters not as much when exactly it happens as the magnitude of its impact. Instead of perpetually lurking around the human-level of intelligence and capability, I would expect AIs to skyrocket in capability far past the human level, limited only by their consideration for the welfare of other beings (if such consideration is indeed present).

Next Page »