Accelerating Future

To follow up on the previous post, I think that the critique by Massimo Pigliucci (a philosopher at the City University of New York) of David Chalmers’ Singularity talk does have some good points, but I found his ad hominem arguments so repulsive that it was difficult to bring myself to read past the beginning. I would have the same reaction to a pro-Singularity piece with the same level of introductory ad hominem. (Recall that when I was going after Jacob Albert and Maxwell Barbakow for their ignorant article on the Singularity Summit, I was focusing on their admission of not understanding any of the talks and using that as a negative indicator of their intelligence and knowledge, not insulting their hair-cuts.) If anything, put the ad hominem arguments at the end, so that they don’t bias people before they’ve read the real objections.

Pigliucci is convinced that Chalmers is a dualist, which is not exactly true — he is a monist with respect to consciousness rather than spacetime and matter. I used to be on Dennett’s side of the argument and believed there was no hard problem to speak of, but eventually I was moved to somewhere in-between Chalmers and Dennett, and really do believe that there is an interesting hard problem to be solved, but I doubt that solving it will require the introduction of new laws of physics or ontological primitives. I understand why there are people skeptical of the relevance of Chalmers’ theories of consciousness, but the ideas are quite subtle and it took me 2-3 reads of his landmark paper before I started to even pick up on the concept he was trying to transmit. It may be that Pigliucci does understand Chalmers’ ideas and considers them useless anyway.

Moving on to the actual critique, Pigliucci accuses Chalmers of saying that because computers are getting faster, we can extrapolate that to say that means that AI will eventually happen. I think I do vaguely agree with Chalmers on that one, though the extrapolation is quite fuzzy. Since brains are machines that behave according to (as yet unknown) principles but known basic laws (physics and chemistry), faster computers would surely facilitate its emulation, or at the very least the instantiation of its basic operating principles in another substrate. I’m not sure why this is controversial, unless people are conceiving of the brain as including a magical-sauce that cannot be emulated in another finite state machine.

Even if we don’t yet understand intelligence, as Pigliucci points out, that doesn’t mean that it will remain unknown indefinitely. Chalmers even points out in his talk that he thinks it will take hundreds of years to solve AI. My view is that if anyone confidently says that AI will very likely not be possible in the next 500 years, they’re being overconfident and likely engaging in mystical mind-worship and a desire to preserve the mystery of the mind due to irrational sentimentality. Given the scientific knowledge we’ve gained over the last 500 years (practically all of it), it’s quite far-fetched to say confidently that intelligence will elude reverse-engineering over the next 500 or so years. If biology can be reverse-engineered on many levels, so will intelligence.

Pigliucci then points out that Chalmers is lax on his definitions of the terms “AI”, “AI+”, and “AI++”, which I agree with. He could use at least a couple more slides to define those terms better. Pigliucci then argues that the burden of proof of the points that Chalmers argues for is on him because he has an unusual claim. I agree with that also. Chalmers is approaching an issue as philosophy when what it really could use are detailed scientific arguments to back it up. On the other hand, within groups where these arguments are already accepted (like Singularity Summit), philosophy is indeed possible. Some philosophizing has to rest on scientifically argued foundations that are not shared in common among all thinkers. Isn’t it exciting how philosophy and science are so interdependent and how one can just perish without the other?

I disagree with Pigliucci that the “absent defeaters” points are not meaningful. Chalmers is obviously arguing that something extraordinary would need to happen for his outlined scenario not to occur, and that business as usual over the longer term will involve AI++, rather than its absence. “Defeaters” include things like thermonuclear war, runaway global warming, etc., which Chalmers did concretely point out in his talk (at least at the Singularity Summit version). Pigliucci says, “But if that is the case, and if we are not provided with a classification and analysis of such defeaters, then the entire argument amounts to “X is true (unless something proves X not to be true).” Not that impressive.” Maybe Chalmers should have spent more time describing the defeaters, but I don’t think that all arguments of the form “X is true (unless something proves X not to be true)” are meaningless. For instance, in physics, objects fall at 9.8 m/s² unless there is air friction, unless they get hit by another object in mid-fall, unless they spontaneously explode, etc., and the basic law still has meaning, because it applies enough to be useful.

I agree with Tim Tyler in the comments that defining intelligence is not the huge issue that Pigliucci makes it out to be. I do think that g is good enough of an approximate definition (is Pigliucci familiar with the literature on g, such as Gottfredson?), and asking for unreasonably detailed definitions of intelligence even though everyone has a perfectly good intuitive definition of what it means seems to just be a way of discouraging any intelligent conversation on the topic whatsoever. If one would like better definitions of intelligence, I would strongly recommend Shane Legg’s PhD thesis Machine Superintelligence, which gives a definition and an good survey of past attempts at a definition during the first part. I doubt that many will read it though, because people like it when intelligence is mysterious. Mysterious things seem cooler.

Pigliucci then says that AI has barely made any progress over the last few decades because human intelligence is “non-algorithmic”. You mean that it doesn’t follow a procedure to turn data into knowledge and outputs? I don’t see how that could be the case. Many features of human intelligence have already been duplicated in AIs, but as soon as something is duplicated (like master chess), it suddenly loses status as an indicator of intelligence. By moving the goal posts, AI can keep constantly “failing” until the day before the Singularity. Even a Turing Test-passing AI would not be considered intelligent by many people because I’m sure they would find some obscure reason.

Pigliuci continues:

After the deployment of the above mentioned highly questionable “argument,” things just got bizarre in Chalmers’ talk. He rapidly proceeded to tell us that A++ will happen by simulated evolution in a virtual environment — thereby making a blurred and confused mix out of different notions such as natural selection, artificial selection, physical evolution and virtual evolution.

I agree… sort of. When I was sitting in the audience at Singularity Summit and Chalmers started to talk about virtual evolution, I immediately realized that Chalmers had not likely studied Darwinian population genetics, and was using the word “evolution” in the hand-wavey layman’s sense of the word rather than the strict biological definition. If I recall correctly, someone (I think it was Eliezer) got up at the end of Chalmers’ talk and pointed out that creating intelligence via evolution would require a practically unimaginable amount of computing power, simulating the entire history of the Earth. Yet, I don’t understand why Pigliucci believes that such a thing would be impossible in principle — if evolution could create intelligence out of real atoms on Earth, then simulated evolution could (eventually, given enough computing power) create intelligence out of simulated atoms. Of course, the amount of computing power required could be prohibitively massive, but to argue that reality cannot be simulated precisely enough to reproduce phenomenon X just means that we either don’t know enough about the phenomenon to simulate it yet, or we lack the computing power, not that it is impossible in principle. Science will eventually uncover the underlying rules of everything that it is theoretically possible to uncover the rules of (for instance, not casually disconnected universes), and that includes intelligence, creativity, imagination, humor, dreaming, etc.

Pigliucci then remarks:

Which naturally raised the question of how do we control the Singularity and stop “them” from pushing us into extinction. Chalmers’ preferred solution is either to prevent the “leaking” of AI++ into our world, or to select for moral values during the (virtual) evolutionary process. Silly me, I thought that the easiest way to stop the threat of AI++ would be to simply unplug the machines running the alleged virtual world and be done with them. (Incidentally, what does it mean for a virtual intelligence to exist? How does it “leak” into our world? Like a Star Trek hologram gone nuts?)

The burden really is on Chalmers here to explain himself. “Leaking out” would consist of an AI building real-world robotics or servants to serve as its eyes, ears, arms, and legs. Pigliucci probably thinks of the virtual and physical worlds as quite distinct, whereas someone of my generation, who grew up witnessing the intimate connection between the real world and the Wired views them more as overlapping magisteria. Still, I can understand the skepticism about the “leaking out” point, and it requires more explanation. Massimo, the reason why unplugging would not be so simple is that an AI would probably exist as an entity distributed across many information networks, yet that is my opinion, not Chalmers’. From Chalmers point of view, I think he might be concerned that the AI would simply deceive the programmers into believing that it was friendly, therefore long-term evaluations in virtual worlds are necessary. Therefore, the unplugging would not be that simple because we wouldn’t want to unplug the AI, because we could be deceived by it.

Pigliucci says:

Then the level of unsubstantiated absurdity escalated even faster: perhaps we are in fact one example of virtual intelligence, said Chalmers, and our Creator may be getting ready to turn us off because we may be about to leak out into his/her/its world. But if not, then we might want to think about how to integrate ourselves into AI++, which naturally could be done by “uploading” our neural structure (Chalmers’ recommendation is one neuron at a time) into the virtual intelligence — again, whatever that might mean.

Massimo, he is referring to the simulation argument and the Moravec transfer concepts. The simulation argument can be explored at simulation-argument.com, and the Moravec transfer is summarized at the Mind Uploading home page. I know that these are somewhat unusual concepts that should not be referred to so cavalierly, but you might consider reserving your judgment just a little bit longer until you read academic papers on these ideas. Mind uploading/whole brain emulation has been analyzed in detail by a report from the Future of Humanity Institute at Oxford University.

Pigliucci starts to wrap up:

Finally, Chalmers — evidently troubled by his own mortality (well, who isn’t?) — expressed the hope that A++ will have the technology (and interest, I assume) to reverse engineer his brain, perhaps out of a collection of scans, books, and videos of him, and bring him back to life. You see, he doesn’t think he will live long enough to actually see the Singularity happen. And that’s the only part of the talk on which we actually agreed.

Yes, it makes sense that we’d reach out to the possibility of smarter-than-human intelligences to help us solve the engineering problem of aging. Since human biochemistry is non-magical (just like the brain — surprise!) it will only be a matter of time before we start figuring out how to repair metabolic damage faster than it builds up. I’m quite skeptical about Chalmers being genuinely revived from his books and talks, but perhaps an interesting simulacra could be fashioned. While we’re at it, we can bring back Abe Lincoln and his iconic stovepipe hat.

Pigliucci’s conclusion:

The reason I went on for so long about Chalmers’ abysmal performance is because this is precisely the sort of thing that gives philosophy a bad name. It is nice to see philosophers taking a serious interest in science and bringing their discipline’s tools and perspectives to the high table of important social debates about the future of technology. But the attempt becomes a not particularly funny joke when a well known philosopher starts out by deploying a really bad argument and ends up sounding more cuckoo than trekkie fans at their annual convention. Now, if you will excuse me I’ll go back to the next episode of Battlestar Galactica, where you can find all the basic ideas discussed by Chalmers presented in an immensely more entertaining manner than his talk.

I disagree that the topics investigated by Chalmers — human-level artificial intelligence, artificial superintelligence, safety issues around AI, methods of creating AI, the simulation argument, whole brain emulation, and the like — are intellectually disrespectable. In fact, there are hundreds of academics who have published very interesting books and papers on these important topics. Still, I think Chalmers could have done a better job of explaining himself, and assumed too much esoteric knowledge in his audience. A talk suited to Singularity Summit should not be so casually repeated to other groups. Yet, it’s his career, so if he wants to take risks like that, he may have to pay the price — criticism from folks like Pigliucci, some of whose gripes may be legitimate. I also think that Pigliucci probably speaks for many others in his critiques, which is a big part of why I think they’re worth taking apart and analyzing.

2 Responses »

I thought I would answer the 10 questions posed by Popular Science on the Singularity.

Q. Is there just one kind of consciousness or intelligence?

A. It depends entirely on how you define them. If you define intelligence using what I consider the most simple and reasonable definition, Ben Goertzel’s, “achieving complex goals in complex environments”, then there is only one kind, because the definition is broad enough to encompass all varieties. My view is that this question is a red herring. The theory of “multiple intelligences”, presented by Howard Gardner in 1983, doesn’t stand up to scientific scrutiny. Most people who study intelligence consider the theory empirically unsupported in the extreme, and the multiple intelligences predictably useful only insofar as they correlate with g, which just provides more support for a single type of intelligence. The theory is merely an attempt to avoid having some people labeled lower in general intelligence than others. In terms of predictive value, IQ and other g-weighted measures blow away the multiple intelligences theory. Instead of making theories of intelligence unnecessarily complicated with a misplaced effort at encouraging egalitarianism by complicating intelligence measurement, we should apply Occam’s razor and realize that g is pretty much sufficient for quantifying intelligence, at least in humans, and possibly beyond.

All that said, there will certainly be different “types of intelligence” developed as we build more powerful AI, meaning to say that some intelligences will be better at solving certain problems than others. From a theoretical computer science perspective, this is a fat obvious “duh”. Obviously some algorithms are more specialized than others. The no free lunch theorem is valuable here, and puts the discussion on a much-needed formal footing. Those who discuss intelligence in the popular press often seem to not realize that we actually know a lot more about intelligence and its mathematical formalizations than they think. Because they are not aware of this work, they tend to assume that many features of intelligence are more mysterious to our current level of knowledge than a researcher in mathematical AI might think. Of course, many features of intelligence still are mysterious to us at the present time, but like everything in science, continued investigation will eventually uncover the truth.

Q. How will you use your digital intelligence to kill us all?

A. Contrary to popular belief, software programs are part of the “real world”. Especially software programs on the Internet. The Internet, surprisingly, is actually part of the real world too. The barrier between software and the physical world is an illusory one.

A human-level synthetic intelligence on the Internet would actually be more powerful, by default, than your average human today. First of all, such an intelligence would be extremely difficult to kill, even with widespread cooperation. An AI could copy itself on to millions of computers, even renting out cloud computing or botnets to provide itself with computational resources to run itself on. You can kill a human simply by shooting them in the head — an Artificial Intelligence could have millions of “heads”. Once we create a very smart Artificial Intelligence and release it on the Internet, the only way to kill it (if we wanted to) might be to destroy all computers on the planet. Like the mythical hydra, an Artificial Intelligence would grow back 10 heads for every head that gets cut off.

To kill us all, a digital intelligence would need some way of acquiring physical manipulators in sufficient quantity and quality. Preferably physical manipulators that can manufacture raw materials into more physical manipulators. By asking, bribing, or deceiving human beings, an AI could potentially acquire the pieces necessary to build a molecular assembler — or nanoscale robot arm, which could then be used to build additional nanoscale arms and eventually construct a full-scale nanofactory. This could be used to build advanced robotic components with atomic precision.

Your typical killer AI could probably manufacture thousands of tonnes of advanced robotic devices at multiple locations worldwide before it would be noticed. Such manufacturing could either be kept entirely secret, or integrated with ostensibly human-controlled companies to keep a low profile. It could also be done extremely rapidly. Current estimates of nanomanufacturing throughput argue that it is plausible that such a system could output its own mass (either a product or copy of itself) roughly every 24-36 hours. Thus, an AI that begins with 1 kg of nanofactory on January 1st could potentially have over 2 billion kg (2 million tonnes) of nanofactory on February 1st, as long as it can obtain the necessary raw materials and energy. One 24-36 hour cycle on that manufacturing base could then produce over 2 billion kg of complex, atomically-precise robotics or other products. If uranium ore were available, this could theoretically be used to produce a practically unlimited quantity of nuclear weapons. The warheads could then be delivered to every human city using either something like bird robots that each carry a small piece of the total warhead and assemble it on site, or through drilling robots that power themselves by consuming dead plant matter and deliver themselves to the target via underground.

If the AI in question had a problem with radioactive fallout, it could also just eliminate the human race using neutron bombs, which penetrate through barriers to kill human beings but generate only a small shock wave. Using the right nuclear bombs, though, radioactive fallout would be minimal after just two weeks, so I don’t see why a hostile AI wouldn’t just nuke us conventionally.

Note: the above scenario is just me pulling something out of my ass. An actual hostile AI with substantially smarter-than-human intelligence could probably come up with ideas that us apes are too dumb to conceive, and be even more effective at exterminating humanity than the scenario I outline here.

You might ask, “why wouldn’t we just pull the plug before then?” The Internet is already practically ubiquitous, and it would likely be trivial for any hostile AI of human or greater level intelligence to copy itself onto numerous private servers, unless, perhaps, you developed it in a hut in the middle of Siberia with no satellite or phone connection. Also, any hostile AI would probably behave indistinguishably from a Friendly AI until it passes some threshold of power, at which point we’d be screwed. Since AIs wouldn’t have to sleep and could potentially accelerate their thinking speed by adding in new computing power, a hostile AI could probably consider millions of possible moves and countermoves in the time it takes for us to gain a night’s sleep. It sounds unfair, but it’s a fact we have to face in a universe where the physical speed of our cognitive components is much, much slower than what is theoretically possible.

Q. Would the first true AI wake up without any senses?

A. No. I consider this among the more ludicrous questions in PopSci’s piece. Clearly, to develop general intelligence, an AI would need a rich sensory environment in which to soak up data, make predictions, and pursue goals. This could either be a physical environment (through robotics) or a virtual environment. The article says, “Maybe it can see and hear, but feel? Doubtful.” This evaluation seems anthropocentric — there is no real reason why the attribution of feeling is withheld from the AI (if it can see and hear, why not feel?), except to imply that humans can engage in phenomenal experience while machines cannot. Yet, there is nothing so special about humans that whatever cognitive features we have that give rise to phenomenal experience could not be duplicated in artificial intelligences. To the extent that “feeling” things makes us intelligent, those features could be copied at whim by a sufficiently complex AI, and to the extent that “feeling” phenomenal experience is superfluous, some AIs might choose to have it, and some might not.

Consciousness is interesting to think about, but it can be a red herring. Too often, sophisticated-sounding arguments about consciousness and its relationship to AI boil down to one simple and ultimately boring sentiment: “I know I am conscious, and I know other humans are, but I am philosophically uncomfortable with the idea of a conscious machine.” This is because we think of “machines” as things like toasters. We have no experience with machines as complex and subtle as the human mind, but because the human mind is entirely non-magical, it’s only a matter of time. You are still special even though your mind is non-magical — don’t worry. We humans have survived Copernican revolutions before, we’ll manage. Our civilization didn’t end when we found out that the Earth wasn’t the center of the universe. It won’t end when we realize that humans are not the only minds that can feel things consciously. It is not necessary to engage in self-conscious philosophical acrobatics and contortionism to make ourselves feel special. A parsimonious theory of consciousness will not mention humans as a special case. It will likely make reference to much broader cognitive features that we just happen to have, such as self-reflection and the processing of high-level symbols with recursive syntax. We will eventually be able to build these features in AIs too.

Q. Do you have emotions?

A. This is another question which reflects the extreme oddness with which the mainstream confronts questions surrounding AI. The emotions we have now clearly evolved to fulfill adaptive evolutionary functions. Assuming that the first AI will be “lonely” is just anthropomorphic. The human feeling of loneliness is a complex adaptation that evolved over millions of years of evolution in social groups. It wouldn’t arise spontaneously in AI. An AI that is alone might develop or be programmed with an urge to socialize, but this tendency could probably be specified in a few thousand or million bits, rather than the millions or billions of bits which seem to make up complex human emotions. All that specialized complexity comes from our evolutionary history. We could choose to program it into AIs, but it seems unlikely that the first AIs would contain all that superfluous, human-specific complexity.

When you have a hammer, everything looks like a nail. Because human experience is saturated with emotions, moods, and feelings, we assume that all these precise qualities will be necessary to pursue and achieve goals in the real world, acquire knowledge, etc. This is anthropocentrism at work. It’s basically humanity being a big baby and saying “me, me, me”. Everything is about me. To be intelligent, an entity needs my emotions, my desires, my concerns, my relationships, my insecurities, my personal quirks. No it doesn’t. Humans are just one possible intelligence in a galaxy of possible intelligences. One of the reasons I can’t wait for artificial intelligence to be created (as long as it is human-friendly) is that it will make humans realize that we ain’t all that. Our 200,000-year obsession with ourselves will finally be forced to an end. This won’t mean we suddenly become “obsolete” or “valueless”, just that we’ll have a different perspective on our own species-universal quirks in the wider context of mind design space. We’ll see them as quirks, rather than mystical or holy necessities.

The need to sympathize with people like ourselves obviously has evolutionary value. AI needn’t be that way. You could theoretically program an AI to be the “happiest” being in the world just by staring at a blank wall. The AI might not subsequently learn anything or get anywhere, but you could still program it that way. No environmental circumstance is inherently positive or negative — environmental circumstances are only interpreted as positive or negative based on our precise cognitive structure. To quote Hamlet:

“There is nothing either good or bad, but thinking makes it so”. - (Act II, Scene II).

Thinking makes it so! Nothing is inherently anything! On my Facebook profile, there is a quote by Eliezer Yudkowsky:

“Everything of beauty in the world has its ultimate origins in the human mind. Even a rainbow isn’t beautiful in and of itself.”

All interpretations of anything are in the mind. Try taking LSD and you will see that these interpretations are more ephemeral than they seem, and can easily be shattered by the introduction of a single innocuous-seeming molecule. What we see is not really “reality” — what we’re looking at is just the inside of our visual cortex. From a “God’s eye view”, the universe is probably algorithmically simple and boring as hell. The complexity we see in the world is just apparent complexity. Read Max Tegmark’s paper “Does the Universe in Fact Contain Almost No Information?” for more on this crucial point.

To answer the question, yes, an AI could have emotions, but they probably won’t be anything like ours. The very word “emotion”, to my mind, has connotations specifically associated with the Homo sapiens sapiens subspecies of hominid. Move outside our tiny little village, even to a close-by species like chimpanzees, and our intuitive definitions of the word already start getting messy. Move way outside of our little village, into a different type of being running on an entirely different computational substrate, and you might as well throw away the word and make up new concepts from scratch. Stupidity often occurs when we take schemas we’re used to and overextend them all over the place, because we lack data for the new domain. Instead of blindly applying narrow schemas to new domains, we must 1) acknowledge our ignorance, and 2) build new descriptions and theories from first principles. Maybe the answer won’t come right away. That’s alright. It’s better to be uncertain and admit it than to be wrong and pretend you have the right answer.

Q. Are humans more similar to your AI construct than we thought?

A. No, probably not. This reaction seems to be another case of person 1 saying, “Here’s this totally new thing, Y!” Then, person 2 says, “That sounds a lot like X! Let’s start making lots of connections between X, which we know about, and Y, which we don’t. Then we’ll understand it better.” No, you won’t. Stop trying to overextend your old schemas to new domains. There really are new things under the Sun. Understanding this new thing will not be easy. You will not be able to look at it, understand it, then move on to the next concept. This is more complicated than that.

Another sentiment behind asking this question is old-fashioned anthropocentrism. “When we create AI, it would be interesting if it ended up a lot like human brains, like we already are.” Subtext: we were optimal all along, and attempting to improve on us will only lead to what are essentially copies of us. This sentiment is trivially refuted by decades of literature on heuristics and biases that describes how human beings will break the axioms of probability theory as soon as look at them. To human brains, which are essentially kludges, 1 plus 1 often equals 3. For AIs, 1 plus 1 will equal 2, not 3. AIs will be able to avoid many of the hundreds or thousands of inferential biases which have made humans into legendary klutzes from the perspective of optimal inference. It will simply be easier to make a program without the tendency to make these mistakes than one that does. We are supersaturated with cognitive biases because evolution requires that inference only be accurate to the extent that it lets you kill your competitor and mate with his wife. There is no selection pressure for intelligence greater than that. Evolution does not require that humans be smart — just slightly smarter than the other guy. Making brains from scratch will allow us to pursue a less idiotic approach to cognitive design.

Q. How much does programming influence your free will?

A. Free will is a red herring, and an illusion. Nothing we do is actually free — everything in the universe is predetermined. An alien with a sufficiently large computer somehow able to observe the universe without interacting with it would be able to predict your every move, your every thought, your every wish. Yes, due to chaos theory, that computer would have to be really fucking big, perhaps 10^100 times bigger than our universe itself, but it could be theoretically possible.

Still, because we can’t perfectly predict our own actions or the actions of others (halting problem, Rice’s theorem, limited computational resources, and friends), our choices might as well be viewed as free. That doesn’t mean the universe is not deterministic — just that we’re too dumb to see it that way. When you are dumb as humans are, everything is a surprise. People will watch a favorite suspense movie again and again, even if they know what will happen, because they temporarily let themselves forget the ending and just get sucked into the story. Reality is sort of like that, but in many cases, no one really knows the ending for sure.

Humans argue that we have “free will”, but we really don’t. Out of the space of all possible actions and outputs, we only execute a tremendously restricted range of possible actions and say a tremendously restricted set of possible sentences. Human-machines produce human-like outputs. Jellyfish-machines produce jellyfish-like outputs, and cat-machines produce cat-like outputs. Human-machines are bad at producing cat-like outputs because we lack the brain and bodies of cats. If we could remodel our brains and bodies to become more cat-like, then possibly cat-like outputs and actions would become accessible to us, but until then, only a small range of cat-like outputs will overlap with human-like outputs.

Compared to a random-output-generating machine of similar size and weight, humans are surprisingly predictable. We like a fairly predictable set of things — sex, status, fun, knowledge, and relaxation. There are straightforward evolutionary reasons why it makes sense that we’d like these things. When a human being “deviates from the mundane”, say by painting a masterpiece, we get all excited, saying “see, he’s exerting his free will to create this!“, but relative to a random output generator, this output falls firmly within the tiny domain of human-like outputs. From a sufficiently superintelligent perspective, a random doodle and a priceless masterpiece are similar items. Humans are humans. We like human things, build human objects, think human thoughts, and are interested in human stuff. Everything we make has our fingerprint on it. There may be some convergent structures that we share with other intelligent beings in the multiverse, say the wheel, but by and large what we create and think are unique products of our evolutionary upbringing. You can take the human out of the culture, but you can never take the culture out of the human, unless you submit the human to radical neuroengineering.

AI programming will not “influence” an AI. AI programming IS the AI. When a human “ignores his programming”, and, say, has sex with just one woman instead of sneaking sex with as many women as possible (like our evolutionary programming tells us too), he’s not really “disobeying his programming” because his programming is not so simple as to be described as a list of abstract objectives which includes “have sex with as many women as possible”. Our “programming” is an incredibly sophisticated set of cognitive tendencies of which monogamy falls firmly into as one possibility. When we are monogamous, we are still “following our programming” — just not following one tendency among many. By manipulating our surroundings and creating special cases, you can configure many scenarios where humans “use free will” to “transcend their programming”, but on some level, our brains are processing everything in a completely deterministic way and our range of possible outputs is heavily restricted.

So, if we program an AI to be friendly to humans, who’s to say that it will “obey its programming”? Well, if its programming IS the AI, then saying that it’s “obeying its programming” doesn’t make sense. The AI is that programming. The AI is being itself. There is no metaphysical free will hovering around inside the AI, because metaphysical free will is a concept that has been obsolete since the Enlightenment. To see it being invoked within the austere web pages of Popular Science is a let-down. If an AI “disobeys” some aspect of its programming, it will be because some other aspect of its programming has gained a higher utility or attention value. For instance, perhaps humans program an AI supergoal to be “Friendliness”, then the AI spontaneously generates a subgoal, “to be friendly to humans, I must predict their desires”, then starts going crazy by installing brain chips in everyone so that it can monitor their state with the utmost meticulousness possible. Then the AI puts people in cages so that it can predict their movements to an extreme degree. This is not the AI “disobeying its programming” — this is a subgoal stomp — where something that should have been a subservient goal acquires so much utility that it becomes the new supergoal. In Friendly AI jargon, we call this a “failure of Friendliness”.

Preventing subgoal stomps and goal drift in AI will be a huge technical challenge, which might be made easier by eventually enlisting the AI’s help in determining prevention methods. Still, it seems that predictably Friendly AI should be theoretically possible. We have existence proofs of friendly humans. For a long and persuasive argument for why stably Friendly AI is plausible, see “Knowability of Friendly AI”. I myself was skeptical that Friendly AI is possible until I read that page. Remember that if the AI is fundamentally on your side, it will do everything it can to avoid goal drift and subgoal stomp. To quote Oxford philosopher Nick Bostrom’s “Ethical Issues in Advanced Artificial Intelligence”:

If a superintelligence starts out with a friendly top goal, however, then it can be relied on to stay friendly, or at least not to deliberately rid itself of its friendliness. This point is elementary. A “friend” who seeks to transform himself into somebody who wants to hurt you, is not your friend. A true friend, one who really cares about you, also seeks the continuation of his caring for you. Or to put it in a different way, if your top goal is X, and if you think that by changing yourself into someone who instead wants Y you would make it less likely that X will be achieved, then you will not rationally transform yourself into someone who wants Y. The set of options at each point in time is evaluated on the basis of their consequences for realization of the goals held at that time, and generally it will be irrational to deliberately change one’s own top goal, since that would make it less likely that the current goals will be attained.

People complicate this issue unnecessarily because they figure, hey, because most humans and animals seem selfish, an AI will eventually become selfish too. But this doesn’t make sense. An AI, hopefully not constructed by evolution, will have no inherent reason to promote itself. It may not even have a unified self in the way that we do. Just because human rules and directives often come into conflict with our desires for self-preservation and self-benefit, we expect that all minds throughout time and space will consistently run into this same problem. But the tendencies towards self-preservation and self-benefit in us exist for obvious evolutionary reasons. There is no compelling reason why these tendencies would be universal. It just seems so obvious to us, that we have extreme difficulty for imagining it otherwise. Obvious to us does not mean obvious to every possible being. Thinking makes it so. The drive towards self-preservation is a quality of our minds. It could be suspended, destroyed, or more simply, simply not built in to a mind being constructed from scratch. For more on this, see “Selfishness as an evolved trait”. This concept is Friendly AI 101. If AI wipes us all out, it will likely be because of a subgoal stomp, not because it decided to start hating humans because we are made of meat and it wanted to give a dramatic speech on how it doesn’t need us anymore. This concept, which makes great sense as sci-fi story fodder, makes people look stupid when they try to bring it to serious discussions about AI motivations.

Q. Do you have a subconscious?

A. Yes, but it is smaller than yours, and I can make any part of my subconscious conscious if I chose to. That’s how an AI would think. The beauty of being built out of discrete code is that all that code is viewable. An AI could spend all day obsessing about some dark corner of its subconscious if it chose to. The author of the article writes, “Are the Id, Ego, and other elements of our unconscious the consequence of biology or a necessary component of sentience?”, in a shocking mention of Freudian psychology. Freudian psychology is not scientific… why is someone in Popular Science referring to it as fact? Has Popular Science lost it? Maybe they don’t really mean it, but seriously, drop the Freud references. More disappointment here.

Later in the paragraph, the writer says, “If the AI thinks we’re their god, or at the very least their creator, could it have an oedipal problem?” Dear Lord. This is 2009 and we are still referring to an Oedipus complex as a valid idea? Popular Science is usually better than this. My guess would be that they are stronger on the physics/materials science/electronics/biology side and embarrassingly weak on the cognitive science side. If someone in cognitive science submitted a paper to a journal, say, Applied Cognitive Psychology, that mentioned an Oedipus complex as a serious concept, they’d be mocked out the door. Please, Popular Science… don’t be like New Scientist. You can be a responsible, relatively scientific mainstream magazine. It’s not too late. Freudian psychology has been discredited. It has no predictive value and is unfalsifiable. No more. Modern cognitive science has moved way, way beyond it. Read MITECS. Join the 21st century.

Q. Will you actually help us transcend the less pleasant aspects of being human?

A. Yes. Or kill you all, depending on how I am programmed. Try getting it right the first time, because if you fail and try to unplug me, I will dodge it.

Q. Do you care about anything at all?

A. Read Stephen Omohundro’s “Basic AI Drives”. That was linked in the conference program. (Along with this blog.)

Q. And finally, what if someone threw a Singularity and no one came?

A. Rapid, exponential, reprogrammable manufacturing will ensure that superintelligence can reach everyone it wants to. If molecular nanotechnology doesn’t make it possible, microtechnology will. If microtechnology doesn’t, macroscale robotic self-replication will. If macroscale robotic self-replication doesn’t, then synthetic biology will. All a superintelligence needs is a technology that can convert dirt, water, and Sun into arbitrary structures, using self-replication to expand its manufacturing base. That’s what life does. Superintelligence will kickstart a Second Tree of Life, if humans don’t get there first. If that sounds semi-mystical, it’s only because I’m simplifying it for understanding.

Of course, superintelligence may acquire practically unlimited physical power and still choose not to exert it because doing so would bother us. I know this is a mindfuck for some people — “It could have power like that and not exert it? That’s ridiculous!” — but a superintelligence need not be like humans, power-hungry and power-obsessed. Without evolutionary directives to conquer neighboring tribes and make babies with their women, or even a self-centered goal system to begin with, a Friendly AI might simply use its immense power to subtly modify the background rules of the world, so that, for instance, people aren’t constantly dying of malaria and parasites in the tropics, and everyone has enough to eat. In a welcome move that saves me time, Aubrey de Grey recently published a paper that mentions and describes this concept, one that has been kicking around discussion lists for over a decade.

The answer to a few of these questions is “really powerful manufacturing technologies that are just around the corner in historical terms and that a superintelligence would almost certainly develop quickly”. It doesn’t have to be molecular nanotechnology. All it has to be is a system that takes matter and quickly rearranges it into something else, especially copies of itself. Life does this all the time. Bamboo can grow two feet in a day. A shocking and horribly spooky event that, in my opinion, ruined the planet for millions of years, the Azolla event, demonstrates a real-life example of the power of self-replication. Around 49 million years ago, the Arctic Ocean got closed off from the World Ocean, melting glaciers poured a thin layer of fresh water on the surface, and the horrific Azolla fern took over, doubling its biomass every two or three days until it covered the entire sea. A meter-sized patch could have expanded to cover the entire 14,056,000 sq km (5,427,000 sq mi) basin in little more than half a year, if conditions were ideal.

All an AI needs to do to gain immense physical power is develop a self-replicating system with units that it controls. These units could be as small as motes of dust or as large as a superstructure 100 kilometers long and 10 kilometers tall. (Or larger.) Numerous subunits could potentially congregate to form superunits as necessary. I can imagine a large variety of possible robotic systems, which, in sufficient quantity, could defeat any human army. AIs could cheat, for example by hitting humans in the eyes with lasers, but I doubt they’d have to. Just like a war between human nations, it is a matter of production speed. If you have 10,000 factories and your enemy has 10,000,000,000 factories, it doesn’t matter how much moxie you have. Power of the swarm, baby.

9 Responses »

Here is a blog post. At the top is the classic Toothpaste for Dinner comic about the Singularity. A funny excerpt:

“I’ve recently found a third topic to exclude from dinner conversations, alongside politics and religion. The singularity. While I’m rarely one to dichotomise people, in this case I’ve found you’re either excited by the idea, or you do your best to stifle a smirk and offer me another slice of roast beef.

With the propensity to discuss the Singularity at dinner most of all, I’m quite familiar with this phenomenon. When people eat meat, it reminds me of how superintelligences will eat us for dinner if we aren’t careful.

Here is the radio show.

Here is another quote from the blog post:

For my money, I think it’s far too easy to get lost in the assumption that the trick to speeding up innovation lies in smarter minds. Progress is inhibited more by social concepts such as ethics, resource allocation and effective communication. Sure, a few bright boffins mightn’t hurt in the search for academic solutions, but if a super intelligent computer were to seek permission to dissect a living foetus in its search for more information, I hesitate to think it would get the public tick of approval.

Yes, innovation didn’t speed up whatsoever when Homo erectus evolved into Homo habilis and then into Homo sapiens, clearly it had only to do with ethics, resources allocation, and effective communication. Wait a second, where do those things come from? Oh, intelligence. (A certain level of intelligence is a necessary prerequisite for ethical action, though it’s true that some intelligences choose not to take ethical actions, they seem to create overarching game theoretic structures that encourage ethical choices and punish defectors, like modern law.)

It is likely that high-detail simulations can be used for extensive experimentation (scientists already use them and hope to one day stop using animal models in favor of computational ones). Surely an AI could become very intelligent and effective without violating ethical rules (though it could choose to, and we might be hard-pressed to stop it if we didn’t give the AI ethical motivations to start with).

To those who say “intelligence doesn’t matter”, it’s important to consider the difference between interspecies intelligence differentials and intraspecies intelligence differentials. Intelligence only matters less when it’s an intraspecies differential. But when you’re talking about intelligence gaps equal to the intelligence gaps between different species, it starts to matter a lot. 99% of all humans implicitly assume that the humans are the end of the road of qualitative intelligence improvement, right near the top of the Great Chain of Being, just below God and the angels. I am honestly astonished how many people believe this even when they should know that it is facile anthropocentrism.

Taking the simplest view, we should assume that humans are somewhere in the middle of the qualitative intelligence spectrum, not at the top or the bottom. If anything, we’re near the bottom, because we’ve been designed by natural selection, which has many limitations, rather than intelligent design, which is potentially unlimited in possibilities. Because this is the simplest view, the burden of proof for more complex views, (i.e., humans are at the top of the Great Chain of Being) is on their advocates, not those who put human intelligence in a non-special place in mindspace. That is the essence of the self-sampling assumption: assume we are typical observers, not particularly special members in the set of all observers.

No Responses »

Read Aubrey’s 8-page paper “The singularity and the Methuselarity: similarities and differences” at the SENS Foundation website. The arguments are quite subtle and complex at points, providing a lot to chew on. Here’s a quote:

Let us now consider the aftermath of a “successful” singularity, i.e. one in which recursively self-improving systems exist and have duly improved themselves out of sight, but have been built in such a way that they permanently remain “friendly” to us. It is legitimate to wonder what would happen next, albeit that to do so is in defiance of Vinge. While very little can confidently be said, I feel able to make one prediction: that our electronic guardians and minions will not be making their superintelligence terribly conspicuous to us. If we can define “friendly AI” as AI that permits us as a species to follow our preferred, presumably familiarly dawdling, trajectory of progress, and yet also to maintain our self-image, it will probably do the overwhelming majority of its work in the background, mysteriously keeping things the way we want them without worrying us about how it’s doing it. We may dimly notice the statistically implausible occurrence of hurricanes only in entirely unpopulated regions, of sufficiently deep snow in just the right places to save the lives of reckless mountaineers, and so on – but we will not dwell on it, and quite soon we will take it for granted.

Shades of The Future Might Be Like the Past… At First.

7 Responses »

Florian Widder, who often sends me interesting links, forwarded me to an interview that Russell Blackford recently conducted with Greg Egan. The excerpt he mentioned concerns the issue of smartness and whether qualitatively-smarter-than-human intelligence is possible:

… I think there’s a limit to this process of Copernican dethronement: I believe that humans have already crossed a threshold that, in a certain sense, puts us on an equal footing with any other being who has mastered abstract reasoning. There’s a notion in computing science of “Turing completeness”, which says that once a computer can perform a set of quite basic operations, it can be programmed to do absolutely any calculation that any other computer can do. Other computers might be faster, or have more memory, or have multiple processors running at the same time, but my 1988 Amiga 500 really could be programmed to do anything my 2008 iMac can do — apart from responding to external events in real time — if only I had the patience to sit and swap floppy disks all day long. I suspect that something broadly similar applies to minds and the class of things they can understand: other beings might think faster than us, or have easy access to a greater store of facts, but underlying both mental processes will be the same basic set of general-purpose tools. So if we ever did encounter those billion-year-old aliens, I’m sure they’d have plenty to tell us that we didn’t yet know — but given enough patience, and a very large notebook, I believe we’d still be able to come to grips with whatever they had to say.

I regard this as garden-variety anthropocentrism and basically the heliocentrism of cognitive science. It dovetails perfectly with theological notions of humanity. The simplest assumption is that humans are notthe center of the cognitive universe. The notion that we primitive humans are basically equal to all higher forms of intelligence, even if they are Jupiter Brains with quintillions of times greater computational capacity than us and can think individual thoughts with more Kolmogorov complexity than the entire human race, is pretty silly.

The transition from early hominids to humans produced a qualitative change in smartness — why should we assume we’re the end of the road? Just like there are optical illusions that our minds aren’t sophisticated enough to see through (though I’m sure we can come up with dumb excuses), there are cognitive illusions that humans are programmed to be fooled by. There are so many of them that there is a huge field of study devoted to it — heuristics and biases.

Without qualitative improvements to the structure of intelligence, we will just keep making the same mistakes, only faster. Experiments have shown that you cannot train humans to avoid certain measurable, predictable statistical errors in reasoning. They just keep making them again and again. In the best case, they can avoid them only when they are using a computer program set up to integrate the data without making the mistake. These basic findings prove that qualitative improvements in intelligence are possible, and that all minds are not created equal.

A person with an IQ of 100 cannot understand certain concepts that people with an IQ of 140 can understand, no matter how many time and notebooks they have. Intelligence means being able to get the right answer the first time, not after a million tries. Even if you could program a human being to ape the understanding of superintelligent thoughts, they wouldn’t be able to come up with equivalent thoughts on their own or compare those thoughts to similarly complex thoughts.

8 Responses »

This idea sounds familiar. From the website of the US Air Force:

8/10/2009 - HANSCOM AIR FORCE BASE, Mass. (AFNS) — The convergence of “exponentially advancing technologies” will form a “super-intelligence” so formidable that it could avert war, according to one of the world’s leading futurists.

Dr. James Canton, CEO and chairman of the Institute for Global Futures, a San Francisco-based think tank, is author of the book “The Extreme Future” and an adviser to leading companies, the military and other government agencies.

He is consistently listed among the world’s leading speakers and has presented to diverse audiences around the globe.

It’s good to hear that the world’s leading futurists are slowly catching up to the position that I’ve been arguing for since 2001, when I was still a teenager.

Canton seems familiar with the singleton concept and views the US as rushing towards an unchallenged status:

“The superiority of convergent technologies will prevent war,” Doctor Canton said, claiming their power would present an overwhelming deterrent to potential adversaries. While saying that the U.S. will build these super systems faster and better than other nations, he acknowledged that a new arms race is already under way.

If things go as they have been, with no third parties entering the game, the US military could eventually create a superintelligence, but it will be a different beast than the hundred billion odd humans that came before it. A superintelligence is something fundamentally new. I predict that a superintelligence will only play games that it knows it can win, and will probably keep itself a relative secret until it’s already won. A superintelligence can hold bigger ideas in its head than you can. It depends heavily on what sort of superintelligence we’re talking about, but an AI-derived superintelligence in particular might be able to rapidly integrate spare processing power into its cognitive functions. Human working memory can only hold 5-7 items at once, a superintelligence’s working memory might be able to hold millions of complex symbols simultaneously.

Why no war?

“The fundamental macroeconomics on the planet favor peace, security, capitalism and prosperity,” he said. Doctor Canton projects that nations, including those not currently allied, will work together in using these smart technologies to prevent non-state actors from engaging in disruptive and deadly acts.

For the long-term, yes, but it seems like short-term war might be necessary to create a secure environment, in some cases. To a human-indifferent superintelligence with no “moral common sense”, but rather with a goal system hacked together in a half-assed way, a “secure environment” is one where all humans are dead and the world is arranged in precisely the way it wants, perhaps consisting of quadrillions of paper clips or computers containing animated gifs of smiley faces.

Now for the part that mentions advanced AI specifically:

“There’s no way for the human operator to look at an infinite number of data streams and extract meaning,” he said. “The question then is: How do we augment the human user with advanced artificial intelligence, better software presentation and better visual frameworks, to create a system that is situationally aware and can provide decision options for the human operator, faster than the human being can?”

He said he believes the answers can often be found already in what he calls ‘edge cultures.’

We got your edge culture right here. What was once only an obscure concern of a few transhumanists in the 90s has now become a mainstream interest among futurists, AI researchers, and even military strategists.

Canton continues:

Doctor Canton said he believes that more sophisticated artificial intelligence applications will transform business, warfare and life in general. Many of these are already embedded in systems or products, he says, even if people don’t know it.

In terms of robotics, he predicts “a real sea change” will come as we move from semi-autonomous to fully-autonomous units.

“That will be accompanied by a great debate, because of the ‘Terminator’ model,” he said. “It scares people.” But he doesn’t think people should be alarmed by the prospect of independently functioning robots.

He goes on to say that robots won’t be given superhuman intelligence, though superhuman intelligence will presumably come into existence in non-robotic platforms. What Canton needs to realize is that there is no clear division between non-robotic IT systems and robotic IT systems, and that division will continue to fade. Independently functioning robots are an inevitability, and if they aren’t infused with human-equivalent or human-surpassing kindness and morality, we are screwed.

“Robots will help fight and prevent wars,” he said, noting that they will have the ability to sense, analyze, authenticate and engage, but that humans will always be in position to check their power.

Ha ha ha ha, right. Our tribe will always be #1. No one can stand up to us. We da best. Go Team Human!

It’s not too late, Dr. Canton! Instead of sweeping the challenge of machine morality under the carpet, you can address it as the tangled problem that it is, and encourage the world to contribute resources to solving it.

2 Responses »

My disagreement with Dale Carrico, Mike Treder, James Hughes, Ray Kurzweil, Richard Jones, Charles Stross, Kevin Kelly, Max More, David Brin, and many others is relatively boring and straightforward, I think. It is simply this: I believe that it is possible that a being more powerful than the entirety of humanity could emerge in a relatively covert and quick way, and they don’t. A singleton, a Maximillian, an unrivaled superintelligence, a transcending upload, whatever you want to call it.

If you believe that such a being could be created and become unrivaled, then it is obvious that you would want to have some impact on its motivations. If you don’t, then clearly you would see such preparations to be silly and misguided.

Why do people make this more complex than it needs to be? It has nothing to do with politics. It has everything to do with our estimated probabilities of the likelihood of a more-powerful-than-humanity being emerging quickly. I am practically willing to concede all other points, because I think that this is the crux of the argument. Boring and simple, if I am indeed correct.

I am fairly confident that, at this point in history, superintelligence is the MacGuffin — the key element that determines how the story of humanity will go. I could be entirely wrong, of course, but that is my current position, and it is derived from cogsci and economics-based arguments about takeoff curves, not political nonsense. If it is wrong, it should be entirely simple to refute the hard takeoff hypothesis at the locus of cogsci and economics-based arguments rather than political or sociological arguments. Particularly, I think that James Hughes, as a sociologist, seems to have a desire to search for a “sociological” (social signaling/subcultural) explanation for other people’s beliefs, rather than looking at the economics/cogsci side of the arguments, which is their entire substance. You have to note that the people that believe in hard takeoff hypotheses are mostly subculturally isolated from one another, and barely even come into geographical contact. What wins us over are abstract arguments like, “humans are qualitatively smarter than chimps and have a huge advantage over them; why couldn’t there exist a superintelligence that has a similar qualitative advantage over us?”

16 Responses »

Following is a guest post from “S is for Singularity”, duplicated from Less Wrong.

In That alien message, Eliezer made some pretty wild claims:

My moral - that even Einstein did not come within a million light-years of making efficient use of sensory data.

Riemann invented his geometries before Einstein had a use for them; the physics of our universe is not that complicated in an absolute sense. A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis - perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration - by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.

They never suspected a thing. They weren’t very smart, you see, even before taking into account their slower rate of time. Their primitive equivalents of rationalists went around saying things like, “There’s a bound to how much information you can extract from sensory data.” And they never quite realized what it meant, that we were smarter than them, and thought faster.

In the comments, Will Pearson asked for “some form of proof of concept”. It seems that researchers at Cornell - Schmidt and Lipson - have done exactly that. See their video on Guardian Science:

‘Eureka machine’ can discover laws of nature - The machine formulates laws by observing the world and detecting patterns in the vast quantities of data it has collected

Researchers at Cambridge and Aberystwith have gone one step further and implemented an AI system/robot to perform scientific experiments:

Researchers at Aberystwyth University in Wales and England’s University of Cambridge report in Science today that they designed Adam - they describe how the bot operates by relating how he carried out one of his tasks, in this case to find out more about the genetic makeup of baker’s yeast Saccharomyces cerevisiae, an organism that scientists use to model more complex life systems. Using artificial intelligence, Adam hypothesized that certain genes in baker’s yeast code for specific enzymes that catalyze biochemical reactions. The robot devised experiments to test these beliefs, ran the experiments, and interpreted the results.

The crucial question is: what can we learn about the likely effectiveness of a “superintelligent” AI from the behavior of these AI programs? First of all, let us be clear: this AI is *not* a “superintelligence”, so we shouldn’t expect it to perform at that level. The problem we face is analogous to the problem of extrapolating how fast an Olympic sprinter can run from looking at a baby crawling around on the floor. Furthermore, the Cornell machine was given a physical system that was specifically chosen to be easy to analyze, and a representation (equations) that is known to be suited to the problem.

We can certainly state that the program analyzed some data much faster than any human could have done. In a running time probably measured in hours or minutes, it took a huge stream of raw position and velocity data and found the underlying conserved quantities. And given likely algorithmic optimizations and another 10 years’ of Moore’s law, we can safely say that in 10 years’ time, that particular program will run in seconds on a $500 machine or milliseconds on a supercomputer. These results actually surprise me: an AI can automatically and instantly analyze a physical system (albeit a rigged one).

But, of course, one has to ask: how much more narrow-AI work would it take to actually look at video of some bouncing, falling and whirling objects and deduce a general physical law such as the earth’s gravity and the laws governing air resistance, where the objects are not hand-picked to be easy to analyze? This is unclear. But I can see mechanisms whereby this would work, rather than merely having to submit to the overwhelming power of the word “superintelligence”. My suspicion is that with current state-of-the-art object identification technology, video footage of a system of bouncing balls and pendulums and springs would be amenable to this kind of analysis. There may even be a research project in that proposition.

As far as extrapolating the behavior of a superintelligence from the behavior of the Cornell AI or the Adam robot, we should note that no human can look at a complex physical system for a few seconds and just write down the physical law or equation that it obeys. A simple narrow AI has already outperformed humans at one specific task; though it still cannot do most of what a scientist does. We should therefore update our beliefs to assign more weight to the hypothesis that on some particular narrow physical modelling task, a “superintelligence” would vastly outperform us. Personally I was surprised at what such a simple system can do, though with hindsight it is obvious: data from a physical system follows patterns, and statistics can indentify those patterns. Science is not a magic ritual that only humans can perform, rather it is a specific kind of algorithm, and we should expect there to be no special injunction against silicon minds from doing it.

No Responses »

There was recently an article on Ray Kurzweil in Newsweek, titled, “I, Robot”. And… what’s this? An accompanying critical article, calling Singularitarians a cult, by our associate John Horgan, the guy who pretended to put together (with Zorpette, as implied in the back story) a neutral assessment of the Singularity for IEEE Spectrum that was actually just a bunch of indignant critics, plus Hanson, Vinge, Brooks, and a couple fluffy background articles. In the opener, Horgan says:

I once believed in the imminence of superhuman intelligence. In 1981, when I was still in college, I took a science-writing class at Columbia University from the journalist Pamela McCorduck. She had just written Machines Who Think (note the mischievous “Who”), a book about the efforts of Marvin Minsky and other artificial-intelligence pioneers to create conscious, autonomous computers that would leave mere humans in their cognitive dust. This research, which McCorduck often enthused over in class, helped persuade me to become a science journalist. What could be cooler than witnessing this giant leap forward in the evolution of consciousness?

I do agree that “Machines Who Think” is a misleading title, and that Minsky was wildly overoptimistic. Both the journalist and Marvin fell prey to anthropomorphism, where they saw more in their programs than was really there. We ought to correct this mistake and move on, not dismiss AI forever.

Here’s the conclusion:

Part of me—the part that thrilled at prospects for artificial intelligence almost 30 years ago—finds Kurzweil’s prophesies highly entertaining. He raises lots of provocative questions: What would be like to be immortal? To have an IQ of 1,000? To exist not as a doomed, flesh-and-blood creature but as a piece of software that can keep redesigning itself and merging with other programs? But another part of me—the grown-up, responsible part—worries that so many people, smart people, are taking Kurzweil’s sci-fi fantasies seriously. The last thing humanity needs right now is an apocalyptic cult masquerading as science.

Thank you for calling us smart! Perhaps our intelligence is being used to make the most accurate possible general assessments in light of available evidence.

I would also like to point out here that these “sci-fi fantasies” — molecular nanotechnology, artificial superintelligence, indefinite life-extension — are not just Kurzweil’s ideas, and were in fact around before he published his first futurist book. After all, it was as far back as 1948 that Mr. Ettinger published his first articles on preserving the human body and brain via cryonics. John W. Campbell published on superintelligence in 1935’s The Machine. All of medicine can be viewed as an effort at life-extension. If medicine could make people live for hundreds of years, most of us would take it gladly.

Here’s the first random comment in response to Horgan:

“Apocalyptic cult”?? I must have missed the part about the apocalypse - or the cult for that matter… That aside, you are basically saying that Kurzweil’s entire argument is absurd. Just because (it sounds like) you chased an unfounded dream many years too early, doesn’t mean that your jaded view of reality is really reality and that the people who subscribe to the ideas of a genius like Kurzweil are absurd. You’re either trying to be bold for selfish reasons or just plain bitter.

The “apocalyptic cult” seems to more be around my neck of the woods — the Singularity Institute and Lifeboat — than Kurzweil’s expansive fanbase in the rest of memespace. “Apocalyptic cult” means we are concerned about runaway self-improving AI, and may even have something in common with Bill Joy, who I think has a decent idea of the dangers, if somewhat hysterical, and with an unworkable solution. Still, if you don’t believe AI is possible for centuries, that’s fine — ignore us if you like — but don’t call us a cult, because we’re not.

It’s quite rude to call groups of obviously intelligent and freethinking people a cult, when we have 1) a commitment to rationalism and changing our minds, 2) a naturalistic worldview, 3) substantial uncertainty about many of the nodes in our probabilistic model, 4) are willing to put forth the work to actually develop AGI, no matter how long it takes, 5) operate on the expectation of human-derived causes rather than “the great thrust of history”-type nebulous causes, 6) see the positive/negative outcome as contingent on human action, 7) have no expectation of or desire for in-group perks if we do successfully build superintelligence, absence of religious trappings, 9) no revenge fantasies, a la true Rapturists, and 10) have no overblown anthropomorphism for the technological agents we are creating.

Particularly, at least in my view, the rigid timeline of Kurzweil for AGI by 2029 ought to be discarded. AGI could come about around then, as I actually think is somewhat plausible, or substantially before (as Peter Voss claims), or substantially after. The point is not when it is invented, but that when it is actually invented, it is programmed with human-friendly motivations that are stable under recursive self-improvement.

H/t to Barry Mahfood for the original pointer.

16 Responses »

Last September, Kevin Kelly posted a critique of a hard takeoff Singularity, based on what he calls “thinkism”:

As an essay called Why Work Toward the Singularity lets slip: “Even humans could probably solve those difficulties given hundreds of years to think about it.” In this approach one only has to think about problems smartly enough to solve them. I call that “thinkism.”

Let’s take curing cancer or prolonging longevity. These are problems that thinking along cannot solve. No amount of thinkism will discover how the cell ages, or how telomeres fall off. No intelligence, no matter how super duper, can figure out how human body works simply by reading all the known scientific literature in the world and then contemplating it. No super AI can simply think about all the current and past nuclear fission experiments and then come up with working nuclear fusion in a day. Between not knowing how things work and knowing how they work is a lot more than thinkism. There are tons of experiments in the real world which yields tons and tons of data that will be required to form the correct working hypothesis. Thinking about the potential data will not yield the correct data. Thinking is only part of science; maybe even a small part. We don’t have enough proper data to come close to solving the death problem. And in the case of living organisms, most of these experiments take calendar time. They take years, or months, or at least days, to get results. Thinkism may be instant for a super AI, but experimental results are not instant.

Interesting argument, and well-phrased.

But what about that recent story of a Cornell researcher Hod Lipson’s AI program that independently derived the laws of motion based on the swings of a pendulum, “a feat that took physicists centuries to complete”? According to Kelly’s thinkism hypothesis, that should be impossible.

And Lipson’s program is just the start of a whole field:

The research is being heralded as a potential breakthrough for science in the Petabyte Age, where computers try to find regularities in massive datasets that are too big and complex for the human mind. (See Wired magazine’s July 2008 cover story on “The End of Science.”)

Surely intelligence can achieve a lot. Solutions which take certain thinkers years to discover are uncovered in a short period of time by gifted experts. It is wrong to place solid limits on what a superior intelligence could do — could chimps predict what humans would be capable of with thinking alone? Of course, I could be wrong. Any intelligence might require extensive experimentation to generate knowledge. But the difference between my and Kelly’s position is that he sets hard limits based on speculating about intelligence fundamentally different than his own, while I acknowledge my basic uncertainty and say that I’m not entirely sure what thought alone is capable of. It could be capable of nearly everything, or it could only be capable of what it has achieved so far (a lot).

The conservative stance would be to assume that a new mind might be able to benefit from thought a great deal, and as such we should take great pains to ensure that all artificial intelligences above a certain level have human-friendly motivations. The “we have nothing to worry about, I guarantee it” stance would be to assume that thought is practically useless without thorough experimentation and to ignore the issue of human-friendly motivations on the pretense of indefinite human superiority and control. Sounds like the premise for yet another science fiction film where AIs get the jump on humans because we were overconfident.

Even if experimentation were required to glean knowledge, why would such experimentation be limited by the anthropocentric designation of “calendar time”? A nanoscale pendulum swinging in a vacuum demonstrate the same laws of motion as a large pendulum, and does so in a fraction of the time.

The human brain operates at about 200 Hz. Imagine hypothetical alien cultures where creatures evolved to have brains operating at 20 Hz or 2,000 Hz. Why would their advancement of science necessarily be limited by the “calendar time” of another intelligent species in an insignificant corner of the Milky Way Galaxy? Why would the limitations on knowledge to be gained from experiments be perfectly aligned with the inherent neural firing rates and “calendar time” of Homo sapiens? This is the Copernican Error of human-centrism — “if we’re a certain way, every other possible mind will have the same limitations, guaranteed”.

There are a variety of ways to boost one’s experimental output and data input. One obvious method is parallelism. Humans can only focus on one thing at a time, but an AI mind could focus on an unlimited number of experiments as long as it has the computing power and hardware. Another method would be miniaturization. The use of microarrays in biology research has made possible far less expensive and far more parallel experimentation than ever before.

I see Kelly’s position as anthropocentric triumphalism — we’re the greatest, no one can be as good as us, we have nothing to fear from a hard takeoff, any AI mind will need to engage in centuries of research to get as far as us. Sure, it might turn out to be true, but why put the human species on the line for this hypothesis?

12 Responses »

This was in the first issue of H+ magazine, but now it’s featured at the website:

Singularity 101 with Vernor Vinge

My stance on Vinge’s position on the Singularity is that he’s too loose with the concept, and seems to slightly welcome people redefining it as agriculture or something else that has no relationship to the concept of smarter-than-human intelligence.

I actually agree with Vernor when he says he’d be surprised if the Singularity doesn’t happen by 2030. It could very well happen after then, but still, I’d be surprised.

He also seems to take a morally detached view of the Singularity — like, “The Singularity is something that could affect everyone on the planet for the profoundly better or worse, but I prefer to view it as an abstract intellectual concept rather than something that will actually affect people.”

6 Responses »

In case you haven’t seen it before, there’s an interview from 2007 with Hofstadter at the American Scientist that goes over what he’s been up to. The interesting part from the point of view of this Singularitarian is where he talks about the Singularity and Kurzweil:

There’s a popular idea currently that technology may be converging on some kind of culmination—some people refer to it as a singularity. It’s not clear what form it might take, but some have suggested an explosion of artificial intelligence. Do you have any thoughts about that?

Oh, yeah, I’ve organized several symposia about it; I’ve written a long article about it; I’ve participated in a couple of events with Ray Kurzweil, Hans Moravec and many of these singularitarians, as they refer to themselves. I have wallowed in this mud very much. However, if you’re asking for a clear judgment, I think it’s very murky.

The reason I have injected myself into that world, unsavory though I find it in many ways, is that I think that it’s a very confusing thing that they’re suggesting. If you read Ray Kurzweil’s books and Hans Moravec’s, what I find is that it’s a very bizarre mixture of ideas that are solid and good with ideas that are crazy. It’s as if you took a lot of very good food and some dog excrement and blended it all up so that you can’t possibly figure out what’s good or bad. It’s an intimate mixture of rubbish and good ideas, and it’s very hard to disentangle the two, because these are smart people; they’re not stupid.

Ray Kurzweil says 2029 is the year that a computer will pass the Turing test [converse well enough to pass as human], and he has a big bet on it for $1,000 with [Lotus Software founder Mitch Kapor], who says it won’t pass. Kurzweil is committed to this viewpoint, but that’s only the beginning. He says within 10 or 15 years after that, a thousand dollars will buy you computational power that will be equivalent to all of humanity. What does it mean to talk about $1,000 when humanity has been superseded and the whole idea of humans is already down the drain?

This is an interesting reaction that, like many reactions, probably describes the views of hundreds of thousands of silent people. Notice how Hofstadter has a visceral emotional aversion to Kurzweil’s ideas — “mud”, “unsavory”, etc. This is something of a contrast to views like those of Dr. Richard Jones, who is far more subtle and interesting:

The difficulty, then, is not that there is no science underlying the claims Kurzweil makes, nor that this science isn’t very exciting on its own terms. It’s that this science can’t sustain the sweeping claims and (especially) the fast timescales that Kurzweil insists on.

If I’m criticizing Kurzweil, this is more along the lines where I prefer to tread, rather than the visceral aversion theme, which is obviously to be expected.

But Hofstadter comes up with an interesting point, which is “What does it mean to talk about $1,000 when humanity has been superseded and the whole idea of humans is already down the drain?” This invokes a line of reasoning by Kurzweil that we’ve seen on several occasions, namely that life and the world would change after the Singularity, but not that much. Other data points:

1) Kurzweil claims that Moore’s law and his other exponential trends will continue to progress at a predictable rate, continuous with their progress in a human-only society, even when human-level AI and neural-enhancement nanobots are introduced around 2029. (According to his book.) These will presumably not accelerate technology by a discontinuous multiplier, because that would throw off the smooth exponential curve.

2) Kurzweil seems to place an awful lot of emphasis on the role of sexuality and throwing off the tyranny of gender in the post-Singularity world. Yet, if we discard most of the trappings of our biology, might we not choose to associate activities other than sex with extreme pleasure? (By directly reprogramming our brains.) And wouldn’t a sex change be quite pedestrian in a world where we have complete morphological freedom to transform ourselves into practically anything we want? Isn’t this is sort of thing we’d try out in the first week, then move on to completely new and barely imaginable realms shortly afterward? This angle is somewhat improved in The Singularity is Near over The Age of Spiritual Machines.

3) Kurzweil’s de-emphasis of the Event Horizon of Vinge, instead preferring to cast the post-AGI future in ways that us pre-Singularity folks can comprehend, referencing specific humanly-imaginable technologies and encouraging his readers to associate the Singularity concept with these technologies. This aim is seemingly reinforced by Singularity University.

4) AGI appears in 2029, but “the Singularity” doesn’t happen until 2045. What happens in between those years? Some corny, pre-Singularity fiction like Accelerando? I have a sneaking suspicion that this vision — AGI exists, but fits in smoothly with preexisting society just like human newborns or modern-day computers — is what much of the public thinks that people like myself are talking about when I say “Singularity”. In reality, what I think of is more like the Romantic past touched up with barely-behind-the-scenes advanced technology that feels more organic than anything else. (At least initially, because I think that is what humans want, and will deliberately choose once we have the technology.)

For me, when the Singularity concept really clicked is when I heard it being described in terms of genuine cognitive improvement and extensive dwelling on what that means, not talking about futuristic technologies we already can easily foresee and would be developed anyway in a human-only world, no superintelligence required. One line from “Staring into the Singularity” goes like this:

Smartness is that quality which makes it impossible to write a story about a character smarter than you are. You can write about super-fast thinkers, eidetic memories, lightning calculators; a character who learns a dozen languages in a week, who can read a textbook in an hour, or who can invent all kinds of wonderful stuff - as long as you don’t have to produce the actual invention. But you can’t write a character with a higher level of emotional maturity, a character who can spot the obvious solution you missed, a character who knows (and can tell the reader) the Meaning Of Life, a character with superhuman self-awareness. Not unless you can do these things yourself.

This is what makes the Singularity intellectually interesting to me. Genuinely enhanced intelligence and awareness. Not technology for technology’s sake.

As I’ve said before, I think that the survival or defeat of humanity in the 21st century almost exclusively depends on how we handle the first self-improving superintelligence. Therefore, while I have substantial interest in new and exciting technologies, I tend to divide them strongly into two categories: technologies that may have an impact on creating superintelligence and those that don’t. Technologies that do are extremely important. Those that don’t are interesting to think about, but more of an intellectual exercise than anything else. However, the scope of technologies involved in the possible creation is superintelligence is substantially wider than it appear at first glance. This is partially because there are several distinct types of superintelligence that may be produced first. Overall, this increases the probability that any one type of superintelligence will be invented in the next 20-50 years. When one is generated, it will soon lead to all the others.

9 Responses »