The Singularity is Far: A Neuroscientist’s View
I haven't read this, I'm just posting it because other people are talking about it.
Ray Kurzweil, the prominent inventor and futurist, can't wait to get nanobots into his brain. In his view, these devices will be equipped with a variety of sensors and stimulators and will communicate wirelessly with computers outside of the body. In addition to providing unprecedented insight into brain function at the cellular level, brain-penetrating nanobots would provide the ultimate virtual reality experience.
The Last Post Was an Experiment
+1 for everyone who saw through my lie.
I thought it would be interesting to say stuff not aligned with what I believe to see the reaction.
The original prompt is that I was sort of wondering why no one was contributing to our Humanity+ matching challenge grant.
Maybe because many futurist-oriented people don't think transhumanism is very important.
They're wrong. Without a movement, the techno-savvy and existential risk mitigators are just a bunch of unconnected chumps, or in isolated little cells of 4-5 people. With a movement, hundreds or even thousands of people can provide many thousands of dollars worth of mutual value in "consulting" and work cooperation to one another on a regular basis, which gives us the power to spread our ideas and stand up to competing movements, like Born Again bioconservatism, which would have us all die by age 110.
I believe the "Groucho Marxes" -- who "won't join any club that will have them" are sidelining themselves from history. Organized transhumanism is very important.
I thought quoting Margaret Somerville would pretty much give it away, but apparently not.
To me, cybernetics etc. are just a tiny skin on the peach that is the Singularity and the post-Singularity world. To my mind, SL4 transhumanism is pretty damn cool and important. I've written hundreds of thousands of words for why I think so, but there must be something I'm missing.
To quote Peter Thiel, those not looking closely at the Singularity and the potentially discontinuous impacts of AI are "living in a fantasy world".
The Benefits of a Successful Singularity
What is the point of a beneficial Singularity? A challenging question because there are so many potential benefits. Some of the benefits I enjoy more might not be the same as the benefits you would enjoy. People can disagree.
What kind of Singularity happens depends on what kind of singleton we end up with, but we can wistful and optimistic, right? The Singularity I'm working towards would have the following components:
1) Invention of molecular nanotechnology or superior manufacturing technology, enabling the production of near-unlimited food, housing, clean water, and other products.
2) Enforcement of local "volitional bubbles" that reduce the rate of non-consensual violent crime to zero. I'd be curious to see how altruistic superintelligence or the CEV output would handle cases where people join "fight clubs" where the risk of death is part of the bylaws.
3) Unless the current overall system is objectively optimal even to an altruistic superintelligence, presumably this would be rearranged for the better as well, though exactly how and in light of what drives and freedoms is hard to say. Probably this won't be a straightforward extension of the politics of the 21st century, like how human politics isn't a straightforward extension of conodont politics.
4) Possible amplification and diversification of every single object, skill, or practice. So instead of a few general different types of asteroids they might become 10^12 different kinds of asteroids. At first I thought of saying "amplification and diversification of everything of which intelligence is capable", but why not amplify and diversify everything in the entire cosmos? This would include art, music, aesthetics, "dance", communication, "philosophy", world building, etc.
5) Presumably, if the Church-Turing thesis is true and phenomenally conscious uploads are possible, then the mass conversion of matter into "computronium", though maybe not all matter. The simple reason why is that this would allow more space, more joy, more possible experiences, more security, etc. If phenomenally conscious uploads are not possible, then similar actions in the same space might include making space colonies, or hollowing out the underground and pumping water, air, and sunlight down to create vast new living spaces.
6) The possibility of guided transformation of willing humans to superintelligence, through pathways determined to be of less "risk", i.e., wireheading. How risk is defined will be partially subjective and partially objective, like most things.
7) Thoughtful preservation of the outlines of existing human societies and cultures (minus violence presumably, which is a central part of many cultures) by those who wish to do so. This would be in contrast to the default today, which is the disintegration of most cultures and integration into Anglosphere or east Asian hegemons.
8) The possibility of eliminating suffering and exploring "gradients of bliss" as everyday reality, along the lines of the Hedonistic Imperative. We might find that it is pleasant to increase our happiness set-points somewhat, or possibly even experiment with lowering them temporarily. We might find that it is pleasant to be in a state of revelatory or orgiastic bliss non-stop, or maybe not. A Singularity would at least give us the option of exploring those possibilities.
9) The potential of dispelling the mystery of human interactions by using "x-ray glasses" (advanced analytical AI linked directly to our brains, or part of our brains) to see their complex internal structure in a deterministic fashion, if only briefly, using fine-grained simulations. From the perspective of a higher intelligence, to what extent is human nature truly computationally "chaotic" in the sense of chaos theory, and to what extent is it entirely deterministic, like a Newtonian universe? We may be surprised by the answer.
10) Pursuit of higher aesthetics and moralities beyond the human realm. Hopefully whatever singleton we are stuck with still allows a wide berth for personal experimentation, perhaps with supervision by AI experts who have already "been there". Surely there must exist interesting value systems and aesthetic points of view which we would be quite excited to experience which we have neither yet seen nor even thought of yet.
11) Exploration of the entire space of thoughts not only directly adjacent to the human realm, but also far beyond it. It sounds pretty simple to just state a sentence like that, but in practice such fringe ideas can instill powerful feelings of confusion, conflicting intuitions, and even awe when they turn out to be correct or useful. The purpose is not just exploring thoughtspace for its own sake but for the complex manifold of emotional and intellectual interactions that emerge from connecting disparate concepts and exploring the "outer solar system" of thought.
12) Most seriously and urgently, ending the orgy of killing and torture upon the human and conscious animal species by man, other animals, disease, environmental factors, and "other".
The Final Weapon
It's not really "fair", but history generally consists of people getting better and better weapons, and whoever has the best weapons and the best armies makes the rules. The number of historical examples of this phenomenon are practically unlimited. The reason America is respected and feared today is because of our military capabilities, particularly nuclear weapons. Complain if you want, this is reality.
I am excited by the possibility that the 200,000 year arms race will finally come to an end by a singleton. It had to end sometime. Personally, it will be a relief, if we survive. While many people can happily enjoy their lives on a daily basis, just focusing on their tiny sphere, myself and others are cursed with concerns about the overall trajectory of humanity and human conflict. My relationship with Murphy's law is so close that I would hardly be surprised to hear the detonation of nuclear weapons in the distance, practically anytime or anywhere.
Nuclear weapons, of course, are toys in comparison to the products of MNT, or worse yet, true superintelligence. MNT could enable the creation of "fingers of god", large satellites that condense sunlight to deliver terrific beams to the surface. A square mile of space-based solar panels, properly utilized, would provide enough energy to set entire districts alight. Anti-satellite missiles could be quickly detected by their thermal signature and neutralized long before reaching the target.
In comparison to the weapons of the future, nuclear weaponry is quite mundane, which makes living in 2011 less distressing than living in the future. However, there's plenty to look forward to. The world's most sophisticated militaries are aggressively pursuing AI technology for robotic warfighters. The problem with mindless AI soldiers is that it decouples the delivery of force from the ostensible wisdom and benevolence of human actors. This is the problem with nukes as well -- for all of Reagan's faults, at least he was terrified of nuclear war, and would have done anything he could to avoid one. One can only hope Obama would have the same attitude. Bush Sr. probably would have as well, but who knows? Eventually there will be a President that won't hesitate to press the button. I have sympathy for Robin Hanson's position that he votes for Presidents entirely based on their propensity to initiate wars or not. Everything else is truly secondary.
It might be overly masculine of me, but I see the Singularity as the final chapter in this ongoing and stressful arms race called human history. A recursively self-improving AI is the way of securing the highest power available in this time and place, and directing it towards constructive ends. Once we press the big red button, that's that. It won't matter so much who did it, as the effects that flow from that cause. It's easy to imagine AI or human goals drifting under self-improvement to the point where the singleton doesn't really care whether we're around or not, and starts sucking up all the air or free energy for its own purposes. After all, our planet is tiny. Truly bite-size from the perspective of a self-improving superintelligence, it could be consumed in a matter of days.
Transhumanism itself is secondary to this event. In retrospect, no one will care so much about cyborgs. They will care about recursively self-improving superintelligence undergoing a hard takeoff. Cyborgs, implants, gene therapy, life extension, seasteading, the Internet -- these are toys. Superintelligence is what really matters. I'm truly happy that more people are beginning to understand that.
Replying to Alex Knapp, July 2nd
Does Knapp know anything about the way existing AI works? It’s not based around trying to copy humans, but often around improving this abstract mathematical quality called inference.
I think you missed my point. My point is not that AI has to emulate how the brain works, but rather that before you can design a generalized artificial intelligence, you have to have at least a rough idea of what you mean by that. Right now, the mechanics of general intelligence in humans are, actually, mostly unknown.
What’s become an interesting area of study in the past two decades are two fascinating strands of neuroscience. The first is that animal brains and intelligence are much better and more complicated than we thought even in the 80s.
The second is that humans, on a macro level, think very differently from animals, even the smartest problem solving animals. We haven’t begun to scratch the surface.
Based on the cognitive science reading I've done up to this point, this is false. Every year, scientists discover cognitive abilities in animals that were previously thought to be uniquely human, such as episodic memory or the ability to deliberately trigger traps. Chimps have a "near-human understanding of fire" and complex planning abilities. Articles such as this one in Discover, "Are Humans Really Any Different from Other Animals?", and this one in New Scientist, "We're not unique, just at one end of the spectrum" are typical from scientists who compare human and chimp cognition. It's practically become a trope for the (often religious) person to say humans and animals are completely different, and the primatologist or cognitive scientist to say, "not nearly as much as you think..."
One primate biologist says this:
"If we really want to talk about the big differences between humans and chimps — they're covered in hair and we're not," Taglialatela told LiveScience. "Their brains are about one-third the size of humans'. But the major differences come down to ones of degree, not of kind."
There's a really good paper somewhere out there on cognitive capacities in humans and chimps and how human cognitive abilities seem to be exaggerations of chimp abilities rather than different in kind, but I can't find it.
Arguments that chimps and humans are fundamentally different tend to be found more often on Christian apologetics sites than in scientific papers or articles. The overall impression I get is that scientists think chimp cognition and human cognition are different in degree, not in kind. There are humans out there so dumb that chimps are probably more clever than them in many important dimensions. Certainly if Homo heidelbergensis and Neanderthals were walking around, we would have even more evidence that the difference between humans and chimps is one of degree, not kind.
Another point is that even if humans were radically different in thinking than animals, why would that automatically mean AI is more difficult? We already have AI that utterly defeats humans in narrow domains traditionally seen as representative of complex thought, no magical insights necessary.
Yet another possibility is one of AI that very effectively gathers resources and builds copies of itself, yet does not do art or music. An AI that lacks many dimensions of human thought could still be a major concern with the right competencies.
But before scientists knew anything about birds, we basically knew: (a) they can fly, (b) it has something to do with wings and (c) possibly the feathers, too. At that stage, you couldn’t begin to design a plane. It’s the same way with human intelligence. Very simplistically, we know that (a) humans have generalized intelligence, (b) it has something to do with the brain and (c) possibly the endocrine system as well.
I should think that many tens of thousands of cognitive scientists would object to the suggestion that we only know a "few basic things" about intelligence. However, it's quite subjective and under some interpretations I would agree with you.
The above paragraph is a vast oversimplification, obviously, but the point is to analogize. Right now, we’re at the “wings and feathers†stage of understanding the science of intelligence. So I find it unlikely that a solution can be engineered until we understand more of what intelligence is.
The impression that one has here probably correlates with how much cognitive science you read. If you read a lot, then it's hard not to think of all that we do know about intelligence. Plenty is unknown, but we don't know how much more needs to be known to build AI. It could be a little, it could be a lot -- we have to keep experimenting and trying to build general AI.
Now, once we understand intelligence, and if (and I think this is a big if), it can be reproduced in silicon, then the resulting AGI probably doesn’t necessarily have to look like the brain, anymore than a plane looks like a bird. But the fundamental principles still have to be addressed. And we’re just not there yet.
Yet formalisms of intelligence, like Solmonoff induction, are not particularly algorithmically complicated, just computationally expensive. Gigerenzer and colleagues have shown that many aspects of human decision making rely on "fast and frugal heuristics" that are so simple they can be described in pithy phrases like Take the Best and Take the First. Robyn Dawes has shown how improper linear models regularly outperform "expert" predictors, including medical doctors. Rather than possessing a surplus of cognitive tools for addressing problems and challenges, humans seem to just possess a surplus of overconfidence and arrogance. It is easy to invent problems that humans cannot solve without computer help. Humans are notoriously bad at paying attention to base rates, for instance, even though base rates tend to be the most epistemologically important variable in any reasoning problem. After you read about many dozens of experiments in heuristics and biases research where people embarrass themselves in spectacular fashion, you start to roll your eyes a bit more when people gloat about the primacy of human reasoning.
I correspond with lots of neuroscientists. Virtually all of them tell me that the big questions remain unanswered and will for quite some time.
I correspond with neuroscientists who believe that the brain is complex but that exponentially better tools are helping quickly elucidate many of the important questions. Regardless, AI might be a matter of computer science, not cognitive science. Have you considered that possibility?
AIXI is a thought experiment, not an AI model. It’s not even designed to operate in a world with the constraints of our physical laws.
Sure it is. AIXI is "a Bayesian optimality notion for general reinforcement learning agents", a yardstick that finite systems can compare against. It may be that the only reason our brains work at all is because they are approximations of AIXI.
My point is to recognize that the way machine intelligence operates, and will for the conceivable future, is in a manner that is complementary to human intelligence. And I’m fine with that. I’m excited by AI research. I just find it unlikely, given the restraints of physical laws as we understand them today, that an AGI can be expected in the near term, if ever.
"If ever"? You must be joking. That's like saying, "I just find it unlikely, given the restraints of physical laws as we understand them today, that a theory of the vital force that animates animate objects can be expected in the near term, if ever", or "I just find it unlikely, given the restraints of physical laws as we understand them today, that a theory of aerodynamics that can produce heavier-than-air flying machines can be expected in the near term, if ever". Why would science figure out how everything else works, but not the mind? You're setting the mind apart from everything else in nature in a semi-mystical way, in my view.
I am, however, excited at the prospect of using computers to free humans from grunt work drudgery that computers are better at, so humans can focus on the kinds of thinking that they’re good at.
To be pithy, I would argue that humans suck at all kinds of thinking, and any systems that help us approach Bayesian optimality are extremely valuable because humans are so often wrong and overconfident in many problem domains. Our overconfidence in our own reasoning even when it explicitly violates the axioms of probability theory routinely reaches comic levels. In human thinking, 1 + 1 really can equal 3. Probabilities don't add up to 100%. Events with base rates of ~0.00001%, like fatal airplane crashes, are treated as if their probabilities were thousands of times the actual value. Even the stupidest AIs have a tremendous amount to teach us.
The problem with humans is that we are programmed to violate Bayesian optimality routinely with half-assed heuristics that we inherited because they are "good enough" to keep us alive long enough to reproduce and avoid getting murdered by conspecifics. With AI, you can build a brain that is naturally Bayesian -- it wouldn't have to furrow its brow and try real hard to obey simple probability theory axioms.
Singularity Summit 2011
The press release for SS11 is posted. Featuring Ken Jennings, Christof Koch, Tyler Cowen, Ray Kurzweil, and many others. The venue will be the same as 2009 -- the 92nd St. Y in New York City.
The theme we are pegging this year's conference to is the Watson victory.
Registration is here, register early because price will go up:
https://www.singularitysummit.com/registration/
October 15-16, 2011. Press passes are available for actual press, ask me about it.
If the Singularity movement is an outgrowth of the Russian Orthodox Church, as Stross argues, then this would be our Pascha.
Ben Goertzel Scolds Alex Knapp for Calling People Who Advocate Approaches to AGI other than Brain Emulation “Magical Thinkers”
In the comments thread, Ben Goertzel scolds Alex Knapp:
Heh… thanks for this post Alex, it helped me understand your world-view a lot better.
You previously wrote an article accusing one of my H+ Magazine articles of “magical thinking†— and in this article, you use the same phrase to jab at Michael Anissimov…
Calling those of us working on strong AI from approaches other than brain emulation “magical thinkers†is incorrect, obnoxious, and (sorry to be harsh, but…) poor journalism. It’s the kind of opinionated, non-fact-based journalism more appropriate for the politics or arts page than the science page, IMO.
It’s certainly your right to be skeptical of the possibility of creating human-level AI via non-brain-emulative methods. Many serious scientists are. However, there are also some serious scientific arguments as to why non-brain-emulative human-level AI **may** be possible within a few decades of work.
If you are not familiar with these arguments, so be it. You’re just one man, and you write about an impressive variety of topics. But that doesn’t make it right for you to insult the rationality, scientific-ality or intellectual honesty of those of us scientists and engineers pursuing non-brain-emulative AGI.
I might be wrong about the possibility of non-brain-emulative human-level AI, but if so, it’s not because of engaging in “magical thinking.†And nor is Michael Anissimov (who I know fairly well) engaging in thinking of that nature.
I’m disappointed. You seem an intelligent person, and you share a lot of interests with those of us in the Singularitarian world. And you’re a good writer, with the ability to turn out an amazing diversity of sci-tech articles each day. But then you sink to the level of ad hominem attacks against the thought processes of individuals whose views differ from your own! Tsk, tsk, tsk…
Alex, I can't wait for your next post on brain emulation! Not as good as Dale Carrico's writing, I must admit, but still entertaining in the same vein.
Responding to Alex Knapp at Forbes
From Mr. Knapp's recent post:
If Stross’ objections turn out to be a problem in AI development, the “workaround†is to create generally intelligent AI that doesn’t depend on primate embodiment or adaptations. Couldn’t the above argument also be used to argue that Deep Blue could never play human-level chess, or that Watson could never do human-level Jeopardy?But Anissmov’s first point here is just magical thinking. At the present time, a lot of the ways that human beings think is simply unknown. To argue that we can simply “workaround†the issue misses the underlying point that we can’t yet quantify the difference between human intelligence and machine intelligence. Indeed, it’s become pretty clear that even human thinking and animal thinking is quite different. For example, it’s clear that apes, octopii, dolphins and even parrots are, to certain degrees quite intelligent and capable of using logical reasoning to solve problems. But their intelligence is sharply different than that of humans. And I don’t mean on a different level — I mean actually different. On this point, I’d highly recommend reading Temple Grandin, who’s done some brilliant work on how animals and neurotypical humans are starkly different in their perceptions of the same environment.
My first point is hardly magical thinking -- all of machine learning works to create learning systems that do not copy the animal learning process, which is only even known on a vague level. Does Knapp know anything about the way existing AI works? It's not based around trying to copy humans, but often around improving this abstract mathematical quality called inference. (Sometimes just around making a collection of heuristics and custom-built algorithms, but again that isn't copying humans.) Approximations Solomonoff induction works quite well on a variety of problems, regardless of the state of comparing human and machine intelligence. Many "AI would have to be exactly like humans to work, because humans are so awesome, so there" proponents, like Knapp and Stross, talk as if Solomonoff induction doesn't exist.
Answering how much or how little of the human brain is known is quite a subjective question. The MIT Encyclopedia of Cognitive Sciences is over 1,000 pages and full of information about how the brain works. The Bayesian Brain is another tome that discusses how the brain works, mathematically:
A Bayesian approach can contribute to an understanding of the brain on multiple levels, by giving normative predictions about how an ideal sensory system should combine prior knowledge and observation, by providing mechanistic interpretation of the dynamic functioning of the brain circuit, and by suggesting optimal ways of deciphering experimental data. Bayesian Brain brings together contributions from both experimental and theoretical neuroscientists that examine the brain mechanisms of perception, decision making, and motor control according to the concepts of Bayesian estimation.
After an overview of the mathematical concepts, including Bayes' theorem, that are basic to understanding the approaches discussed, contributors discuss how Bayesian concepts can be used for interpretation of such neurobiological data as neural spikes and functional brain imaging. Next, contributors examine the modeling of sensory processing, including the neural coding of information about the outside world. Finally, contributors explore dynamic processes for proper behaviors, including the mathematics of the speed and accuracy of perceptual decisions and neural models of belief propagation.
The fundamentals of how the brain works, as far as I see, are known, not unknown. We know that neurons fire in Bayesian patterns in response to external stimuli and internal connection weights. We know the brain is divided up into functional modules, and have a quite detailed understanding of certain modules, like the visual cortex. We know enough about the hippocampus in animals that scientists have recreated a part of it to restore rat memory.
Intelligence is a type of functionality, like the ability to take long jumps, but far more complicated. It's not mystically different than any other form of complex specialized behavior -- it's still based around noisy neural firing patterns in the brain. To say that we have to exactly copy a human brain to produce true intelligence, if that is what Knapp and Stross are thinking, is anthropocentric in the extreme. Did we need to copy a bird to produce flight? Did we need to copy a fish to produce a submarine? Did we need to copy a horse to produce a car? No, no, and no. Intelligence is not mystically different.
We already have a model for AI that is absolutely nothing like a human -- AIXI.
Being able to quantify the difference between human and machine intelligence would be helpful for machine learning, but I'm not sure why it would be absolutely necessary for any form of progress.
As for universal measures of intelligence, here's Shane Legg taking a stab at it:
Even if we aren't there yet, Knapp and Stross should be cheering on the incremental effort, not standing on the sidelines and frowning, making toasts to the eternal superiority of Homo sapiens sapiens. Wherever AI is today, can't we agree that we should make responsible effort towards beneficial AI? Isn't that important? Even if we think true AI is a million years away because if it were closer then that would mean that human intelligence isn't as complicated and mystical as we had wished?
As to Anissmov’s second point, it’s definitely worth noting that computers don’t play “human-level†chess. Although computers are competitive with grandmasters, they aren’t truly intelligent in a general sense – they are, basically, chess-solving machines. And while they’re superior at tactics, they are woefully deficient at strategy, which is why grandmasters still win against/draw against computers.
This is true, but who cares? I didn't say they were truly intelligent in the general sense. That's what is being worked towards, though.
Now, I don’t doubt that computers are going to get better and smarter in the coming decades. But there are more than a few limitations on human-level AI, not the least of which are the actual physical limitations coming with the end of Moore’s Law and the simple fact that, in the realm of science, we’re only just beginning to understand what intelligence, consciousness, and sentience even are, and that’s going to be a fundamental limitation on artificial intelligence for a long time to come. Personally, I think that’s going to be the case for centuries.
Let's build a computer with true intelligence first, and worry about "consciousness" and "sentience" later, then.
Forbes Blogger Alex Knapp on “What is the Likelihood of the Singularity?”
Alex Knapp over at Forbes is writing a series of blog posts around Charles Stross' recent Singularity criticisms. Knapp goes after my last post pretty enthusiastically, so check it out.
Response to Charles Stross’ “Three arguments against the Singularity”
super-intelligent AI is unlikely because, if you pursue Vernor's program, you get there incrementally by way of human-equivalent AI, and human-equivalent AI is unlikely. The reason it's unlikely is that human intelligence is an emergent phenomenon of human physiology, and it only survived the filtering effect of evolution by enhancing human survival fitness in some way. Enhancements to primate evolutionary fitness are not much use to a machine, or to people who want to extract useful payback (in the shape of work) from a machine they spent lots of time and effort developing. We may want machines that can recognize and respond to our motivations and needs, but we're likely to leave out the annoying bits, like needing to sleep for roughly 30% of the time, being lazy or emotionally unstable, and having motivations of its own.
"Human-equivalent AI is unlikely" is a ridiculous comment. Human level AI is extremely likely by 2060, if ever. (I'll explain why in the next post.) Stross might not understand that the term "human-equivalent AI" always means AI of human-equivalent general intelligence, never "exactly like a human being in every way".
If Stross' objections turn out to be a problem in AI development, the "workaround" is to create generally intelligent AI that doesn't depend on primate embodiment or adaptations.
Couldn't the above argument also be used to argue that Deep Blue could never play human-level chess, or that Watson could never do human-level Jeopardy?
I don't get the point of the last couple sentences. Why not just pursue general intelligence rather than "enhancements to primate evolutionary fitness", then? The concept of having "motivations of its own" seems kind of hazy. If the AI is handing me my ass in Starcraft 2, does it matter if people debate whether it has "motivations of its own"? What does "motivations of its own" even mean? Does "motivations" secretly mean "motivations of human-level complexity"?
I do have to say, this is a novel argument that Stross is forwarding. Haven't heard that one before. As far as I know, Stross must be one of the only non-religious thinkers who believes human-level AI is "unlikely", presumably indefinitely "unlikely". In a literature search I conducted in 2008 looking for academic arguments against human-level AI, I didn't find much -- mainly just Dreyfuss' What Computers Can't Do and the people who argued against Kurzweil in Are We Spiritual Machines? "Human level AI is unlikely" is one of those ideas that Romantics and non-materialists find appealing emotionally, but backing it up is another matter.
(This is all aside from the gigantic can of worms that is the ethical status of artificial intelligence; if we ascribe the value inherent in human existence to conscious intelligence, then before creating a conscious artificial intelligence we have to ask if we're creating an entity deserving of rights. Is it murder to shut down a software process that is in some sense "conscious"? Is it genocide to use genetic algorithms to evolve software agents towards consciousness? These are huge show-stoppers — it's possible that just as destructive research on human embryos is tightly regulated and restricted, we may find it socially desirable to restrict destructive research on borderline autonomous intelligences ... lest we inadvertently open the door to inhumane uses of human beings as well.)
I don't think these are "showstoppers" -- there is no government on Earth that could search every computer for lines of code that are possibly AIs. We are willing to do whatever it takes, within reason, to get a positive Singularity. Governments are not going to stop us. If one country shuts us down, we go to another country.
We clearly want machines that perform human-like tasks. We want computers that recognize our language and motivations and can take hints, rather than requiring instructions enumerated in mind-numbingly tedious detail. But whether we want them to be conscious and volitional is another question entirely. I don't want my self-driving car to argue with me about where we want to go today. I don't want my robot housekeeper to spend all its time in front of the TV watching contact sports or music videos.
All it takes is for some people to build a "volitional" AI and there you have it. Even if 99% of AIs are tools, there are organizations -- like the Singularity Institute -- working towards AIs that are more than tools.
If the subject of consciousness is not intrinsically pinned to the conscious platform, but can be arbitrarily re-targeted, then we may want AIs that focus reflexively on the needs of the humans they are assigned to — in other words, their sense of self is focussed on us, rather than internally. They perceive our needs as being their needs, with no internal sense of self to compete with our requirements. While such an AI might accidentally jeopardize its human's well-being, it's no more likely to deliberately turn on it's external "self" than you or I are to shoot ourselves in the head. And it's no more likely to try to bootstrap itself to a higher level of intelligence that has different motivational parameters than your right hand is likely to grow a motorcycle and go zooming off to explore the world around it without you.
YOU want AI to be like this. WE want AIs that do "try to bootstrap [themselves]" to a "higher level". Just because you don't want it doesn't mean that we won't build it.
Steve Wozniak a Singularitarian?
Apple co-founder Steve Wozniak has seen so much stunning technological advances that he believes a day will come when computers and humans become virtually equal but with machines having a slight advantage on intelligence.
Speaking at a business summit held at the Gold Coast on Friday, the once co-equal of Steve Jobs in Apple Computers told his Australian audience that the world is nearing the likelihood that computer brains will equal the cerebral prowess of humans.
When that time comes, Wozniak said that humans will generally withdraw into a life where they will be pampered into a system almost perfected by machines, serving their whims and effectively reducing the average men and women into human pets.
Widely regarded as one of the innovators of personal computing with his works on putting together the initial hardware offerings of Apple, Wozniak declared to his audience that "we're already creating the superior beings, I think we lost the battle to the machines long ago."
I always think of this guy when I go by Woz Way in San Jose.
So, if artificial intelligence can become smarter than humans, shouldn't we be concerned about maximizing the probability of a positive outcome, instead of just saying that AI will definitely do X and that there's nothing we can do about it, or engaging in some juvenile fantasy that we humans can directly control all AIs forever? (We can indirectly "control" AI by setting its initial conditions favorably, that is all we can do, the alternative is to ignore the initial conditions.)
Hard Takeoff Sources
Definition of "hard takeoff" (noun) from Transhumanist Wiki:
The Singularity scenario in which a mind makes the transition from prehuman or human-equivalent intelligence to strong transhumanity or superintelligence over the course of days or hours (Yudkowsky 2001). The high likelihood of a hard takeoff once a roughly human-equivalent AI is created has been argued by the Singularity Institute in Yudkowsky 2003.
Hard takeoff sources and references, which includes hard science fiction novels, academic papers, and a few short articles and interviews:
Blood Music (1985) by Greg Bear
Fire Upon the Deep (1992) by Vernor Vinge
"The Coming Technological Singularity" (1993) by Vernor Vinge
The Metamorphosis of Prime Intellect (1994) by Roger Williams
"Staring into the Singularity" (1996) by Eliezer Yudkowsky
Creating Friendly AI (2001) by Eliezer Yudkowsky
"Wiki Interview with Eliezer" (2002) by Anand
"Impact of the Singularity" (2002) by Eliezer Yudkowsky
"Levels of Organization in General Intelligence" (2002) by Eliezer Yudkowsky
"Ethical Issues in Advanced Artificial Intelligence" by Nick Bostrom
"Relative Advantages of Computer Programs, Minds-in-General, and the Human Brain" (2003) by Michael Anissimov and Anand
"Can We Avoid a Hard Takeoff?" (2005) by Vernor Vinge
"Radical Discontinuity Does Not Follow from Hard Takeoff" (2007) by Michael Anissimov
"Recursive Self-Improvement" (2008) by Eliezer Yudkowsky
"Artificial Intelligence as a Positive and Negative Factor in Global Risk" (2008) by Eliezer Yudkowsky
"The Hanson-Yudkowsky AI Foom Debate" (2008) on Less Wrong wiki
"Brain Emulation and Hard Takeoff" (2008) by Carl Shulman
"Arms Control and Intelligence Explosions" (2009) by Carl Shulman
"Hard Takeoff" (2009) on Less Wrong wiki
"When Software Goes Mental: Why Artiï¬cial Minds Mean Fast Endogenous Growth" (2009)
"Thinking About Thinkism" (2009) by Michael Anissimov
"Technological Singularity/Superintelligence/Friendly AI Concerns" (2009) by Michael Anissimov
"The Hard Takeoff Hypothesis" (2010), an abstract by Ben Goertzel
Economic Implications of Software Minds by S. Kaas, S. Rayhawk, A. Salamon and P. Salamon
Critiques
"The Age of Virtuous Machines" (2007) by J. Storrs Hall
"Thinkism" by Kevin Kelly (2008)
"The Hanson-Yudkowsky AI Foom Debate" (2008) on Less Wrong wiki
"How far can an AI jump?" by Katja Grace (2009)
"Is The City-ularity Near?" (2010) by Robin Hanson
"SIA says AI is no big threat" (2010) by Katja Grace
I don't mean to say that the critiques aren't important by putting them in a different category, I'm just doing that for easy reference. I'm sure I missed some pages or articles here, so if you have any more, please put them in the comments.