John Baez Interviews Eliezer Yudkowsky
From Azimuth, blog of mathematical physicist John Baez (author of the Crackpot Index):
This week I’ll start an interview with Eliezer Yudkowsky, who works at an institute he helped found: the Singularity Institute of Artificial Intelligence.
While many believe that global warming or peak oil are the biggest dangers facing humanity, Yudkowsky is more concerned about risks inherent in the accelerating development of technology. There are different scenarios one can imagine, but a bunch tend to get lumped under the general heading of a technological singularity. Instead of trying to explain this idea in all its variations, let me rapidly sketch its history and point you to some reading material. Then, on with the interview!
The Navy Wants a Swarm of Semi-Autonomous Breeding Robots With Built-In 3-D Printers
Popular Science and Wired reporting. Here is the proposal solicitation.
This is a fun headline, but we're still far from useful functionality in this direction. 3D printers can barely even print circuit boards except for a few exotic prototypes of trivial complexity at hilariously low resolution. More impressive than the progress so far in the DIY community is Xerox's silver printed circuits. Various conductive inks have been developed before and nothing came of them in terms of commercialization. Development by Xerox started in late 2009, it's been over a year now and no news yet.
In terms of strength, the products of 3D printers are weak and can easily be pulled apart with your bare hands. If you want a strong product you still have to go to the machine shop or foundry.
Interesting proposal solicitation, however it is worth remembering that military commanders have been making breathless requests for futuristic technologies since time immemorial. There will be no "semi-autonomous breeding robots with built-in 3D printers" of practical battlefield value until at least 2025. However, this is the sort of thing a superintelligence could build millions of to do its bidding.
Converging Technologies Report Gives 2085 as Median Date for Human-Equivalent AI
From the NSF-backed study Converging Technologies in Society: Managing Nano-Info-Cogno-Bio Innovations (2005), on page 344:
2070
48. Scientists will be able to understand and describe human intentions,
beliefs, desires, feelings and motives in terms of well-defined computational
processes. (5.1)2085
50. The computing power and scientific knowledge will exist to build
machines that are functionally equivalent to the human brain. (5.6)
This is the median estimate from 26 participants in the study, mostly scientists.
Only 74 years away! WWII was 66 years ago, for reference. In the scheme of history, that is nothing.
Of course, the queried sample is non-representative of smart people everywhere.
Artificial Intelligence as a Positive and Negative Factor in Global Risk Now Available in Chinese
Here the chapter in English, here's the Chinese version.
Phil Bowermaster on the Singularity
Over at the Speculist, Phil Bowermaster understands the points I made in "Yes, the Singularity is the biggest threat to humanity", which, by the way, was recently linked by Instapundit, who unfortunately probably doesn't get the point I'm trying to make. Anyway, Phil said:
Greater than human intelligences might wipe us out in pursuit of their own goals as casually as we add chlorine to a swimming pool, and with as little regard as we have for the billions of resulting deaths. Both the Terminator scenario, wherein they hate us and fight a prolonged war with us, and the Matrix scenario, wherein they keep us around essentially as cattle, are a bit too optimistic. It's highly unlikely that they would have any use for us or that we could resist such a force even for a brief period of time -- just as we have no need for the bacteria in the swimming pool and they wouldn't have much of a shot against our chlorine assault.
"How would the superintelligence be able to wipe us out?" you might say. Well, there's biowarfare, mass-producing nuclear missiles and launching them, hijacking existing missiles, neutron bombs, lasers that blind people, lasers that burn people, robotic mosquitos that inject deadly toxins, space-based mirrors that set large areas on fire and evaporate water, poisoning water supplies, busting open water and gas pipes, creating robots that cling to people, record them, and blow up if they try anything, conventional projectiles... You could bathe people in radiation to sterilize them, infect corn fields with ergot, sprinkle salt all over agricultural areas, drop asteroids on cities, and many other approaches that I can't think of because I'm a stupid human. In fact, all of the above is likely nonsense, because it's just my knowledge and intelligence that is generating the strategies. A superintelligent AI would be much, much, much, much, much smarter than me. Even the smartest person you know would be an idiot in comparison to a superintelligence.
One way to kill a lot of humans very quickly might be through cholera. Cholera is extremely deadly and can spread very quickly. If there were a WWIII and it got really intense, countries would start breaking out the cholera and other germs to fight each other. Things would really have to go to hell before that happened, because biological weapons are nominally outlawed in war. However, history shows that everyone breaks the rules when they can get away with it or when they're in deep danger.
Rich people living in the West, especially Americans, have forgotten the ways that people have been killing each other for centuries, because we've had a period of relative stability since WWII. Sometimes Americans appear to think like teenagers, who believe they are apparently immortal. This is a quintessentially ultra-modern and American way of thinking, though most of the West thinks this way. For most of history, people have realized how fragile they were and how aggressively they need to fight to defend themselves from enemies inside and out. With our sophisticated electrical infrastructure (which, by the way, could be eliminated by a few EMP-optimized nuclear weapons detonated in the ionosphere), nearly unlimited food, water, and other conveniences present themselves to us on silver platters. We overestimate the robustness of our civilization because it's worked smoothly so far.
Superintelligences would eventually be able to construct advanced robotics that could move very quickly and cause major problems for us if they wanted to. Robotic systems constructed entirely of fullerenes could be extremely fast and powerful. Conventional bullets and explosives would have great difficulty damaging fullerene-armored units. Buckyballs only melt at roughly 8,500 Kelvin, almost 15,000 degrees Fahrenheit. 15,000 degrees. That's hotter than the surface of the Sun. (Update: Actually, I'm wrong here because the melting point of bulk nanotubes has not been determined and is probably significantly less. 15,000 degrees is roughly the temperature that a single buckyball apparently breaks apart at. However, some structures, such as nanodiamond, would literally be macroscale molecules and might have very high melting points.) Among "small arms", only a shaped charge, which moves at around 10 km/sec, could make a dent in thick fullerene armor. Ideally you'd have a shaped charge made out of a metal with extremely high mass and temperature, like molten uranium. Still, if the robotic system moved fast enough and could simply detect where the charges were, conventional human armies wouldn't be able to do much against it, except for perhaps use nuclear weapons. Weapons like rifles wouldn't work because they simply wouldn't deliver enough energy in a condensed enough space. To have any chance of destroying a unit that moves at several thousands of mph and can dodge missiles, nuclear weapons would likely be required.
When objects move fast enough, they will be invisible to the naked eye. How fast something needs to move to be unnoticeable varies based on its size, but for an object a meter long it's about 1,100 mph, approximately Mach 1. There is no reason why engines could not eventually be developed that propel person-sized objects to those speeds and beyond. In this very exciting post, I list a few possible early-stage products that could be built with molecular nanotechnology that could take advantage of high power densities. Google "molecular nanotechnology power density" for more information on the kind of technology a superintelligence could develop and use to take over the world quite quickly.
A superintelligence, not being stupid, would probably hide itself in a quarantined facility while it developed the technologies it needed to prepare for doing whatever it wants in the outside world. So, we won't know anything about it until it's all ready to go.
Here's the benefits of molecular manufacturing page from CRN. Remember this graph I made? Here it is:
We'll still be stuck in the blue region while superintelligences develop robotics in the orange and red regions and have plenty of ability to run circles around us. There will be man-sized systems that move at several times the speed of sound and consume kilowatts of energy. Precise design can minimize the amount of waste heat produced. The challenge is swimming through all that air without being too noticeable. There will be tank-sized systems with the power consumption of aircraft carriers. All these things are probably possible, no one has built them yet. People like Brian Wang, who writes one of the most popular science/technology blogs on the Internet, take it for granted that these kind of systems will eventually be built. The techno-elite know that these sorts of things are physically possible, it's just a matter of time. Many of them might consider technologies like this centuries away, but for a superintelligence that never sleeps, never gets tired, can copy itself tens of millions of times, and parallelize its experimentation, research, development, and manufacturing, we might be surprised how quickly it could develop new technologies and products.
The default understanding of technology is that the technological capabilities of today will pretty much stick around forever, but we'll have spaceships, smaller computers, and bigger televisions, perhaps with Smell-O-Vision. The future would be nice and simple if that were true, but for better or for worse, there are vast quadrants of potential technological development that 99.9% of the human species has never heard of, and vaster domains that 100% of the human species has never even thought of. Superintelligence will happily and casually exploit those technologies to fulfill its most noble goals, whether those noble goals involve wiping out humanity, or maybe healing all disease, aging, and creating robots to do all the jobs we don't feel like doing. Whatever its goals are, a superintelligence will be most persuasive in arguing for how great and noble they are. You won't be able to win an argument against a superintelligence unless it lets you. It will simply be right and you will be wrong. One could even imagine a superintelligence so persuasive that it convinces mankind to commit suicide by making us feel bad about our own existence. In that case it might need no actual weapons at all.
The above could be wild speculation, but the fact is we don't know. We won't know until we build a superintelligence, talk to it, and see what it can do. This is something new under the Sun, no one has the experience to conclusively say what it will or won't be able to do. Maybe even the greatest superintelligence will be exactly as powerful as your everyday typical human (many people seem to believe this), or, more likely, it will be much more powerful in every way. To confidently say that it will be weak is unwarranted -- we lack the information to state this with any confidence. Let's be scientific and wait for empirical data first. I'm not arguing with extremely high confidence that superintelligence will be very strong, I just have a probability distribution over possible outcomes, and doing an expected value calculation on that distribution leads me to believe that the prudent utilitarian choice is to worry. It's that simple.
Remember, most transhumanists aren't afraid of superintelligence because they actually believe that they and their friends will personally become the first superintelligences. The problem is that everyone thinks this, and they can't all be right. Most likely, none of them are. Even if they were, it would be rude for them to clandestinely "steal the Singularity" and exploit the power of superintelligence for their own benefit -- possibly at the expense of the rest of us. Would-be mavericks should back off and help build a more democratic solution, a solution that ensures that the benefits of superintelligence are equitably distributed among all humans and perhaps (I would argue) to some non-human animals, such as vertebrates.
Coherent Extrapolated Volition (CEV) is one idea that has been floated for a more democratic solution, but it is by no means the final word. We criticize CEV and entertain other ideas all the time. No one said that AI Friendliness would be easy.
Bill Gates Mentions the Risk of Superintelligence in the Wall Street Journal
Bill Gates is smart in a way that other corporate titans of the 90s and 00s just aren't. Smart as in intellectual with a broad range of knowledge and information diet, not "smart" as in wears a trendy turtleneck and has a good design and business sense.
In a recent article in the Wall Street Journal, Gates takes on Matt Ridley's books like The Rational Optimist: How Prosperity Evolves. Gates writes:
Exchange has improved the human condition through the movement not only of goods but also of ideas. Unsurprisingly, given his background in genetics, Mr. Ridley compares this intermingling of ideas with the intermingling of genes in reproduction. In both cases, he sees the process as leading, ultimately, to the selection and development of the best offspring.
The second key idea in the book is, of course, "rational optimism." As Mr. Ridley shows, there have been constant predictions of a bleak future throughout human history, but they haven't come true. Our lives have improved dramatically—in terms of lifespan, nutrition, literacy, wealth and other measures—and he believes that the trend will continue. Too often this overwhelming success has been ignored in favor of dire predictions about threats like overpopulation or cancer, and Mr. Ridley deserves credit for confronting this pessimistic outlook.
Yes, this is common -- who wants to be the doomsayer? It's just not popular. Although dire predictions often fail, terrible things still happen completely unpredicted, like Hurricane Katrina, the global financial disaster, the East Asian Tsunami, and the Holocaust. Pretending that because history has been mostly good, we should take a blanket optimistic outlook is just Whig history nonsense. Whig history is the line we were all fed in school, and its main purpose seems to be to tell us that the status quo is great and there is nothing to worry about.
Gates goes on to talk about how Ridley's two other arguments, that 1) Africa is hurt by foreign aid and will do better without it, and 2) climate change is not as big of a deal as people think, I won't comment on either of these, because most peoples' opinions are based on cultural theology rather than critical thinking. What did get me excited, though, was this part:
There are other potential problems in the future that Mr. Ridley could have addressed but did not. Some would put super-intelligent computers on that list. My own list would include large-scale bioterrorism or a pandemic. (Mr. Ridley briefly dismisses the pandemic threat, citing last year's false alarm over the H1N1 virus.) But bioterrorism and pandemics are the only threats I can foresee that could kill over a billion people. (Natural catastrophes might seem like good candidates for concern, but I've been persuaded by Vaclav Smil, in "Global Catastrophes and Trends," that the odds are very low of a large meteor strike or a massive volcanic eruption at Yellowstone.)
Ridley shouldn't dismiss the pandemic threat, obviously. You'd think that a deadly natural plague that killed 3% of the world population and infected 27% a century ago would be enough to take it seriously for centuries to come, simply based on Bayesian likelihood estimations, but I guess not. I wonder if the widespread availability of genetic engineering tools for creating new microbes causes Ridley to update his estimation of disaster from the likelihood estimation simply based on history upwards more than a couple percent.
The quoted paragraph is also interesting because it's the first time I'm aware of that Gates has come out this strongly about the machine threat, and even uses the term "super-intelligent". I wouldn't be shocked if Gates has read all of Nick Bostrom's papers on the superintelligence threat or perhaps has even visited this blog. Who knows? A little unwarranted optimism is cute and harmless when it comes to celebrities visiting one's blog, but it becomes dangerous and destructive when applied to the course of civilization as a whole.
No optimism. No pessimism. Realism. Optimism and pessimism are inherently irrational because they imply a bias across all possible hypotheses, and the emphasis is on the affect (feeling), not the descriptive content of the hypothesis itself. If anything, pessimism is more rational. See the planning fallacy and rational pessimism. One study on the planning fallacy found that people who were depressed tended to be the most accurate when estimating the completion time of projects.
I find it funny how many people in the transhumanist community, miffed at the attention the Singularity has been getting, seem to wish that transhumanists would just ignore the risk of superintelligent machines, while people like Bill Gates are just starting to write about it in public. This is the time to step forward, not back. The finance giants of Wall Street should know that they can have a personal impact on the risk of superintelligence by donating to non-profits like the Singularity Institute and the Future of Humanity Institute. Peter Thiel certainly realizes this, but most moguls don't. The people and infrastructure exist to make use of much larger funding levels, and it's incumbent on philanthropists to step forward.
Mitchell Porter on “Emotion” in AIs, How This Relates to Developing Friendly AI
Thank you to all the bright people who participate with great enthusiasm in the comments on Accelerating Future. The quality of discussions here is often very high -- thank you!
In the "future superintelligences indistinguishable from today's financial markets?" thread, DMan and Mitchell Porter engage in a discussion about emotions and their relevance to AI.
Correction: Mitchell Porter not Howe. Sorry to Mr. Porter, both him and Mr. Howe have insightful comments but I usually find myself reposting the latter.
DMan says:
If I understand Rinesi correctly, he is equating distributed networks of information processing with AI. This model has been used by some neuroscientists as a model for consciousness, without much success in my opinion.
The problem, it seems to me, is that such a system would always be too diffuse to ever have an internal model of self that’s required for a useful form of consciousness. As far as I can see, there has to be a central point to which information flows and is interpreted by the overseer of the ‘self’.
It’s really like saying that a flock of birds flying in formation, schools of fish, or the coordinated activities of a termite nest are ‘intelligences’ in of themselves. In many ways the cooperative activities of social animals and hive insects resemble the financial market metaphor – would you call them an ‘intelligence’?
It’s kind of an ‘emergent hive mind’ theory with more in common with magical thinking than science.
As to AI being no existential threat:
I would (again) refer you to the excellent book by science writer Rita Carter, ‘Consciousness’ that collates evidence from neuroscience, cognitive studies and psychiatry in an attempt to understand the only working model of consciousness we have – us.
In it, research appears to indicate that:
a) There can be no ‘intelligence’ without ‘consciousness’
b) Consciousness – no matter human or otherwise – requires some form of embodiment to function (so, again, Rinesi’s hive mind is not embodiment).
c) Consciousness requires an emotional component in order to function. The very act of making even an abstract judgement requires a ‘feel’ for what is ‘right’ or ‘wrong’ – that’s emotion.
My point in mentioning this is that an AI (perhaps better name is AC – Artificial Consciousness?) will in all likelihood have to be able to experience emotion to be useful to us. Without emotion there is only a pallid facsimile of creativity – and without creativity what could it invent?
So if it can – or must – have emotion, it most certainly can and will be dangerous. It could feel resentment, jealously, ambition and hatred.
This is why,I suspect, many so-called AI ‘experts’ (experts on something that is currently not a reality doesn’t really warrant the title does it?) are uncomfortable with the idea of emotion being necessary, and go out of their way to dismiss it. Because it implies a potential existential threat to us.
But if they’re wrong and emotion is indeed needed for AI, then Rinesi’s telling us not to worry is – at best – irresponsible.
Mitchell replied:
“Without emotion there is only a pallid facsimile of creativity – and without creativity what could it invent?â€
Computers can invent in the way that evolution does – through trial and error. They can also employ smarter search algorithms than that, e.g. to explore a space of possible designs.
The whole significance of computing is that it provides a mathematical theory and a material technology capable of duplicating every mental function – goal-directed behavior, pattern recognition, communication – in a way which makes no reference to mind, thought, meaning, consciousness, emotion, etc. It’s all just physical “state machines†interacting with each other.
And since neuroscience is analyzing the human brain and human behavior in the same way, this raises serious questions about the relationship between subjectivity and the world of mindless physical cause and effect. That debate has been around for centuries now, but the theory of computation gives it a new twist. Lots of people now like to think that e.g. emotion or consciousness occurs wherever a certain type of computation occurs. Whether you believe that or not, the significance for AI is that emotion and consciousness are not a necessary part of AI theory or practice. You don’t need to think of a thermostat’s operation in terms of liking and disliking in order to design and make it, and the same goes for the far more intricate feedbacks and calculations which would make up an artificial “intelligence†capable of rivaling the human mind.
There's a bit more back-and-forth, then Mitchell posts the following, which I think is especially on-target:
The creative limits of chess computers and story grammars don’t tell us about the limits of unconscious computation in general. Consider any cognitive process which involves emotion, creativity, consciousness, compassion, etc. It’s going to have a cause-and-effect description, in which there are transitions between various psychological states. To produce the same outputs, all that’s required is a process with the same cause-and-effect structure. The “states†involved don’t have to be psychological or conscious – unless functionalist philosophy of mind is right, and all those psychological properties really are present whenever you have the right sort of causal structure.
Even if I take that view – suppose I want to make a creative AI, and I decide to follow your advice and make it “emotionalâ€. How do I even do that? If I adopt a particular software design, how do I know whether or not it corresponds to the existence of emotion in the AI?
The ultimate reason that this doesn’t seem like very useful advice (for someone who wants to make an intellectually powerful AI – we’ll get to the ethical issue in a moment) is that emotion itself doesn’t solve problems, even in humans. If the problems themselves involve emotions, then an emotion can *be* the answer – happiness might be the answer to unhappiness, just as a glass of water can be the answer to thirst. But if you’re a monkey in a room trying to get at a banana on the ceiling, emotion itself does not tell you that the answer consists of stacking boxes and climbing on top of them. Or rather, emotion is not the process which will materialize that possibility in your mind. Emotion may motivate you to devote cognitive resources to the problem, and your mind may be wired to produce an emotion (excitement) when an imagined solution looks like it will work. But the consideration of possible actions – visualization, combinatorial explorations – all that is more “computational†than “emotionalâ€, and that’s the process which generates possible solutions. (Embodiment also plays a role here, because it permits, not just formal trial and error, but also a more formless experimentation which will suggest possibilities and components of possibilities.)
If a process is to produce the solution to a problem, it has to generate possible solutions and then evaluate whether they are useful. It’s a psychological fact about human beings that emotion and consciousness play a role in this evaluation of possibilities, and they even play a role in determining what we will think of as a problem. But from a computational perspective, it doesn’t have to be emotion or consciousness which performs the evaluative function. There just needs to be a sub-process which discriminates or guides appropriately, and that can be yet more unconscious computation. If you look at problem-solving algorithms searching a space of possibilities for solutions, they typically alternate between the formal generation of new possibilities, and the formal evaluation of the newly generated possibilities – do they offer progress towards a complete solution.
Summing up this stage of my argument: The intellectual power of an AI would not reside in the existence of emotions or the existence of an emotionlike structure of cognitive control and guidance. It would depend on the quality and power of its basic problem-solving algorithms. I am trying to finesse the hard problems associated with consciousness and subjectivity by being agnostic, in this discussion, about their relationship to material and computational reality. Further down the page, David Pearce and “Continuously Computed†have given us statements of the two main approaches, namely, choice of material substrate matters for the existence of subjectivity, and, only causal structure matters; consciousness reduces to substance, or consciousness reduces to function. Of course it’s a complex and very important issue, but I do want to emphasize just how far we can expect to go in the creation of general-purpose AI, employing only computational concepts.
The other topic you bring up is whether the creation of emotional AI is a way to achieve what our blog-host would call “Friendlinessâ€: rather than trying to engineer the functional equivalent of friendliness in an emotionless AI, you make an emotional AI and start it off compassionate. But that strategy requires that you begin to solve the hard problem of consciousness, as it pertains to emotion and compassion; you would need to say *how* to make an AI emotional or compassionate. And you would need to understand something of the developmental dynamics in an artificial emotional system. I assume you don’t want it *going mad* out of extreme sensitivity.
Even people who want to make an emotionless but Friendly AI have to find solutions to those problems anyway, because, even if emotions are not part of the AI’s mechanism, they have to be part of its supposed domain of competence. A general-purpose AI could not know how to treat human beings ethically or even safely, without having a highly refined understanding of emotion and all these other aspects of conscious subjective experience. Part of SIAI’s current thinking about the achievement of Friendliness seems to involve outsourcing some of these problems to the proto-friendly AI, which will engage in neuroscientific studies aimed at identifying what real-world material phenomenon or attribute is intended by all this vague human talk about emotions and consciousness and so on. It’s an interesting idea but it still requires as a starting point some minimal idea, on the part of the programmers or design theorists, about how to tell the AI what to investigate and how to value it, e.g. something like “We want the world to be optimized according to the criteria that are used by the part of the brain responsible for the judgements which ultimately produce confident assertions of happiness with the overall situation.â€
Does anyone else believe that phenomenological consciousness is necessary for general problem-solving? I'm sort of confused why anyone would think that. This line in particular is confusing to me:
And how do they judge, from ‘trial and error’ which is the best solution? Other than something blowing up of course? Without the ability to judge – which according to cognitive studies, requires emotion?
This statement sort of implies that "judge" is a natural category distinct from trial-and-error, and that the two are naturally separated and distinct clusters in algorithm-space, rather than two points on a continuum. It doesn't make sense because there's obviously a million shades of competence between the most simplistic trial-and-error algorithms and human judgement, to refer to them as two natural categories is most confusing. Surely animals have solved numerous "judgement" problems without human-level "emotions".
The thinking seems to be that the universe can grant a special gift, "consciousness", on certain hallowed beings, then they get magical judgement powers. But phenomenological consciousness doesn't seem particularly related to judgement capabilities, and thinkers like Chalmers never suggest that it does in their papers on consciousness.
Checking out the reviews for Consciousness on Amazon, I see that Carter is a science writer rather than a scientist, and makes basic errors like thinking that an atom becomes positively charged when it gains an electron. Still, I don't think that being a science writer rather than a scientist should ruin someone's scientific credibility. Some science writers have a much more thorough interdisciplinary knowledge of science than many scientists.
Future Superintelligences Indistinguishable from Today’s Financial Markets?
It seems obvious that Singularity Institute-supporting transhumanists and other groups of transhumanists speak completely different languages when it comes to AI. Supporters of SIAI actually fear what AI can do, and other transhumanists apparently don't. It's as if SL3 transhumanists view smarter-than-human AI with advanced manufacturing as some kind of toy, whereas we actually take it seriously. I thought a recent post by Marcelo Rinesi at the IEET website, "The Care and Feeding of Your AI Overlord", would provide a good illustration of the difference:
It’s 2010 — our 2010 — and an artificial intelligence is one of the most powerful entities on Earth. It manages trillions of dollars in resources, governments shape their policies according to its reactions, and, while some people revere it as literally incapable of error and others despise it as a cathastrophic tyrant, everybody is keenly aware of its existence and power.
I’m talking, of course, of the financial markets.
The opening paragraph was not metaphorical. Financial markets might not match pop culture expectations of what an AI should look like — there are no red unblinking eyes, nor mechanically enunciated discourses about the obsolesence of organic life — and they might not be self-aware (although that would make an interesting premise for a SF story), but they are the largest, most complex, and most powerful (in both the computer science and political senses of the word) resource allocation system known to history, and inarguably a first-order actor in contemporary civilization.
If you are worried about the impact of future vast and powerful non-human intelligences, this might give you some ease: we are still here. Societies connected in useful ways to “The Market†(an imprecise and excessively anthropomorphic construct) or subsections thereof are generally wealthier and happier than those than aren’t. Adam Smith’s model of massively distributed economic calculations based on individual self-interest has more often than not surpassed in effectivity competing models of centralized resource allocation.
This post is mind-blowing to me because I consider it fundamentally un-transhumanist. It essentially says, "don't worry about future non-human intelligences, because they won't be so powerful that they aren't indistinguishable from the present day aggregations of humans".
Isn't the fundamental idea of transhumanism that augmented intelligences and beings can be qualitatively different and more powerful than humans and human aggregations? If not, what's the point?
If a so-called transhumanist thinks that all future non-human intelligences will basically be the same as what we're seen so far, then why do they even bother to call themselves "transhumanists"? I don't understand.
Recursively self-improving artificial intelligence with human-surpassing intelligence seems likely to lead to an intelligence explosion, not more of the same. An intelligence explosion would be an event unlike anything that has ever happened before on Earth -- intelligence building more intelligence. Intelligence in some form has existed for at least 550 million years, but it has never been able to directly enhance itself or construct copies rapidly from raw materials. Artificial Intelligence will. Therefore, we ought to ensure that AI has humans in mind, or we will be exterminated when its power inevitably surges.
If there are any other transhumanists who agree that future superintelligences will be directly comparable to present-day financial markets, please step forward. I'd love to see a plausible argument for that one.
Another Nick Bostrom Quote
"One consideration that should be taken into account when deciding whether to promote the development of superintelligence is that if superintelligence is feasible, it will likely be developed sooner or later. Therefore, we will probably one day have to take the gamble of superintelligence no matter what. But once in existence, a superintelligence could help us reduce or eliminate other existential risks, such as the risk that advanced nanotechnology will be used by humans in warfare or terrorism, a serious threat to the long-term survival of intelligent life on earth. If we get to superintelligence first, we may avoid this risk from nanotechnology and many others. If, on the other hand, we get nanotechnology first, we will have to face both the risks from nanotechnology and, if these risks are survived, also the risks from superintelligence. The overall risk seems to be minimized by implementing superintelligence, with great care, as soon as possible."
-- Nick Bostrom, "Ethical Issues in Advanced Artificial Intelligence"
Randal Koene and Suzanne Gildert: “Remarkable as Our Brains Are, It Is Clear That There are Many Limitations.”
Randal Koene and Suzanne Gildert have a post up at KurzweilAI.net on their new networking/outreach community, carboncopies.
Advancing Substrate-Independent Minds 2010 (ASIM-2010), the organization's launching conference, will be held on Monday and Tuesday evening after Singularity Summit 2010.
Finally, an organization focused on this topic. A substrate-independent mind would be a beautiful thing, as long as its values are aligned with ours.
From their FAQ:
Remarkable as our brains are, it is clear that there are many limitations. We experience limited working memory, finite and unreliable long-term memory, the inability to multi-task effectively, and our sensory experience and comprehension are a small subset of what may be possible. Substrate-independence can enable us to exceed those limitations and to suitably adapt to novel environments, to explore other ways of thinking, and to experience virtual environments from a truly first-person perspective.
Furthermore, human embodiment, while in all likelihood responsible for much of the astonishing capability of our minds within that embodiment, is also increasingly a constraint on the possible capabilities of both mind and body. Our brains, evolved to suit the local environment and embodiment are not easily redirected at other possible optimizations. Consider for example, that we do not think fully even in three spatial dimensions when we build our cities. More complex geometric problems entirely elude our natural capabilities.
Similarly, our potential embodiments and the corresponding experiences are limited by the need to sustain a local environment suitable for the biological substrate of brain and body. We could fly – but we must do so within an unnecessarily complicated aircraft that is designed to incorporate and sustain human passengers. We could simultaneously interact with the marine environment of the Great Barrier Reef while also experiencing the freedom of near vacuum on the far side of the moon. But, we must always temper such pursuits for the needs of human physical embodiment.
In addition to these limitations in terms of capability or environment, we have the limitations of speed and time. Our ability to experience and act is constrained by the rate at which we can process thoughts. Time is limited by the mortality of our biology. For those who wish to their capabilities in these dimensions, substrate-independence offers unique potential safeguards against these limitations, while at the same time removing concerns about human over-population within the bounds of our current biosphere.
See also my post on "substrate-independent minds".
Nick Bostrom Superintelligence Quote
Here's the new quote I'm using on my Facebook:
"Disease, poverty, environmental destruction, unnecessary suffering of all kinds: these are things that a superintelligence equipped with advanced nanotechnology would be capable of eliminating."
Nick Bostrom, "Ethical Issues in Advanced Artificial Intelligence"