Peter Thiel Agrees that the Future is Not Accelerating
National Review Online (yes... them) recently interviewed Peter Thiel about various topics. He said a lot that I resonate with, such as that college isn't for everyone (dropout right here), but this caught my eye... Peter, like myself, reckons that technological progress hasn't been so impressive in the last 15 years:
SHAFFER: You once said that the tech bubble of the ’90s migrated into an entire financial-services bubble. What does that mean?
THIEL: There’ve been a whole series of these booms or bubbles in the last few decades, and I think it’s a very complicated question why there have been so many and why things have been so far off from equilibrium. There’s something about the U.S. in the last several decades where people had great expectations about the future that didn’t quite come true. Every form of credit involves a claim on the future: I’ll pay you a dollar on Tuesday for a hamburger today; I’ll buy this house, and I’ll pay off the mortgage over 30 years; and so you lend me money based off expectations on the future. A credit crisis happens when the future turns out not to be as good as expected.
The Left-versus-Right debate tends to be that the Left argues that the expectations were off because of ruthless lenders who sold a bill of goods to people and pushed all this debt on people, and that it was basically the problem of the creditors. The Right tends to argue that it was a problem with the borrowers, and people were sort of crazy in borrowing all this money. In the Left narrative, it starts with Reagan in the ’80s, when finance became more important. The Right narrative starts in the ’60s when people became more self-indulgent and began to live beyond their means.
My orthogonal take is that the whole thing happened because there was not enough technological innovation. It was not really the fault of the borrowers or the lenders; the problem was that everybody had tremendous expectations that the country was going to be a much wealthier place in 2010 than it was in 1995, and in fact there’s been a lot less progress. The future is fundamentally about technology in an advanced country — it’s about technological progress. So a credit crisis happens when the technological progress is not as good as people expected. That’s not the standard account of the last decades, but that’s the way I would outline it.
Phil Bowermaster on the Singularity
Over at the Speculist, Phil Bowermaster understands the points I made in "Yes, the Singularity is the biggest threat to humanity", which, by the way, was recently linked by Instapundit, who unfortunately probably doesn't get the point I'm trying to make. Anyway, Phil said:
Greater than human intelligences might wipe us out in pursuit of their own goals as casually as we add chlorine to a swimming pool, and with as little regard as we have for the billions of resulting deaths. Both the Terminator scenario, wherein they hate us and fight a prolonged war with us, and the Matrix scenario, wherein they keep us around essentially as cattle, are a bit too optimistic. It's highly unlikely that they would have any use for us or that we could resist such a force even for a brief period of time -- just as we have no need for the bacteria in the swimming pool and they wouldn't have much of a shot against our chlorine assault.
"How would the superintelligence be able to wipe us out?" you might say. Well, there's biowarfare, mass-producing nuclear missiles and launching them, hijacking existing missiles, neutron bombs, lasers that blind people, lasers that burn people, robotic mosquitos that inject deadly toxins, space-based mirrors that set large areas on fire and evaporate water, poisoning water supplies, busting open water and gas pipes, creating robots that cling to people, record them, and blow up if they try anything, conventional projectiles... You could bathe people in radiation to sterilize them, infect corn fields with ergot, sprinkle salt all over agricultural areas, drop asteroids on cities, and many other approaches that I can't think of because I'm a stupid human. In fact, all of the above is likely nonsense, because it's just my knowledge and intelligence that is generating the strategies. A superintelligent AI would be much, much, much, much, much smarter than me. Even the smartest person you know would be an idiot in comparison to a superintelligence.
One way to kill a lot of humans very quickly might be through cholera. Cholera is extremely deadly and can spread very quickly. If there were a WWIII and it got really intense, countries would start breaking out the cholera and other germs to fight each other. Things would really have to go to hell before that happened, because biological weapons are nominally outlawed in war. However, history shows that everyone breaks the rules when they can get away with it or when they're in deep danger.
Rich people living in the West, especially Americans, have forgotten the ways that people have been killing each other for centuries, because we've had a period of relative stability since WWII. Sometimes Americans appear to think like teenagers, who believe they are apparently immortal. This is a quintessentially ultra-modern and American way of thinking, though most of the West thinks this way. For most of history, people have realized how fragile they were and how aggressively they need to fight to defend themselves from enemies inside and out. With our sophisticated electrical infrastructure (which, by the way, could be eliminated by a few EMP-optimized nuclear weapons detonated in the ionosphere), nearly unlimited food, water, and other conveniences present themselves to us on silver platters. We overestimate the robustness of our civilization because it's worked smoothly so far.
Superintelligences would eventually be able to construct advanced robotics that could move very quickly and cause major problems for us if they wanted to. Robotic systems constructed entirely of fullerenes could be extremely fast and powerful. Conventional bullets and explosives would have great difficulty damaging fullerene-armored units. Buckyballs only melt at roughly 8,500 Kelvin, almost 15,000 degrees Fahrenheit. 15,000 degrees. That's hotter than the surface of the Sun. (Update: Actually, I'm wrong here because the melting point of bulk nanotubes has not been determined and is probably significantly less. 15,000 degrees is roughly the temperature that a single buckyball apparently breaks apart at. However, some structures, such as nanodiamond, would literally be macroscale molecules and might have very high melting points.) Among "small arms", only a shaped charge, which moves at around 10 km/sec, could make a dent in thick fullerene armor. Ideally you'd have a shaped charge made out of a metal with extremely high mass and temperature, like molten uranium. Still, if the robotic system moved fast enough and could simply detect where the charges were, conventional human armies wouldn't be able to do much against it, except for perhaps use nuclear weapons. Weapons like rifles wouldn't work because they simply wouldn't deliver enough energy in a condensed enough space. To have any chance of destroying a unit that moves at several thousands of mph and can dodge missiles, nuclear weapons would likely be required.
When objects move fast enough, they will be invisible to the naked eye. How fast something needs to move to be unnoticeable varies based on its size, but for an object a meter long it's about 1,100 mph, approximately Mach 1. There is no reason why engines could not eventually be developed that propel person-sized objects to those speeds and beyond. In this very exciting post, I list a few possible early-stage products that could be built with molecular nanotechnology that could take advantage of high power densities. Google "molecular nanotechnology power density" for more information on the kind of technology a superintelligence could develop and use to take over the world quite quickly.
A superintelligence, not being stupid, would probably hide itself in a quarantined facility while it developed the technologies it needed to prepare for doing whatever it wants in the outside world. So, we won't know anything about it until it's all ready to go.
Here's the benefits of molecular manufacturing page from CRN. Remember this graph I made? Here it is:
We'll still be stuck in the blue region while superintelligences develop robotics in the orange and red regions and have plenty of ability to run circles around us. There will be man-sized systems that move at several times the speed of sound and consume kilowatts of energy. Precise design can minimize the amount of waste heat produced. The challenge is swimming through all that air without being too noticeable. There will be tank-sized systems with the power consumption of aircraft carriers. All these things are probably possible, no one has built them yet. People like Brian Wang, who writes one of the most popular science/technology blogs on the Internet, take it for granted that these kind of systems will eventually be built. The techno-elite know that these sorts of things are physically possible, it's just a matter of time. Many of them might consider technologies like this centuries away, but for a superintelligence that never sleeps, never gets tired, can copy itself tens of millions of times, and parallelize its experimentation, research, development, and manufacturing, we might be surprised how quickly it could develop new technologies and products.
The default understanding of technology is that the technological capabilities of today will pretty much stick around forever, but we'll have spaceships, smaller computers, and bigger televisions, perhaps with Smell-O-Vision. The future would be nice and simple if that were true, but for better or for worse, there are vast quadrants of potential technological development that 99.9% of the human species has never heard of, and vaster domains that 100% of the human species has never even thought of. Superintelligence will happily and casually exploit those technologies to fulfill its most noble goals, whether those noble goals involve wiping out humanity, or maybe healing all disease, aging, and creating robots to do all the jobs we don't feel like doing. Whatever its goals are, a superintelligence will be most persuasive in arguing for how great and noble they are. You won't be able to win an argument against a superintelligence unless it lets you. It will simply be right and you will be wrong. One could even imagine a superintelligence so persuasive that it convinces mankind to commit suicide by making us feel bad about our own existence. In that case it might need no actual weapons at all.
The above could be wild speculation, but the fact is we don't know. We won't know until we build a superintelligence, talk to it, and see what it can do. This is something new under the Sun, no one has the experience to conclusively say what it will or won't be able to do. Maybe even the greatest superintelligence will be exactly as powerful as your everyday typical human (many people seem to believe this), or, more likely, it will be much more powerful in every way. To confidently say that it will be weak is unwarranted -- we lack the information to state this with any confidence. Let's be scientific and wait for empirical data first. I'm not arguing with extremely high confidence that superintelligence will be very strong, I just have a probability distribution over possible outcomes, and doing an expected value calculation on that distribution leads me to believe that the prudent utilitarian choice is to worry. It's that simple.
Remember, most transhumanists aren't afraid of superintelligence because they actually believe that they and their friends will personally become the first superintelligences. The problem is that everyone thinks this, and they can't all be right. Most likely, none of them are. Even if they were, it would be rude for them to clandestinely "steal the Singularity" and exploit the power of superintelligence for their own benefit -- possibly at the expense of the rest of us. Would-be mavericks should back off and help build a more democratic solution, a solution that ensures that the benefits of superintelligence are equitably distributed among all humans and perhaps (I would argue) to some non-human animals, such as vertebrates.
Coherent Extrapolated Volition (CEV) is one idea that has been floated for a more democratic solution, but it is by no means the final word. We criticize CEV and entertain other ideas all the time. No one said that AI Friendliness would be easy.
Katja Grace Positions
Here:
THINGS PEOPLE SAY
‘Do what your heart tells you’ means ‘stop making up excuses and do what my heart tells you’. ‘Clearly’ means ‘so unclearly I don’t want to explain it’. ‘We’ means different things to those with different political leanings, which helps them disagree.
Aphorisms tend to be cynical because only knowledge you don’t want to believe is short and easily verifiable enough to be an aphorism. People are more inclined to praise long, poorly written writing than short well written ones because it is easier for the former to cheat quality heuristics. Thinking is more fun than reading because it is more like ‘chasing’ than ‘searching‘. It’s interesting that reading isn’t better suited to chasing.
Steven Pinker’s The Stuff of Thought is full of other interesting thoughts about language, a few of which I wrote about. I’m still curious about why we use poorly veiled language so much. Potential veiled meanings are abundant because it is hard to get rid of connotations.
There's a lot to digest.
Long, poorly written writing... Ayn Rand?
Tallinn-Evans Challenge Grant Successful
As many of you probably know, I'm media director for the Singularity Institute, so I like to cross-post important posts from the SIAI blog here. Our challenge grant was a success -- we raised $250,000. I am extremely appreciative to everyone who donated. Without SIAI, humanity would be kind of screwed, because very few others take the challenge of Friendly AI seriously -- at all. The general consensus view on the questions is "Asimov laws, right?" No, not Asimov Laws. Many AI researchers still aren't clear on the fact that Asimov laws were a plot device.
Anyway, here's the announcement:
Thanks to the effort of our donors, the Tallinn-Evans Singularity Challenge has been met! All $125,000 contributed will be matched dollar for dollar by Jaan Tallinn and Edwin Evans, raising a total of $250,000 to fund the Singularity Institute's operations in 2011. On behalf of our staff, volunteers, and entire community, I want to personally thank everyone who donated. Keep watching this blog throughout the year for updates on our activity, and sign up for our mailing list if you haven't yet.
Here's to a better future for the human species.
We are preparing a donor page to provide a place for everyone who donated to share some information about themselves if they wish, including their name, location, and a quote about why they donate to the Singularity Institute. If you would like to be included in our public list, please email me.
Again, thank you. The Singularity Institute depends entirely on contributions from individual donors to exist. Money is indeed the unit of caring, and one of the easiest ways that anyone can contribute directly to the success of the Singularity Institute. Another important way you can help is by plugging us into your networks, so please email us if you want to help.
If you're interested in connecting with other Singularity Institute supporters, we encourage joining our group on Facebook. There are also local Less Wrong meetups in cities like San Francisco, Los Angeles, New York, and London.
Accelerating Future, not Accelerating Futures
There will be only one future, that we're all forced to share, that's the thing. If you do something stupid, someone on the other side of the world can suffer.
There aren't "futures", there is one Future.
The future is not accelerating, by the way.
Why do I still call my blog "accelerating future"? 1) transition costs, 2) I want to bring in people who think the future actually is accelerating and change their minds, of course!
Humanity+ @ Caltech Videos
Watch live video from TechZulu on Justin.tv
These will be further edited and posted on the main website, but some of us want to watch them now.
session 1
http://www.justin.tv/techzulu/b/275297488
http://www.justin.tv/techzulu/b/275297621
session 2
http://www.justin.tv/techzulu/b/275297906
http://www.justin.tv/techzulu/b/275297970
session 3
http://www.justin.tv/techzulu/b/275298583
http://www.justin.tv/techzulu/b/275298693
http://www.justin.tv/techzulu/b/275298796
session 4
http://www.justin.tv/techzulu/b/275298920
http://www.justin.tv/techzulu/b/275299057
http://www.justin.tv/techzulu/b/275299209
Update:
All the edited videos are here, thanks Bryan Bishop.
Yes, The Singularity is the Biggest Threat to Humanity
Some folks, like Aaron Saenz of Singularity Hub, were surprised that the NPR piece framed the Singularity as "the biggest threat to humanity", but that's exactly what the Singularity is. The Singularity is both the greatest threat and greatest opportunity to our civilization, all wrapped into one crucial event. This shouldn't be surprising -- after all, intelligence is the most powerful force in the universe that we know of, obviously the creation of a higher form of intelligence/power would represent a tremendous threat/opportunity to the lesser intelligences that come before it and whose survival depends on the whims of the greater intelligence/power. The same thing happened with humans and the "lesser" hominids that we eliminated on the way to becoming the #1 species on the planet.
Why is the Singularity potentially a threat? Not because robots will "decide humanity is standing in their way", per se, as Aaron writes, but because robots that don't explicitly value humanity as a whole will eventually eliminate us by pursuing instrumental goals not conducive to our survival. No explicit anthropomorphic hatred or distaste towards humanity is necessary. Only self-replicating infrastructure and the smallest bit of negligence.
Why will advanced AGI be so hard to get right? Because what we regard as "common sense" morality, "fairness", and "decency" are all extremely complex and non-intuitive to minds in general, even if they seem completely obvious to us. As Marvin Minsky said, "Easy things are hard." Even something as simple as catching a ball requires a tremendous amount of task-specific computation. If you read the first chapter of How the Mind Works, the bestselling book by Harvard psychologist Stephen Pinker, he harps on this for almost 100 pages.
Basic AI Drives
There are "basic AI drives" we can expect to emerge in sufficiently advanced AIs, almost regardless of their initial programming. Across a wide range of top goals, any AI that uses decision theory will want to 1) self-improve, 2) have an accurate model of the world and consistent preferences (be rational), 3) preserve their utility functions, 4) prevent counterfeit utility, 5) be self-protective, and 6) acquire resources and use them efficiently. Any AI with a sufficiently open-ended utility function (absolutely necessary if you want to avoid having human beings double-check every decision the AI makes) will pursue these "instrumental" goals (instrumental to us, terminal to an AI without motivations strong enough to override them) indefinitely as long as it can eke out a little more utility from doing so. AIs will not have built in satiation points where they say, "I've had enough". We have to program those in, and if there's a potential satiation point we miss, the AI will just keep pursuing "instrumental to us, terminal to it" goals indefinitely. The only way we can keep an AI from continuously expanding like an endless nuclear explosion is to make it to want to be constrained (entirely possible -- AIs would not have anthropomorphic resentment against limitations unless such resentment were helpful to accomplishing its top goals), or design it to replace itself with something else and shut down.
The easiest kind of advanced AGI to build would be a type of idiot savant -- a machine extremely good at performing the tasks we want, and which acts reasonably within the domain for which it was intended, but starts to act in unexpected ways when ported into domains outside those that the programmers anticipated. To quote Omohundro:
Surely no harm could come from building a chess-playing robot, could it? In this paper we argue that such a robot will indeed be dangerous unless it is designed very carefully. Without special precautions, it will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety. These potentially harmful behaviors will occur not because they were programmed in at the start, but because of the intrinsic nature of goal driven systems.
Goal-Driven Systems Care About Their Goals, Not You
Goal-driven systems strive to achieve their goals. "Common sense", "decency", "respect", "the Golden Rule", and other "intuitive" human concepts, which are extremely complicated black boxes, need not enter into the picture. Again, I strongly recommend the first chapter of How the Mind Works to get a better grasp of how the way we think is not "obvious", but highly contingent on our evolutionary history and the particular constraints of our brains. Our worlds are filled with peculiar sensory and cognitive illusions that our attention is rarely drawn to because we all share the same peculiarities. In the same sense, human "common sense" morality is not something we should expect to pop into existence in AGIs unless explicitly programmed in.
Intelligence does not automatically equal "common sense". Intelligence does not automatically equal benevolence. Intelligence does not automatically equal "live and let live". Human moral sentiments are complex functionality crafted to meet particular adaptive criteria. They weren't handed to us by God or Zeus. They are not inscribed into the atoms and fundamental forces of the universe. They are human constructions, produced by evolving in groups for millions of years where people murdered one another if they didn't follow the rules, or simply for one another's mates. Only in very recent history did a mystical narrative emerge that attempts to portray human morality as something cosmically universal and surely intuitive to any theoretical mind, including ogres, fairies, aliens, interdimensional beings, AIs, etc.
It will be easier and cheaper to create AIs with great capabilities but relatively simple goals, because humans will be in denial that AIs will eventually be able to self-improve more effectively than we can improve them ourselves, and potentially acquire great power. Simple goals will be seen as sufficient for narrow tasks, and even somewhat general tasks. Humans are so self-obsessed that we'd probably continue to avoid regarding AIs as autonomous thinkers even if they beat us on every test of intelligence and creativity that we could come up with.
Combine the non-obvious complexity of common sense morality with great power and you have an immense problem. Advanced AIs will be able to copy themselves onto any available computers, stay awake 24/7, improve their own designs, develop automated and parallelized experimental cycles that far exceed the capabilities of human scientists, and develop self-replicating technologies such as artificially photosynthetic flowers, molecular nanotechnology, modular robotics, machines that draw carbon from the air to build carbon robots, and the like. It's hard to imagine what an advanced AGI would think of, because the first really advanced AGI will be superintelligent, and be able to imagine things that we can't. It seems so hard for humans to accept that we may not be the theoretically most intelligent beings in the multiverse, but yes, there's a lot of evidence that we aren't.
Try Merging With Your Toaster
The sci-fi fantasy of "merging with AI" will not work because self-improving AI capable of reaching criticality (intelligence explosion) will probably emerge before there are brain-computer interfaces invasive enough to truly channel a human "will" into an AI. More likely, an AI will rely upon commands, internal code, and cues that it is programmed to notice. The information bandwidth will be limited. If brain-computer interfaces exist that allow us to "merge" with AI and direct its development favorably, great! But why count on it? If we're wrong, we could all perish, or at least fail to communicate our preferences to the AI and get stuck with it forever.
In The Singularity is Near, Ray Kurzweil briefly addresses the Friendly AI problem. He writes:
Eliezer Yudkowsky has extensively analyzed paradigms, architectures, and ethical rules that may help assure that once strong AI has the means of accessing and modifying its own design it remains friendly to biological humanity and supportive of its values. Given that self-improving strong AI cannot be recalled, Yudkowsky points out that we need to "get it right the first time", and that its initial design must have "zero nonrecoverable errors".
Inherently there will be no absolute protection against strong AI. Although the argument is subtle I believe that maintaining an open free-market system for incremental scientific and technological progress, in which each step is subject to market acceptance, will provide the most constructive environment for technology to embody widespread human values.
Kurzweil's proposal for a solution above is insufficient because even if several stages of AGI are gated by market acceptance, there will come a point at which one AGI or group of AGIs exceeds human intelligence and starts to apply its machine intelligence to self-improvement, resulting in a relatively quick scaling up of intelligence from our perspective. The top-level goals of that AGI or group of AGIs will then be of utmost importance to humanity. To quote Nick Bostrom's "Ethical Issues in Advanced Artificial Intelligence":
Both because of its superior planning ability and because of the technologies it could develop, it is plausible to suppose that the first superintelligence would be very powerful. Quite possibly, it would be unrivalled: it would be able to bring about almost any possible outcome and to thwart any attempt to prevent the implementation of its top goal. It could kill off all other agents, persuade them to change their behavior, or block their attempts at interference. Even a “fettered superintelligence†that was running on an isolated computer, able to interact with the rest of the world only via text interface, might be able to break out of its confinement by persuading its handlers to release it. There is even some preliminary experimental evidence that this would be the case.
It seems that the best way to ensure that a superintelligence will have a beneficial impact on the world is to endow it with philanthropic values. Its top goal should be friendliness. How exactly friendliness should be understood and how it should be implemented, and how the amity should be apportioned between different people and nonhuman creatures is a matter that merits further consideration.
Why must we recoil against the notion of a risky superintelligence? Why can't we see the risk, and confront it by trying to craft goal systems that carry common sense human morality over to AGIs? This is a difficult task, but the likely alternative is extinction. Powerful AGIs will have no automatic reason to be friendly to us! They will be much more likely to be friendly if we program them to care about us, and build them from the start with human-friendliness in mind.
Humans overestimate our robustness. Conditions have to be just right for us to keep living. If AGIs decided to remove the atmosphere or otherwise alter it to pursue their goals, we would be toast. If temperatures on the surface changed by more than a few dozen degrees up or down, we would be toast. If natural life had to compete with AI-crafted cybernetic organisms, it could destroy the biosphere on which we depend. There are millions of ways in which powerful AGIs with superior technology could accidentally make our lives miserable, simply by not taking our preferences into account. Our preferences are not a magical mist that can persuade any type of mind to give us basic respect. They are just our preferences, and we happen to be programmed to take each other's preferences deeply into account, in ways we are just beginning to understand. If we assume that AGI will inherently contain all this moral complexity without anyone doing the hard work of programming it in, we will be unpleasantly surprised when these AGIs become more intelligent and powerful than ourselves.
We probably make thousands of species extinct per year through our pursuit of instrumental goals, why is it so hard to imagine that AGI could do the same to us?
Part of the reason is that people have a knee-jerk reaction to any form of negativity. Try going to a cocktail party and bringing up anything in the least negative, and most people will stop talking to you. There is a whole mythos around this, to the effect that anyone that ever mentions anything negative must have a chip on their shoulder or otherwise be a negative person in general. Sometimes there actually is a real risk!
What Would it Cost to Develop a Nanofactory?
Here is Freitas' latest estimate, from his "comprehensive nanorobotic control" article:
What will it cost to develop a nanofactory? Let’s assume research funds are spent in a completely focused manner toward the goal of a primitive diamondoid nanofactory that could assemble rigid diamondoid structures involving carbon, hydrogen, and perhaps a few other elements. In this case, we estimate that an ideal research effort paced to make optimum use of available computational, experimental, and human resources would probably run at a $1–5 M/yr level for the first 5 years of the program, ramp up to $20–50 M/yr for the next 6 years, then finish off at a ~$100 M/yr rate culminating in a simple working desktop nanofactory appliance in year 16 of a ~$900 M effort. Of course the bulk of this work, after the initial 5 year period, would be performed by people, companies, and university groups recruited from outside the Nanofactory Collaboration. The key early milestone is to demonstrate positionally-controlled carbon placement on a diamond surface by the end of the initial 5 year period. We believe that successful completion of this key experimental milestone would make it easier to recruit significant additional financial and human resources to undertake the more costly later phases of the nanofactory development work.
So, about a billion dollars and 16 years. Say they started in 2020, we'd expect a nanofactory around 2036, by this estimate.
Running for Board of Humanity+
I'm running for the Humanity+ board, it will be announced early next week on their website. To vote in the elections, you have to be a registered member of the organization by Monday, which is only $5/month.