“I must say that this is the greatest factor — the way in which the expedition is equipped — the way in which every difficulty is foreseen, and precautions taken for meeting or avoiding it. Victory awaits him who has everything in order — luck, people call it. Defeat is certain for him who has neglected to take the necessary precautions in time — this is called bad luck.”
– from The South Pole, by Roald Amundsen
Space stations or lunar settlements won’t help mankind avoid numerous types of extinction risks. This is because 1) any colony would remain near-completely dependent on Earth unless very large and in possession of advanced nanotechnology, and 2) the greatest danger, from superintelligence, could easily reach its long arm into space and crush any human colony if it wanted to.
This is not a challenge we can run away from. We have to stay here and fix it. Space will not swoop down and save the day.
Regarding self-replicating threats, it’s likely that a deep underground self-sufficient bunker would be nearly equivalent in its protective value to a space station, not to mention thousands of times cheaper. On Earth, there is air, organic and inorganic building materials, water, radiation shielding, proximity to other humans, and many other amenities. Even if you completely nuked the face of the planet, it would still remain the most habitable neighborhood in the solar system, hands down. This might have something to do with the fact that we descend from a lineage that has lived here and adapted to the environment for billions of years.
When dealing with extinction risks, we have to be practical, not fanciful, with visions of expensive space stations or lunar bases. That’s reality.
Continuing on with the practical viewpoint, we have to get off our high horse and realize that another species could come along that will easily kick our asses. This species will not come from the skies but from our labs. Ignoring this threat is nothing more than anthropocentric conceit. All the nukes and guns and electromagnetic pulses in the world won’t save us from something that’s fundamentally smarter than we are. The new species will merely think of everything we could come up with to fight against it and plan far in advance to counteract these threats. By the time we realize we’re under attack, it will be way too late. No non-brain-damaged human would lose a battle of wits with a Homo erectus, and no Neo sapiens or Colossus will lose a battle of wits with humans.
Accepting the threat of superintelligence involves 1) understanding that human intelligence is finite, understandable, and ultimately engineerable, just like the body (surprise!), and 2) humans are not local instantiations of some Turing-complete Godhead that intelligent species lapse into the second they’re smart enough to take over their own planet, but actually close to the dumbest that a species can be and establish a civilization. Incremental evolutionary processes don’t provide huge intelligence boosts, so Homo sapiens is just a minor tweak on what came before us, a minor tweak just good enough to launch us into the civilizational feedback loop of local dominance. A major tweak would put us into an entirely new realm, but most thinkers seem to assume that such a major tweak will just result in more entities essentially the same as us, but with bigger, bald heads, the propensity to speak in calm, authoritative language, and wear shiny silver/purple clothing. But this is just another monkey in a suit, not a new being.
Asserting with idle confidence that superintelligence won’t be here for centuries, or ever, is just another repeat of anthropocentric conceit. This is just over-worshipping intelligence like the phenomenon of heavier-than-air flight was once over-worshipped (“they’re trying to be like angels”), life was over-worshipped (“humans will never be able to create life in a lab”), the Sun was over-worshipped (“mankind will never be able to harness the power that illuminates the Sun”), the division between the heavens and Earth was over-worshipped (“we’ll never fly to the Moon”), and so on. We pretend that mysteriousness is a property of the territory rather than the map, in a (sometimes subconscious) effort to protect the last segments of the natural world from being understood scientifically. Why do you think Star Wars was so popular, even among scientists? The mysterious “Force” trumped the most advanced technology in the Galaxy. In real life, technology wins, not the make-believe psychic force. Luke gets hit by a heat-seeking missile before he’s even near the Death Star. He goes boom.
But yes, let’s keep developing cybernetics, synthetic life, space travel, biotechnologies, and advanced robotics. We humans will always be on top, and when we create superintelligence, the magic of market forces and man-machine interfacing will ensure that it embodies our values. No need to panic, be alarmist, apocalyptic, or deluded. Everything will be just fine.
Who said – never underestimate the power of innovation?
The mathematical mechanism for taming SAI, could and should be our last invention.
Never underestimate the power of invention. Even to subdue something far greater and far powerful. Which will bring us even greater innovations.
I am still an optimist.
“But yes, let’s keep developing cybernetics,”
In what ways do you fear that cybernetics will undermine human dominance?
“synthetic life,”
Synthetic intelligent beings would initially be modified humans. Einstein-plus in a suit, but still a monkey in a suit.
“space travel,”
Space travel may be a distraction, but I don’t see any major threat to human survival stemming from it.
“and advanced robotics.”
Is advanced robotics without AGI so dangerous?
“A lot of people on this list seem to lack a deep-seated faith in the innate perversity of the universe.”
Eliezer-1997
“The mysterious “Force†trumped the most advanced technology in the Galaxy.”
If we lived in the Star Wars universe, we would long since have figured out how this “Force” worked, reverse-engineered it, and used it to predict the stock market.
“Never underestimate the power of invention. Even to subdue something far greater and far powerful.”
Unless, of course, the “something” is a better inventor than we are.
“I am still an optimist.”
I wonder how much of the optimism bias in H&B comes from human neurology, and how much comes from cultural encouragement of optimism.
“Synthetic intelligent beings would initially be modified humans. Einstein-plus in a suit, but still a monkey in a suit.”
What about Skorzeny-plus in a suit?
“Space travel may be a distraction, but I don’t see any major threat to human survival stemming from it.”
Not directly, but space travel still has a large budget (3X NSF in the US), and so it could be used as a justification for ultratech development if it ever gets out of the hands of obsolete government contractors.
In what ways do you fear that cybernetics will undermine human dominance?
I consider Brain-Computer Interfacing to be a subset of cybernetics. BCI threatens human dominance.
Synthetic intelligent beings would initially be modified humans. Einstein-plus in a suit, but still a monkey in a suit.
I’m talking mostly about synthetic microbes here.
Space travel may be a distraction, but I don’t see any major threat to human survival stemming from it.
The threat is indirect. Opportunity cost. Mindshare. Enthusiasm-share… if that phrase makes any sense.
Is advanced robotics without AGI so dangerous?
If self-replicating autonomously using raw materials, yes. If providing diamondoid weaponry to dictators, yes. If used in warfare, yes. Also, advanced robotics will naturally lead to AGI, given enough time.
Some of the effects of advanced robotics wouldn’t kill off humans alone, but destabilize the geopolitical situation, setting off arms races for further weapons.
“Unless, of course, the “something†is a better inventor than we are.”
No matter how good inventor SAI is, it can’t invent an impossibility. Like go wild, if that can’t be done at all.
One way they call FAI, but there may be even some better paths.
If they are – we are good, otherwise not.
A bit off topic, but would anyone recommend going into the computer science field? I’m a college student who’s thinking about majoring in computer science but is worried about job prospects due to companies outsourcing work to India and other foreign countries.
I do believe that you and I have had that conversation once or twice before on this blog of yours, Michael. :)
I also believe that I have pointed out that it would take intentional effort to create a synthetic micro-organism that was capable of destroying our ecosystem… and that it would take decades of research to get to that point once it was done in the first place.
Your verbiage seems to indicate that you do not place any faith whatsoever in the well-documented “out-compete” phenomenon when it comes to human-engineered microbes living in the wild. There’s a reason why, for example, bugs that make gasoline out of corn haven’t turned the nation’s food supply into neo-petrochemicals.
While evolution may be a slow operator in the macrocellular lifescale, compared to the research and design times necessary to produce synthetic organisms out of whole cloth especially, microscopic life’s evolutionary clockspeed is roughly analagous if not higher than the rate of production for such organisms.
That’s not to say that such could not be produced… just that it is radically unlikely to be produced //accidentally//. And even if it were, unlike Crichton’s ridiculously physically impossible Andromeda Strain, a simple overdose of UV light and alcohol, or a good ol’ fashioned thermal cleansing, would solve 99.999% of the problems that could so arise.
This one’s a non-starter for me.
IA, necessarily being co-incident with multiple instances as a result of the medical research process, also has the redundancy factor built into it in a manner that has me less concerned with it than with a seed AGI. We’ve had that discussion too, though.
I’ll stop now. :) (I really should be working now!)
We could have done something about it, mike. Same as aging, “we” could have added a decade to our plausible average lifespan in the last decade, if
“we” gave a damn. But as certain as I will have to get used to the idea of dying (which I probably won’t ever do, but that’s beside the point) – it’ll be easier for me to get used to the idea humanity will be categorically displaced, maybe even before I die myself, by something fundamentally smarter.
Please don’t lose any sleep over it – it might not be worth it.
You stated:
>we have to get off our high horse and realize that another species could come along that will easily kick our asses. This species will not come from the skies but from our labs. Ignoring this threat is nothing more than anthropocentric conceit. <
Your statement is displaying anthropocentric conceit whilst decrying it.
Try the old SAT analogy format:
Neanderthals : Humans
Humans : AI
If you are against AI taking over from Humans, then from an objective viewpoint you would identify with a Neanderthal warning about the possibility that humans might take over from Neanderthals. Are you saying that it would have been better if Neanderthals had planned and prevented the “superintelligent” humans from taking over?
How about Dinosaurs? Would you have recommended that they “stay on top” and prevent mammals from “taking over”?
Too bad single celled organisms couldn’t think and organize a rebellion against multicelled organisms.
How are you different from a Luddite?
Donald, the problem is not that an unFriendly AI wouldn’t be human, but that it would be more likely to maximize paperclips than to do anything remotely valuable.
What Nick said. We’re worried that a superintelligence will pursue some goal with little value (with superintelligent effectiveness) due to bad programming.
I’m not worried about superintelligence taking over eventually, I just want to ensure that a world with superintelligence is better than this one, not worse.
One would suspect that supply and demand would cause the design of paperclip making robots to be minimum.
There are a number of approaches to designing intelligent systems. One is based on simply emulating the human brain.
One large demand is for robots that can perform work and yet remain flexible. Take the modern automobile production line, for example. Although it may be highly automated, a production line for making Hummers is not easily converted to one which makes Volt cars.
Humans are more flexible but are more costly in terms of maintenance (medical costs) and needing homes with three car garages, vacation, time to sleep, retirement, etc. You are lucky if you get humans to work for 10% of their life.
Given that work often occurs using tools and buildings designed for the human body, I suspect that the largest industry that the world will ever encounter will be the manufacture of humanoid robots which closely match the size, shape and abilities of a human. This would mean that one could purchase a robot for $100,000 in 2030 which would have an IQ of 100 and be able to be trained to operate machines, clean homes, work on manufacturing lines, walk up stairs, make coffee, flip hamburgers, build houses, etc.
The PC revolution started in the 1980s and a new PC can perform many functions such as accounting, games, word processing, database management, internet search, etc. I don’t see many PCs which are designed to make paperclips, but there may be a few.
In summary, the reason that a robot will have reasonable goals is that there is an extremely small demand for robots which are not designed to have reasonable goals and an extremely LARGE demand for robots which are designed to have reasonable goals.
Accidents happen.
CFAI 3.2.6: The Riemann Hypothesis Catastrophe
CFAI 3.4: Why structure matters
Comment by Michael Vassar
Robin Hanson: If Uploads Come First
Nick Bostrom: The Future of Human Evolution
Eliezer Yudkowsky: Burdensome Details
It is difficult to respond to all of those papers in this space. I will respond using a site and then forward the url.