Response to “What is friendly?”
Over at the Streeb-Greebling diaries, Bob Mottram watched the Google video on the Risks of AGI panel and writes,
In this video a panel of luminaries discuss the future risks which advanced forms of AI might pose. Much hinges upon the idea of "friendliness", and trying to ensure that decisions made by powerful intelligences will always be somewhat in tune with human desires. The elephant in the room here though is that there really is no good definition for what qualifies as "friendly". What's a good decision for me, might not be a good decision for someone else. When humans make decisions they're almost never following Asimov's zeroth law.
Asimov's zeroth law is "A robot may not injure humanity, or, through inaction, allow humanity to come to harm."
It's not really "an elephant in the room". There is a common definition for "friendly", and it is accepted by many in the field:
"A "Friendly AI" is an AI that takes actions that are, on the whole, beneficial to humans and humanity; benevolent rather than malevolent; nice rather than hostile."
Not too difficult. Then comes the objection, "there can be no such thing". Well, then you'd want to build an AI that is as close to that as possible.
So should future AIs always be engineered to follow the zeroth law?
Yes... not really as a "law", but as an innate part of its motivations.
If an AI could override all human political decision making and impose an equitable world food distribution network I think that would be a very positive development. But would national leaders be willing to have their own self-interested agendas overridden by automation? I suspect they would be unhappy about that, owing to the inherently tribal nature of human psychology.
If you can't please everyone all the time, then try to please as many people as possible most of the time. Again, there's no dilemma here. I do suspect that a superintelligence with advanced nanotechnology would be able to go a long way towards appealing the national leaders even if their people are fed.
One fallacy in my opinion is that it will be possible to control and predict the decision making quality of very complex AGIs. It's already hard for us to predict how existing, relatively simple, computer programs will operate under all possible conditions.
Well, a complex AGI will be built by a simpler AGI that human programmers write. Of course we cannot predict anything with 100% accuracy, but I do think that we can build an AGI such that we can place more confidence in it crossing the line to the superintelligent regime than we would in any particular human or combination of humans.
An AGI with a supergoal of maximizing the number of black objects in the universe will not be convinced to change its goal system through any learning experience, however anti-black it may be.
A problem in the conceptualization of Friendly AI is that some people think that we are aiming for perfection. Not so - we're just aiming for the best we can do, and something better than the alternatives. We can't ask for anything more.
Once you introduce general learning capabilities into the equation it soon becomes impossible to say what the system will do in the long run.
Not necessarily. A static human cognitive architecture will always do the same things in the long run - humanlike things. A humanlike brain has humanlike goal attractors, which are preserved in the abstract regardless of any amount of learning. In the space of all possible goal attractors, the human mind stays within a very constrained area.
It will be possible to write utility functions that remain invariant regardless of new knowledge that is acquired, or remain invariant within certain constraints. New information that changes the particulars of subgoals, but does nothing to change the supergoal. "Learning" implies acquiring knowledge, but does not necessarily imply changing goals.
Another assumption made by Goertzel and others is that there will only be a few powerful AGIs in the world.
The assumption is that there will be a hard takeoff whereby the first AGI to engage in recursive self-improvement is basically leagues ahead of any other AGI. Given that silicon transistors have switching speeds millions of times greater than biological neurons, this is indeed plausible.
Goertzel and others are not saying that there will only be a few powerful AGIs. Just that there will be a first mover, and that the existence of all future AGIs will be contingent upon the first AGI accepting their existence. The future space of all created AGIs will be limited by lines drawn by the first AGI, or human wishes channeled through that AGI.
Then there's also the Gates scenario, where there will be a super-powerful AGI on every desktop and in every home. In this situation anybody will be able to have AGIs do whatever they wish, with no guarantees on friendliness.
The first AGI that reaches superintelligence is likely to become so powerful as to qualify for near-omnipotence. It would be trivial to prevent the creation of AGIs antithetical to its goals.
Further material on Friendly AI:
What is Friendly AI?
Creating Friendly AI
Anyone interested in the field of Friendly AI should read that last one from start to finish. Also: note that 'friendly', the English word, is not the same thing as "Friendly", which is extremely complex and subtle.
Fantastic New Paper by Jason Matheny on Extinction Risk
An area of study more important than any other is that of extinction risks. An average-intelligence person devoting their life to the study and mitigation of existential risks can accomplish far more ethical good than lifetimes of work by thousands of the best and brightest politicians, scientists, writers, and programmers. Morality-wise, it's a pursuit that blows all others out of the water. Why? Because the negative value represented by the possibility of existential disaster is much greater in magnitude than all the other evils in the world, including poverty, torture, disease, and tyranny. We can't make a better world if we're dead.
If our species survives this century and goes on to colonize the stars, the people who were instrumental in minimizing the probability of risk during this century will deserve a lot of the credit. If you choose to devote your life to mitigating existential risk and actually end up having a significant impact, you could actually be famous for the rest of eternity. Think about that!
This is why it's of such massive importance whenever a new paper comes out about the subject. This area of study is neglected. Only in the past five years has it become an area of significant focus. Today, big names like Stephen Hawking and Martin Rees are on our side. However, there is an explicit lack of publications in the area.
It's my pleasure to upload a paper by Jason Matheny of the University of Maryland, entitled "Reducing the risk of human extinction". Matheny is known publicly for his involvement with New Harvest, a non-profit whose purpose is to develop artificial substitutes for meat. Recently he came across Nick Bostrom's paper on existential risk, and decided to contribute to the field.
Here's a chunk from the conclusion:
We may be poorly equipped to recognize or plan for extinction risks. We may not be good at grasping the significance of very large numbers (catastrophic outcomes) or very small numbers (probabilities) over large timeframes. We struggle with estimating the probabilities of rare or unprecedented outcomes. Policymakers may not plan far beyond current political administrations and rarely do risk analyses consider the existence future generations. (For a welcome exception, see Kent 2004.) We may unjustifiably discount the value of future lives. Finally, extinction risks are classic market failures where an individual enjoys no perceptible benefit from her investment in risk reduction. Human survival may thus be a good requiring deliberate policies to protect.
It might be feared that consideration of extinction risks would lead to a reductio ad absurdum: we ought to invest all our funds in asteroid defense, for instance, instead of AIDS, pollution, world hunger or other problems we face today. However, even if it were found that reducing extinction risks is highly cost-effective, it would not imply that public funds should be spent on asteroid defense, et al. at the exclusion of all other public programs. Many programs reduce extinction risk by maintaining a healthy, educated, and content population, and should be seen as part of a portfolio of risk-reducing projects.
In the concluding chapter of Reasons and Persons, Parfit (1984) wrote:
I believe that if we destroy mankind, as we now can, this outcome will be much worse than most people think. Compare three outcomes:
1. Peace
2. A nuclear war that kills 99% of the world’s existing population
3. A nuclear war that kills 100%2 would be worse than 1, and 3 would be worse than 2. Which is the greater of these two differences? Most people believe that the greater difference is between 1 and 2. I believe that the difference between 2 and 3 is very much greater. . . . The Earth will remain habitable for at least another billion years. Civilization began only a few thousand years ago. If we do not destroy mankind, these thousand years may be only a tiny fraction of the whole of civilized human history. The difference between 2 and 3 may thus be the difference between this tiny fraction and all of the rest of this history. If we compare this possible history to a day, what has occurred so far is only a fraction of a second.
This paper tentatively supports Parfit’s conclusion. Human extinction in the next few centuries could reduce the number of future generations by thousands or more. We take extraordinary measures to protect some endangered species from extinction. It might be reasonable to take extraordinary measures to protect humanity from the same. To decide whether this is so requires more discussion of the methodological problems mentioned here, as well as research on the extinction risks we face and the costs of mitigating them.
Mitigating existential risk should be the human species' number one priority, right now. If you want to help, mention the idea to your friends, organize your thoughts on the topic, contribute to mailing lists discussing it, blog about it, and lend risk-mitigating organizations your financial support.
As I write this, the nuclear tension on the Korean peninsula regrettably continues... while New York Magazine pokes fun at concern about risk.
Jason's bio:
Jason Matheny is a Ph.D. student in Agricultural Policy at the University of Maryland and a researcher at the Bloomberg School of Public Health at Johns Hopkins University, where he studies the health and environmental consequences of animal agriculture. He directs New Harvest, a nonprofit that funds research on in vitro meat, and previously worked on public health projects for the World Bank and the Center for Global Development.
Eliezer Yudkowsky on “Genius Fools” in AGI
Mr. Yudkowsky harps on about Friendly AI on the Extropy list:
Rafal Smigrodzki wrote:
> On 10/4/06, Eliezer S. Yudkowskywrote:
>
>>No one gives a damn about AI research, and until that changes, changes
>>in other government policies aren't going to affect anything one way or
>>the other.
>
> ### You are right here, but you may dismiss my worries a bit too
> early: One day in the not-too-distant future some high-level
> government officers will approve major funding for a general AI, and
> today's changes in political culture are likely to have an impact on
> the moral and intellectual qualities of these officers.It really doesn't matter whether an improperly shaped intelligence
explosion is set off by altruistic idealists, or patriots, or
bureaucrats, or terrorists. A galaxy turned into paperclips is just as
much of a tragedy either way.> As you note,
> even the best intentions of idealistic researchers may backfire
> horribly, but the intellectual offspring of evil functionaries is
> simply guaranteed to fry us all.Not "may" backfire horribly. Rafal, you work in biology. You know from
experience that biology is difficult. You've probably also met people
who think that biology is easy enough for them to make up cool-sounding
theories about it. Think about what behavior you've witnessed from most
AGI wannabes. Ask yourself if they seem to understand intelligence as
solidly as you understand biology. Now imagine someone with that degree
of understanding of biology, trying to build something really wonderful
and exotic - say, a retrovirus that reverses Alzheimer's disease. It
doesn't matter if they're motivated by the will to heal, or pure greed -
they can't do it at that level of understanding, end of story.If some high-level government officers approve major funding for a
general AI, they'll find a prestigious AI researcher, someone who's been
failing at the problem longer than anyone else, and appoint a
blue-ribbon committee containing not a single researcher under the age
of 40 to oversee the project.Google might be worrisome someday. But for now, Google's founders
apparently believe that it is Google's destiny to become an AI and this
will happen by magic, when their databases grow large enough to be
blessed by the Emergence Fairy. I am more worried about a single one of
Google's resident geniuses spending their days off on an AI project.
Partial understanding is dangerous; great big computer clusters are not.
Which would you be more afraid of, Rafal, an environmental extremist
with a grade school education and a multibillion dollar state-of-the-art
medical research lab, or an environmental extremist with a Ph.D. and a
hundred thousand dollars?Newton wasted much of his life on Christian mysticism, so it's possible
to make great discoveries and still not understand how to think. But
the art of an FAI creator is thought itself - to see cognition as an
engine which accomplishes work. I cannot visualize someone discovering
the basic organizing principles of intelligence, in a scientific field
presently full of confusion and dismay, without their having an
intuitive appreciation of how rationality works. It would be like
Newton not being able to see mathematics incarnate in a falling apple.
It would be like Carnot not being able to visualize heat flows. Maybe
someday there'll be textbooks that teach idiots how to build AI, but to
discover it yourself, you need an intuitive appreciation of the
results achieved by cognitive workflows. You cannot master that art
from scratch and afterward still be so poor a rationalist as Newton.
No Sharia zombie could get one tenth of the way to independently
discovering how to build and shape an AGI, and still remain a Sharia zombie.A genius fool might stumble over powerful forces they can't control. A
genius fool might build something that sorta works, in a poorly
understood way, but works just well enough to cross the threshold of
recursive self-improvement. A genius fool might spend their days
working with evolutionary algorithms, knowing nothing of Bayescraft.
After all, natural selection originally built humans using no abstract
understanding whatsoever. So, yes, genius fools are dangerous - but not
because they might decide to shape a Sharia enforcer. Genius fools are
dangerous because they can't shape minds at all. It doesn't matter
whether their intentions are good or bad, because the outcome bears no
relation to their intentions for it.It requires a precise understanding to master the shaping art, to set
off an intelligence explosion that relates at all to your original
intentions. If there is any recognizable resemblance you must have
known a very great deal indeed. It requires considerably less
understanding than that, to appreciate how powerful the forces are which
you intend to mess with. No one can know how to build a nuclear reactor
that operates in a state of controlled criticality, without being able
to calculate the doubling time of a chain reaction. No one can
understand intelligence and still think of an intelligence explosion as
a weapon in short-term political squabbles. An astrophysicist knows the
power output of a star, and that a star is brighter than a campfire. An
astrophysicist is not going to say, "Hey, let's set off a supernova to
bust in that terrorist bunker." Maybe the astrophysicist's superiors
fail to comprehend anything about a supernova except that it makes a
bang - but the astrophysicist can't help but know what would happen. If
the astrophysicist is ordered to do it anyway, told to set off a
supernova or face a court martial, the astrophysicist is not going to
shrug and go ahead with it. Imagine this as a literal scenario, not a
metaphor. If you want to keep your job, you might try all sorts of
dodges, but you wouldn't actually set off a supernova - not when you
knew damned well that the Earth would end up as not-even-vapor.So can we have enough of this silly scenario where someone creates and
successfully shapes an intelligence explosion, while simultaneously not
noticing that they are messing with powers vast enough to reconfigure
galaxies? Can we stop pretending that someone might build and shape
an AGI, by the exercise of their precise understanding, for the sake of
a cute little toy weapons system? A genius fool might accidentally fry
the planet, turn our galaxy into paperclips. But no genius fool could
accomplish any specific purpose, anything with a nonzero correlation to
what they originally had in mind. Paperclips are paperclips - whether
that outcome is brought about by terrorists or patriots, universities or
Google, idealists or villains. All you're arguing about is whose logo
will look the prettiest on the tombstone.This is a counterintuitive point, I know.
It is much more satisfying to cheer the Blues, or boo the Greens, or
whatever your accustomed chariot-racing allegiance may be.But this problem is more difficult than that, and old habits will not
solve it.--
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
Mr. Yudkowsky seems very fond of the nuclear criticality metaphor lately. Unfortunately much of this arguing is for naught, because 90% of the folks on the extropy list, and 99.99% of people in the larger world, don't even buy the idea that a self-improving Artificial Intelligence is going to learn or progress any faster than a human being reading a book or brandishing a hammer and nails.
Making Water from Thin Air
From Wired news:
By Audrey Hudson
02:00 AM Oct, 06, 2006
A company that developed technology capable of creating water out of thin air
nearly anywhere in the world is now under contract to nourish U.S. soldiers
serving in Iraq.
The water-harvesting technology was originally the brainchild of the Pentagon's
Defense Advanced Research Projects Agency, which sought ways to ensure
sustainable water supplies for U.S. combat troops deployed in arid regions like
Iraq.
"The program focused on creating water from the atmosphere using low-energy
systems that could reduce the overall logistics burden for deployed forces and
provide potable water within the reach of the war fighter any place, any time,"
said Darpa spokeswoman Jan Walker.
To achieve this end, Darpa gave millions to research companies like LexCarb and
Sciperio to create a contraption that could capture water in the Mesopotamian
desert.
But it was another company, Aqua Sciences, that developed a product on its own
and was first to put a product on the market that can operate in harsh climates.
"People have been trying to figure out how to do this for years, and we just
came out of left field in response to Darpa," said Abe Sher, chief executive
officer of Aqua Sciences. "The atmosphere is a river full of water, even in the
desert. It won't work absolutely everywhere, but it works virtually everywhere."
Sher said he is "not at liberty" to disclose details of the government
contracts, except that Aqua Sciences won two highly competitive bids with "some
very sophisticated companies."
He also declined to comment on how the technology actually works.
"This is our secret sauce," Sher said. "Like Kentucky Fried Chicken, it tastes
good, but we won't tell you what's in it."
He did, however, provide a hint: Think of rice used in saltshakers that acts as
a magnet to extract water and keeps salt from clumping.
"We figured out how to tap it in a very unique and proprietary way," Sher said.
"We figured out how to mimic nature, using natural salt to extract water and act
as a natural decontamination.
"Think of the Dead Sea, where nothing grows around it because the salt
dehydrates everything. It's kind of like that."
The 20-foot machine can churn out 600 gallons of water a day without using or
producing toxic materials and byproducts. The machine was displayed on Capitol
Hill last week where a half-dozen lawmakers and some staffers stopped by for a
drink.
"It was very interesting to see the technology in action and learn about its
possible implementation in natural disasters," said Rep. E. Clay Shaw Jr., a
Republican from Florida whose hurricane-prone district includes Fort Lauderdale.
"It was delicious," Shaw said.
Jason Rowe, chief of staff to Rep. Tom Feeney, another Florida Republican,
called the technology "pretty impressive."
"I was pretty blown away by the things it's able to do," Rowe said. "The fact
that this technology is not tied to humidity like others are makes it an
attractive alternative for military bases in the Mideast where humidity is not
really an option.
"It seems like it's a cheaper alternative to trucking in bottled water, which
has a shelf life," said Rowe, who described himself as a fiscal hawk.
Once deployed, the machines could reduce the cost of logistical support for
supplying water to the troops in Iraq by billions of dollars, said Stuart Roy,
spokesman of the DCI Group, Aqua Sciences' public affairs firm.
The cost to transport water by C-17 cargo planes, then truck it to the troops,
runs $30 a gallon. The cost, including the machines from Aqua Sciences, will be
reduced to 30 cents a gallon, Roy said.
Several systems on the market can create water through condensation, but the
process requires a high level of humidity.
Aqua Sciences' machines only require 14 percent humidity, Roy said. "That's why
this technology is superior and why they are getting the contracts."
---
Love that last line.
Ben Goertzel on a Positive Singularity
My favorite quote from the video:
The human brain and the human mind are a mess. And the reason they're a mess is not just because evolution creates messes. The reason they're a mess is that implementing intelligence on a limited computational substrate, like a brain, or like a network of computers... this process necessarily leads to a complicated and heterogeneous architecture, without the elegance that computer scientists or mathematicians want to see.
It's funny how many people there out there that still think evolution is elegant and magically efficient, like God's hand... evolution only works by its own incredibly shitty standards. This is not just me blowing hot air - it can be objectively proven by designing artificial systems that do the same thing as biology, only with orders-of-magnitude better performance.
For more Ben Goertzel action, check out a recent podcast interview of his at SingularityU.org.
He's still using the technological-asymptote definition of the Singularity rather than the original one which focuses more exclusively on transhuman intelligence... I'm getting so tired of arguing this that I might as well follow Eli's example and start talking about "intelligence explosions" rather than "Singularity", which seems too loaded to even touch sometimes.
Putting Antarctica in the Microwave
(Image of Antarctica without its ice shield.)
So Antartica was warm only 34 million years ago. I originally learned this from Lovecraft, but it's been confirmed by a study of fish teeth. When the ancient continent of Gondwana broke up, it formed the continent of Australia, severing poor Antarctica. A powerful circumpolar current of cold water led to a 9°C temperature decrease around the continent. According to this Eurekalert article, "Mediterranean sun seekers should thank Antarctica":
Europeans who enjoy the Mediterranean's warm climate should thank Antarctica for their good fortune.
Climate modelling by Australian scientists at the University of New South Wales reveals that Antarctica's icy sea currents allow the balmy Gulf Stream to dictate warm weather conditions over much of the North Atlantic.
"The Gulf Stream's climate dominance over Europe relies on events some 30 millions years ago, when Antarctica started to freeze following the final break-up of Gondwana, the great southern continent, according to Dr Matthew England, whose research with PhD student Willem Sijp was published in the Journal of Physical Oceanography.
"The loss of a 'land bridge' between Australia and Antarctica effectively isolated Antarctica and depressed its temperature by up to 9 degrees C," says Dr England. "Once it was cut adrift in the Southern Ocean, a powerful circumpolar current was established that separated Antarctica from warm subtropical waters to the north."
The Antarctic circumpolar current is a massive force. It flows at the rate of over 100 million cubic metres of water a second and takes eight years to circumnavigate the frozen continent. As a result, the icy waters of the polar reaches of the Southern Ocean don't dominate global ocean currents and climate as they did 30 million years ago.
The Gulf Stream is a super warm Atlantic current that moves tropical waters north towards Europe. As it does so it releases heat into the atmosphere that gives adjacent countries a warmer climate than they would otherwise have.
"This means that Portugal and other Mediterranean countries have a much warmer climate than places on the same latitude, such as New York," says England, who is co-director of the UNSW Centre for Environmental Modelling and Prediction.
"After the Gulf Stream waters release their heat, they cool, sink deep into the ocean, and flow south to eventually resurface in the southern hemisphere oceans.
The waters then make their way northward via various ocean routes, being rewarmed in the tropics before returning to the North Atlantic. But the driver for these ocean currents is in the North Atlantic, not the Antarctic, because of the isolating effect of the circumpolar current in the far Southern Ocean.
"Having Antarctica cut-off from the subtropics because of the Southern Ocean reduces the icy continent's impact on the global climate system", says Dr England. "We've shown that the isolation of Antarctica is necessary for the Gulf Stream's warming of Europe to be so pronounced".
So if it weren't for Antarctica's breakaway 30 million years ago, lazing by the Mediterranean today would be a much chillier affair.
This post was originally going to be a proposal to build a 1,000 km sea bridge to block the circumpolar currents in the 3 km-deep ocean between Cape Horn and the Antarctic peninsula. But now it turns out that building such a bridge would actually decrease the temperature in other parts of the world, so it would be quite unacceptable!
I like Antarctica, quite a lot actually. It's remote, exotic, etc. Its only problem is that it's incredibly cold and dry. Apply heat, and we can start to solve both problems. Before we even think about terraforming other planets, we should make our entire home planet inhabitable.
All we have to do is increase the temperature of the totality of the circumpolar current by 9°C and we're off to an excellent start. It costs about 4 kJ to heat 1kg of water by 1°C, and if the Eurkalert article is right, the circumpolar current is 100 million cubic meters of water per second for eight years, which is 25 million cubic kilometers altogether. By comparison, the total volume of water on earth is known to be around 1400 million cubic km.
A cubic km of water weighs a billion metric tons. So the circumpolar current weighs around 2.5 x 10^16 metric tons, which would cost 10^11 terajoules to heat 9°C. Since a watt is a joule per second, to rack up 10^11 terajoules in a year would require only a 7750 terawatt (TW) power plant, a bit larger than the earth's current power consumption of 4 TW. But for that cost we get an entire new continent to colonize!
We'd need to build 1,550,000 5GW solar satellite facilities to meet the power demands of this project. This works out to about 7,750,000 km² of orbiting solar panels, equal to the area of Australia. As big as this sounds, it would only occupy a relatively small portion of the space available in geosynchronous orbit.
Nature Nanotechnology Endorses MNT
Via Nanodot, one of the articles in the recent inaugural issue of the scientific journal Nature Nanotechnology is titled "Making Molecular Machines Work" by Wesley Browne and Ben Feringa. Here is a snippet from the conclusions:
The exquisite solutions nature has found to control molecular motion, evident in the fascinating biological linear and rotary motors, has served as a major source of inspiration for scientists to conceptualize, design and build — using a bottom-up approach — entirely synthetic molecular machines. The desire, ultimately, to construct and control molecular machines, fuels one of the great endeavours of contemporary science. The first primitive artificial molecular motors have been constructed and it has been demonstrated that energy consumption can be used to induce controlled and unidirectional motion. Linear and rotary molecular motors have been anchored to surfaces without loss of function — a significant step towards future nanomachines and devices. Furthermore, it has been demonstrated unequivocally that both linear and rotary motors can perform work and can move objects. However, although the first applications of molecular motors to the control of other functions have been realized, the whole field is still very much in its infancy and offers ample opportunity in the design of nanomechanical devices.
Major challenges in the development of useful nanomachines remain, such as the development of fast and repetitive movement over longer time frames, directional movement along specified trajectories, integration of fully functional molecular motors in nanomachines and devices, catalytic molecular motors, systems that can transport cargo and so on. As complexity increases in these dynamic nanosystems, mastery of structure, function and communication across the traditional scientific boundaries will prove essential and indeed will serve to stimulate many areas of the synthetic, analytical and physical sciences. In view of the wide range of functions that biological motors play in nature and the role that macroscopic motors and machines play in daily life, the current limitation to the development and application of synthetic molecular machines and motors is perhaps only the imagination of the nanomotorists themselves.
Very poetic! Nature Nanotechnology is the nanotech spinoff of the popular Nature.
Matt Bamberger Starts AGI Project
Matt Bamberger has worked at Microsoft and Valve Software (the makers of Half-Life) for over 20 years. A few months ago, he left his job at Valve to work at his new AGI company, Intelligent Artifice. Bamberger joins Ben Goertzel, Ari Heljakka, Eliezer Yudkowsky, Marcello Herreshoff, Steve Omohundro, Peter Voss, Marcus Hutter, Jurgen Schmidhuber, Andrew Rogers, Jeff Hawkins, and several dozen others in the quest for real AI. Having met with him twice this year, I can say that I feel confident about Matt's approach to risk management in advanced AI. His stance is that human-equivalent AI is potentially extremely dangerous, and that every possible precaution must be taken to ensure that the first such AI is helpful to humanity rather than hurtful. Follow the first link in this post to his insightful remarks on the subject - they are a fine example of a classic Singularitarian view of the situation, a view which is thankfully spreading.
Artificial Bacteria Causing Mass Extinction?
Christopher Phoenix invites us to find a flaw in this chain of reasoning:
1) Bacteria are crucially important to the biosphere.
2) Bacteria are largely kept in check by bacteriophages.
3) Existing bacteriophages can only work with standard DNA coding.
4) Artificial or modified bacteria may have non-standard DNA.
coding - of several types - from several different technologies.
5) Invasive species are characterized by having fewer predators and parasites.
6) Novel-DNA bacteria may get a huge relative fitness boost from being
immune to bacteriophages.
7) Novel-DNA bacteria may become an invasive species.
8) Invasive species frequently crowd out native species.
9) Invasive novel-DNA bacteria may significantly distort the food
chain and other aspects of the ecosystem.
10) Ecosystem disruption at a foundational level may cause large-scale
extinction at higher levels.
Any takers?
Anne C. on the Future of Life
From the WTA-talk mailing list, a great post by Anne C. responding to the suggestion that the universe will inherently get boring after living long enough in it:
Why must time be "killed" at all? Why cannot it be savored, lived during, loved during? I don't think of anything as "killing time" waiting for some sort of ultimate end, but rather, as spending time doing and experiencing.
And it's not as if the universe (or even the multiverse if you subscribe to such theories) is static and/or likely to remain static; evolution is happening all the time even though it's often too slow for us to directly observe. The laws of physics might indeed constrain the morphologies and cognitive faculties of organisms and objects (that is, only things capable of existing in our universe will exist in our universe, and this means that the set of such things is necessarily finite), but who is to say that such laws are even eternal at all?
Of course, an upset to such fundamentals might destabilize survival capacity even of a superintelligence, but that would of course provide yet another supergoal: how to deal with changes to the laws of physics when one's own existence is owed to a prior set of laws.
But at any rate, I'm not convinced that over-arching "supergoals" are all that important in the grand scheme of things; once we've dealt with basic survival issues, everything on top of that is rather subjective. Perhaps this sense that everyone and everything needs a supergoal on the order of survival is merely an artifact stemming from the very powerful motivation FOR survival that we've needed all these years.
Why not a series of personal mini-goals? Paint a picture, write a novel, climb a tree, beat Zelda again, prove a theorem, build a supercollider, make cookies, play with kittens, tour China, create your own model of China, invent a new kind of noodle, plug your brain into a machine that lets you watch your dreams, dream, videotape penguins, search for extraterrestrials, grow your own extraterrestrials, run infinite simulations, see how high you can count, knit a blanket, knit spacetime, start your own television show, make holograms, build an entire city out of Duct tape, breed immortal dragonflies, watch stars form and wax and wane over millennia, push on galaxies...you get the idea.
Sure, there might be some sort of "ultimate meaning". However, I don't claim to know that there is such a meaning -- and I generally operate under the assumption that there isn't, since the individual human viewpoint is so incredibly subjective, and I don't stake my existence or enjoyment of life on the potential of that sort of meaning.
Certainly, I'll keep learning and exploring and delving into scientific mysteries, but this process is just as rewarding if there's nothing but knowledge at the "end" (if such an end even exists) or if there's some sort of unforeseeable complex prize that would keep anyone, even a superintelligence, happily occupied basically forever. The process, and what is experienced during the process, is just as important as any sort of goal, "super" or otherwise. And knowledge is no "booby prize"; lacking all that knowledge we stand to gain still, who knows what we might think of to do with it once we have it?
Evolution has granted us the capacity to enjoy things. Being able to enjoy life, regardless of whether you're being chased by a ravenous tiger or not, is an adaptation in the sense that it motivates people to continue existing. Which, of course, allows for the perpetuation of both genes and memes -- if our ancestors had all been suicidally bored between mammoth hunts, we wouldn't be here right now. There's a term for people unable to find significance and meaning in everyday life: clinical depression.
And honestly, I don't see any value in trying to postulate that us, with our unaugmented meat-brains, somehow can conceive of the sorts of meaning, fun, and activity that would be available to augmented persons or superintelligences.
Language like this:
"It only remains to deplete the universe of all the interesting experiences it has to offer."
sounds incredibly nihilistic to me. Do you really see having experiences as "depleting" anything? I certainly don't. It is important to remember that *depth* of experience is every bit as significant as breadth. And as far as I can tell, there is no predetermined limit to the depth one might find in art, music, or mathematics. And I've never found knowing how or why something works to make it any less beautiful or amazing -- to suggest that understanding something means you've "used it up" is, frankly, incoherent to me. All I can say is: speak for yourself!
Meaning isn't some kind of externally-sourced quantity, but a product of symbiosis. I, for one, plan to meet the universe at least halfway in that regard.
Reminds me of some of the arguments in Singularity Fun Theory. By the way, did you know that Madonna is planning to fly into space in 2009?
Leading Countries Race Towards Nanotech
Via TNTLog, the "Nanotech Dragon":
This is a scatterplot of current nanotech funding and scientists/engineers per capita in various countries. The US, Japan, and China are clearly in the lead today - but since "nanotech" is defined very widely, this doesn't necessarily reflect who will develop molecular manufacturing first. Here's a bit from Ed Regis' book Nano, back from when the word "nanotechnology" meant molecular manufacturing:
[Drexler's] reasoning here was that if nanotechnology was going to be developed anyway, whether he helped it along or not, then it was crucial that it be invented here in America, or at least by one of the free democracies wherever they were located, East or West. This was crucial because the first nation to develop nanotechnology would thereby become the world's dominant power, "the Leading Force".
That nation, whoever it was, could build weapons that no other country would have defenses against. Its citizens would become healthy, wealthy, and young overnight. It would be Them against everyone else.
Moreover, it was not out of bounds to imagine one of the more unspoiled worldy monarchies being the first to develop nanotechnology. Nanotechnology research, after all, was not "big science" in the usual sense. You didn't need anything like a Manhattan Project or an Apollo program or a Superconducting Supercollider effort to get the thing going. Conceivably, you could do it in a garage. You could do simulations of molecular machines on a personal computer; you could create billions of molecular structures in a test tube; you could custom-make DNA in a desktop synthesizer. All you needed for the great breakthrough was a laboratory, some extremely smart people and programming, and lots of luck at getting things right.
The above was written in 1995. "Healthy, wealthy and young", perhaps not overnight, but after only a few years is indeed imaginable. Tabletop or industrial nanofactories would allow their owners to fabricate any quantity of medical equipment with raw materials and the engineering design being the only costs. To truly defeat old age will require a thorough understanding of how the 7 mechanisms of senescence do damage and how to heal that damage without unhealthy side effects. Health will be boosted greatly by injecting ourselves with artificial antibodies and bacteriophages when they are developed, which should be before the closing of the second decade of this century. Wealth is probably the easiest item on the list to achieve, because what we consider wealth is largely based on material products, which can be manufactured in abundance when fabrication processes achieve high throughputs and are entirely automated.
Say you have a 10 kg nanofactory invented in an arbitrary country on January 1st, 2020. Let's say that the design is similar to the Phoenix nanofactory, in which case we'd work with the following assumptions:
The size, mass, energy requirement, and duplication time of this nanofactory design depend heavily on the properties of the fabricator. Sections 8.2, 8.3, and 8.4 quantify these relations. With the assumptions made in those sections, a tabletop nanofactory (1x1x1/2 meters) might weigh 10 kg or less, produce 4 kg of diamondoid (~10.5 cm cube) in 3 hours, and require as little as fifteen hours to produce a duplicate nanofactory.
Say that this first nanofactory is used to make a duplicate nanofactory, then both nanofactories are used to make duplicates, and so on, until you have 200 million units, ready for distribution to the majority of households in the nation. How long would this take? Under 28 duplication cycles, or approximately 18 days. In our model that would be January 19th, 2020. Assuming another week for distribution, this would put nanofactories into most homes in under a month since the technology was initially completed. To compare, the time it took for the Internet to be adopted by 50% of American households since its invention was about 15 years. The MP3 player and cell phone have arguably taken far less time to achieve 50% adoption, more like a few years. Nanofactories could achieve 50% adoption in weeks, possibly months or years if the price is kept artificially high, which is Michael Vassar's scenario in his Corporate Cornucopia paper. In any case, once a nation has 200 million nanofactories and the necessary raw materials, it could theoretically fabricate 2.3 billion metric tons of product per year, mostly durable goods, a productivity rate much greater than those seen in contemporary economies.
(If you read Greg's post in the comments, you'll see that early nanofactories would have high power and feedstock requirements, so the exponential explosion outlined above would be rather delayed. My model is partially based on the assumption that, unless a nanofactory has relatively low power requirements and can accept non-perfect feedstock, it isn't really going to be mass produced anyway.)
The question is, will the technology be available to everyone, or will it be guarded by a jealous few?
On the downside, restricting nanotechnology would have a horrible negative effect on many of the poorest people in the world, who have little access to housing, electricity, water, and other basic needs. Because the marginal cost of manufacturing an additional product using a nanofactory is so close to zero, people in poverty have the most to gain if nanotech is widely adopted, and the most to lose if it is restricted.
On the other hand, nanotech opens up a dangerous Pandora's box of problems that few people have even begun to understand. Nanoengineered weapons are in fact one of the greatest threats to humanity's future that has yet been imagined. Even if the threat of extinction were as low as 1/100, that's a 1/100 chance of the entire human future being destroyed, a future that potentially consists of trillions and trillions of beings experiencing worthwhile lives. It would be ethically prudent to hold back this technology until we can be better reassured that we can handle it with minimal risk. Unfortunately, in the real world you can't hold back a technology once international research gets started and investors are pouring money into it, which has already happened for nano. The upshot is that it might actually be beneficial for humanity if nanotechnology did end up being released to the public slowly, or in low-performance versions that make for a more fluid transition from manufacturing technologies of the past. But is that really practical once other companies and governments see the tremendous power of the technology and start developing their own versions?
Simulations
Above is a video of a state of the art physics simulator for the PC and PS3. I am reminded of Mitch Howe's "Simulations, a Primer" which states, "The simulations ahead could be so advanced that an individual would not even recognize being in a simulation without being informed. These simulations could be so versatile that they would become the preferred reality of mankind's future." Last month I posted a short story by John Distazo about a pious human revived in a simulated environment.
Thanks to Our Technological Future for the video. Some of the segments, near the end especially, are truly amazing.