Accelerating Future Transhumanism, AI, nanotech, the Singularity, and extinction risk.

Singularity Letter Again

First draft of aforementioned letter:

To whom it may concern,

Throughout the last half-century, the field of Artificial Intelligence (AI) has been progressing steadily. Today, AI systems help humans accomplish a variety of tasks, from filtering our e-mail to preventing credit card fraud.

If progress continues, the problem-solving abilities of AI systems will approach and then surpass the brightest human minds. These systems will acquire the ability to improve their own source code without human assistance, giving rise to smarter versions that no human team could program directly. These versions will be intelligent enough to have influence over the real world, including improving their underlying hardware.

Humanity has no experience dealing with a species smarter than it. For this reason the creation of Artificial Intelligence should be approached with caution. To distinguish between narrow-purpose AI and AI designed specifically for general intelligence and self-improvement, the term “Artificial General Intelligence” (AGI) was coined.

The creation of a smarter-than-human species has been called a ‘singularity’ by futurists, by analogy to singularities in cosmology. In cosmology, the singularity at the center of a black hole refers to the point at which the laws of physics as we know them cease to apply. This doesn’t imply that laws vanish, but simply that they change in ways we can’t foresee. The analogy is not perfect, but a cosmological singularity captures some of the uncertainty we will experience when confronting a smarter-than-human species for the first time.

The creation of AGI would be unlike prior technological milestones. AGI would be capable of independently intiating actions and making choices, inventing new technologies, and solving difficult problems. Created by human programmers rather than evolution and natural selection, AGI will not necessarily be motivated by the same things that motivate us, find challenging the same obstacles that challenge us, or arrange themselves in social configurations the same way that we do.

The choices an AGI makes when improving upon its own programming will stem from its initial top-level motivations. To minimize the probability of rogue AI, researchers working towards generally intelligent systems need to instill them with positive goal structures – altruism, benevolence, philanthropy. Because the benefits of successful AGI would be so large, arguing about the specifics of the distribution scheme is not as important as ensuring that everyone receives them.
It is our position that AGI cannot be avoided entirely. As AI researchers and futurists, it is our responsibility to do as much as we can to ensure a positive outcome. We have begun by emphasizing the importance of Artificial Intelligence and the fundamental difference between self-improving AI systems and the AI systems of today.

We do not expect to confront these questions in the distant future. Smarter-than-human AI is something we anticipate arriving in the next years or decades, not centuries. The creators of the first Artificial General Intelligence are people that could very well be alive today. The policies and precedents we set in the present will have influence over what will happen in the future. The potential benefits of AI exceed those of any other technology. We will do our best to ensure they are accessible to all.

Signed,