First draft of aforementioned letter:
To whom it may concern,
Throughout the last half-century, the field of Artificial Intelligence (AI) has been progressing steadily. Today, AI systems help humans accomplish a variety of tasks, from filtering our e-mail to preventing credit card fraud.
If progress continues, the problem-solving abilities of AI systems will approach and then surpass the brightest human minds. These systems will acquire the ability to improve their own source code without human assistance, giving rise to smarter versions that no human team could program directly. These versions will be intelligent enough to have influence over the real world, including improving their underlying hardware.
Humanity has no experience dealing with a species smarter than it. For this reason the creation of Artificial Intelligence should be approached with caution. To distinguish between narrow-purpose AI and AI designed specifically for general intelligence and self-improvement, the term “Artificial General Intelligence†(AGI) was coined.
The creation of a smarter-than-human species has been called a ‘singularity’ by futurists, by analogy to singularities in cosmology. In cosmology, the singularity at the center of a …