All the way back in April 2002, my singularitarian colleague Gordon Worley made a post to the SL4 mailing list that I thought explained technological risk quite well. As it’s still stuck in my head today, I decided to dig it up and post it here. The first part is a guy he’s responding to.

> So to let other people do the talking, I recently found an article on
> Wired by David Brin about how society could regulate itself by everybody
> basically being able to spy upon each other. Ramp up the technology a
> bit with prehuman intelligences helping each human do the spying. And
> you could get a stable society.

I doubt this. Let’s jump in the Way Back Machine to see why …

So, here I am, in the ancestral environment. I’m amongst a tribe of
about 150 humans. They live close together and can spy on each other
fairly easily. While a few can hide a little, human societies work
because you can’t go against society without some consequences, so
there’s always someone sticking his or her nose into someone else’s
business. One day Unk is caught not sharing the chicken he caught.
Well, everyone knows that Unk’s family has gone without meet for a few
weeks, so they let this pass. A few days later, he catches another
chicken and again shares none of it. When he does this the third time,
people are pissed. The solution: beat him. Maybe rape his wife and
kill one of his children, too. Unk is upset and he fights a few of his
neighbors and manages to bloody them a little. After that Unk is a good
human and mostly gets along in society.

Now we take the Way Back Machine into the future (yes, we’re going
negatively backwards):

So, here I am in the year 2015 where nanotech spy technology is
everywhere. Just yesterday Knu (the great great great… great
grandson of Unk - unless of course the village had managed to get his
wife pregnant that time Unk did not share his chicken) was caught by his
neighbors cracking the encryption on a DVC (digital video cube) and
watching every Inspector Clouseau film without paying $5 a minute to the
MPAA like all other good, god-fearing people do. Since the MPAA doesn’t
know about this, his neighbors kindly decide to inform them. With the
press of a button, the MPAA sends out nanobots to Knu’s home and has it
liquidated. Literally. Knu, soaking wet and pissed, is still fully
capable, unlike his long dead relative Unk who was badly bruised and
only able to throw a punch or two before a fight was over. Using the
assembler at the nearby Kinko’s, Knu builds and sends out some nanobots
to liquidate this neighbors houses to get revenge. But, as it turns
out, Knu is not the best programmer, so his nanobots accidently
liquidate the Earth. The dolphins enjoy all the extra space (now
they’re really thanking humans for all the fish :-P) but humans are dead
or drowning.

Back in the e-mail I’m writing right now:

So, as we see, humans don’t really change, just the technology does.
Consequently, humans with technology that can destroy the world are very
dangerous. It’s mostly through luck that we have managed not to nuke
ourselves out of existence or back to the Stone Age. One day someone
with nuclear weapons or nanotech or something is going to probably kill
everything on Earth. For many of us, the push to reach the Singularity
as quickly as possible is that the longer we wait the more likely it is
that we’ll be caught on Earth when someone decides to blow it up.

Well and humorously put!