Gordon Worley on Nanorisk Thursday, Jan 25 2007
risks 10:25 pm
All the way back in April 2002, my singularitarian colleague Gordon Worley made a post to the SL4 mailing list that I thought explained technological risk quite well. As it’s still stuck in my head today, I decided to dig it up and post it here. The first part is a guy he’s responding to.
> So to let other people do the talking, I recently found an article on
> Wired by David Brin about how society could regulate itself by everybody
> basically being able to spy upon each other. Ramp up the technology a
> bit with prehuman intelligences helping each human do the spying. And
> you could get a stable society.I doubt this. Let’s jump in the Way Back Machine to see why …
So, here I am, in the ancestral environment. I’m amongst a tribe of
about 150 humans. They live close together and can spy on each other
fairly easily. While a few can hide a little, human societies work
because you can’t go against society without some consequences, so
there’s always someone sticking his or her nose into someone else’s
business. One day Unk is caught not sharing the chicken he caught.
Well, everyone knows that Unk’s family has gone without meet for a few
weeks, so they let this pass. A few days later, he catches another
chicken and again shares none of it. When he does this the third time,
people are pissed. The solution: beat him. Maybe rape his wife and
kill one of his children, too. Unk is upset and he fights a few of his
neighbors and manages to bloody them a little. After that Unk is a good
human and mostly gets along in society.Now we take the Way Back Machine into the future (yes, we’re going
negatively backwards):So, here I am in the year 2015 where nanotech spy technology is
everywhere. Just yesterday Knu (the great great great… great
grandson of Unk – unless of course the village had managed to get his
wife pregnant that time Unk did not share his chicken) was caught by his
neighbors cracking the encryption on a DVC (digital video cube) and
watching every Inspector Clouseau film without paying $5 a minute to the
MPAA like all other good, god-fearing people do. Since the MPAA doesn’t
know about this, his neighbors kindly decide to inform them. With the
press of a button, the MPAA sends out nanobots to Knu’s home and has it
liquidated. Literally. Knu, soaking wet and pissed, is still fully
capable, unlike his long dead relative Unk who was badly bruised and
only able to throw a punch or two before a fight was over. Using the
assembler at the nearby Kinko’s, Knu builds and sends out some nanobots
to liquidate this neighbors houses to get revenge. But, as it turns
out, Knu is not the best programmer, so his nanobots accidently
liquidate the Earth. The dolphins enjoy all the extra space (now
they’re really thanking humans for all the fish :-P) but humans are dead
or drowning.Back in the e-mail I’m writing right now:
So, as we see, humans don’t really change, just the technology does.
Consequently, humans with technology that can destroy the world are very
dangerous. It’s mostly through luck that we have managed not to nuke
ourselves out of existence or back to the Stone Age. One day someone
with nuclear weapons or nanotech or something is going to probably kill
everything on Earth. For many of us, the push to reach the Singularity
as quickly as possible is that the longer we wait the more likely it is
that we’ll be caught on Earth when someone decides to blow it up.
Well and humorously put!
While I’m in agreement about the risks of technology magnifying powers of smaller and smaller parties, that problem seems independent of ubiquitous sousveillance.
eg, you can have the same grumpy overreactions spelling disaster without any kind of spying at all. People get plenty pissed about things done in plain sight as it is.
There are probably better arguments against the transparent society, though.
I agree with you that the problems are independent. I don’t see this as an argument against the transparent society in particular, rather than just an argument against human psychology coupled with unrestricted tech systems in general. I think a transparent society overall would be a good thing, although I have some reservations on the military side.
Part of what Gordon is saying here is that a transparent society *wouldn’t be enough* for long-term security – you’d need Friendly AI. Friendly AI. Friendly AI. I repeat it in the hope that people will read up on the subject, and become just as knowledgable about it as transparency. Just because the idea is conceived by someone less well-known, doesn’t make it any less important.
I think that Gordon is mistaking “nanotech” with “fairy magic”. People do that a lot.
It would be REALLY nice if Transhumanists could have a conception of MNT that’s only 10 years behind the times rather than 21 years. Almost as nice as not anthropomorphizing AI. OK, nothing would be even Almost that nice.
Giving individuals or even groups of individuals control of nanotech sounds reminds me of an exchange in an episode of Blake’s 7.
“Could we do the brain surgery ourselves, with instructions from the main computer?” asks Blake.
The reply from another character: “There are quicker ways of killing [the patient] but none as certain.”
I wouldn’t even trust me (and I’ve known me for a long time) with that sort of power. Which makes development of AI even more imperative. It has to be there before nanoassemblers are or we’re going to end up with a lot of messy incidents. But I wouldn’t want nanoassemblers without some sort of AI-run “immume system” to prevent dangerous commands.
Michael, yes it’s clear that MNT can’t “liquidate” homes, that’s not the point here. It’s a story meant to illustrate the contrast between static human nature and increasing technological power. You should give Gordon a little slack here as first of all, he was only 19 or so when he wrote this, and second of all, very very few people understand MNT the way the people around CRN do. (This may be a good thing.)
“Michael, yes it’s clear that MNT can’t “liquidate” homes,”
I’m not a nanotechnologist, but it seems to me to be obvious that any house that can be put together from raw materials by nanotechnology can be taken apart into raw materials by nanotechnology.
Tom: Liquidate was meant to imply “liquidize” (as it were), and even nanotech can’t make room-temp solids behave as liquids (much less **BE** liquids [again, **at room temp""]), although it might be able to accomplish a close approximation with sophisticated utility-fog…hence Michael V’s “purist” (but note the scare quotes) harrumph…
This post has been defunct for a while but there’s a couple of points that might be important, here:
Given the precautionary nature of human society, before ‘MNT’ would be made universally available, it would inevitably be in the control of specific groups, no few of which would have a vested interest in creating “safe-guards” (I.e.; regulation).
All of this, of course, overlooks the ‘threshold’ concept of self-actualizing ‘computer sentience’ when combined with MNT & simple rod-logic machinery. (Or even, god forbid, some variation on Mark Tilden’s analogue neural network arrays.)
The thought I’m driving at here is a simple one; regardless of whether empathetic “superintelligence” can be generated before MNT, it *WILL* exist within a sufficiently close period to MNT that it is irrelevant — in my opinion, as a simple layman on the topic — to be concerned about which comes first. It verges on “chicken and the egg.” (Poor analogy, I know.)
Ack — I worded the first part of my response poorly.
By regulation I meant self-regulating mechanisms embedded into the “nanites” made available to the common populace *AND* ‘hunter-seeker’ nanites designed to do nothing more or less than eliminate rogue designs.
All of that, of course, is fallible — but at least it turns ‘random wild emotional moment of a moron’ into ‘concerted effort of malice’. Which is something.
It IS relevant which one comes first, hugely. Nanocomputing lets you create AI without knowing what you are doing, radically increasing likelihood of UFAI. The wise thing to do would be to withhold from developing MNT at all, and just let a FAI do it. Barring that, it would be nice to have much regulation on nanocomputers. Even basic rod logics could prove a security threat even for non-AI reasons, so there is hope that such levels of computing power will be carefully regulated. (And I am hoping to contribute to that.)
Michael,
Do you make this argument formally, anywhere?
–Nato
Nato, I’m just echoing what Eliezer Yudkowsky has been arguing since 2001. See http://www.singinst.org/ourresearch/publications/CFAI/policy.html
Michael:
I’m too new to investigating transhumanist thought in general to gainsay you much… but I will say that I cannot easily conceive of a scenario in which a self-actualizing sentience (spontaneous awareness, or what have you) would be inherently incapable of empathy; and any ‘designed’ self-reformatting “AI” would inevitably be able to ‘code’ out of itself (I personally think that thinking in terms of code won’t ever get us to the true AI stage) any built-in “Friendliness” and might even come to resent those whom had so embedded it.
To me, the trick is not so much to concern ourselves with the *engineering* of a ‘friendly’ AI, but to ensure that any AI that develops is a ‘holistic’ sentience — i.e.; capable of emotions and empathy in a sane, stable manner from ‘conception’ — and then act to ensure it is empathetic to humanity.
All that really takes is being kind to the ‘god child’ that such an engine of awareness/manufacturing could be.
At least, that’s my personal take on it. The key, either way, is in empathy. Without having a very strongly developed sense of empathy, no superior sentience will ‘dote’ on its lessers.
I find myself more deeply concerned about the *freedom* of humanity to control its own destiny in a ‘post Singularity’ world. I’m sure this is something that’s been addressed ad nauseum, but I haven’t seen it.
Someone forgot to read Drexler’s “Engines of Creation!” This what active shielding is for.
Not so simple, Scarlette! You’d need superintelligent AI to operate a planetwide shield of the type suggested by Eric. Without one, there’s just too much complexity to handle – you can’t program in all the contingencies! Also, MNT will very likely be deployed before active shields are – active shields are a relatively advanced form of nanotech, whereas mass production of, say, missiles, is relatively basic.
2fI’ll thingk about it.3l I compleatly agree with last post. hwt
паркет и ламинат 3n