Accelerating Future Transhumanism, AI, nanotech, the Singularity, and extinction risk.


The Patternmakers

Cross-posted from SIAI blog.

Consider two classes of AIs. One class of AIs manipulates external objects to direct the world towards a goal state, the other doesn't. AIs with the greatest real-world impact fall into the first category. The objects may be virtual as well as physical, although they're both ultimately the same thing, as reality is harmonious and unified.

Within the first category, there are AIs with motivations that output the open-ended, indefinite manipulation of external objects, and AIs with motivations that cause the manipulations to stop after a critical threshold of utility maximizing (or satisficing) is performed.

A CEV-AI is an example of the latter category. It extrapolates humanity's volition, creates an optimizing process that embodies it, then shuts itself down. There's a technical problem here--how to program it in such a way that it doesn't attempt to turn the planet into a supercomputer to compute humanity's volition, disintegrating humanity in the process? Some version of interim Friendliness no doubt, but remember, the CEV-AI's primary job is to output humanity's collective will, not be nice to humanity on a day-to-day basis. I'll let the Friendly AI theorists try to figure that one out.

But back to the categories, I would think that most possible AIs fall into the first category: the open-ended, indefinite manipulation of external objects. In fact, most intelligences probably do. If a human life were extended to a quadrillion years, those quadrillion years would likely consist of the manipulation of external objects. Same thing if you extended the life of a chimp, or a badger indefinitely. The results might get boring pretty fast (rest, eat, sex, rest, eat, sex), but that manipulation of external objects would keep on going.

Imagine a sim-world, maybe something like that game Spore, that Will Wright thinks will change the world, with an indefinitely-living couple, be it chimp or human, living in it. Eventually their semi-random walk and offspring would encompass the world, and in the case of the human, they might even learn how to convert the world into billions of O'Neill colonies for maximum usefulness. When minds have an open-ended desire to manipulate the external world, in the long run, things never stay the same.

Because AIs would be running on accelerated substrates, the "long run" for them could be a few minutes or hours. An AI with an open-ended desire to manipulate external objects will eventually pattern over anything not to its liking, like a gardener will eventually pluck all the weeds in a garden if he has the time to do so. That's why it's damn important to make sure the first AI considers us, with all our flaws and imperfections, to be in alignment with its goals: if not, we're toast in the long run, and for the AI, the long run ain't very long at all.

Filed under: AI Leave a comment
Comments (8) Trackbacks (1)
  1. Although I tend to agree with the gist of this article, I have to take exception with the second sentence: “Almost all AIs of any use fall into the first category.”

    The second category includes all cases where the AI comes up with a useful plan or design for something, which humans then use as they will. This could include, among other things, designs for radically new and better AI systems. I fail to see how this isn’t useful. In fact, almost ALL computer use today would fall into the second category. Again I fail to see how this isn’t useful.

  2. Okay, I’ve slightly changed the wording.

  3. Adequately defining “do nothing” seems to me like it may be a FAI-complete problem. :-/

  4. Oh man, do you really think so? I wouldn’t be that pessimistic. I mean, can’t you just delineate an infinite utility drop for manipulations that violate certain circumscribed boundaries?

    There must be some AGI applications which aren’t FAI complete, for instance a CEV isn’t FAI complete, right?

  5. I wouldn’t actually try this, but the easiest way to create an information-outputting, safe AI would seem to be to give it absolutely no information input beyond the problem definition – making it impossible for it to deduce anything interesting about the external world, let alone how to manipulate people into letting it out.

  6. Nick-

    …and it would end up being perfectly fucking useless. :)

  7. Chris, not at all. The problem could be proving P=NP, solving the Riemann hypothesis, or calculating the simultaneous trajectories of 5,000 ballistic missiles.

    It’s more likely a priori that Nick would have a point you don’t understand than that he said something for nothing.

  8. If we opt for the safety of residing inside a future supercomputer simulation then the “external” world we would be manipulating need only be confined to the modification of atomic or even sub-atomic particles corresponding to bits. Other incentives that may lead us to adopt this strategy (possibly within the next few centuries) include; the ability to experience a wide-range of enhanced senses, intelligence etc., the ability to create backup copies of your “self” (assuming, of course, that the self can be represented with matter) in multiple locations spatially separated in supposed ‘Ultimate’ reality, and digitally transmittable at the speed of light. Uploading our consciousnesses into a computer probably seems ridiculous to you now, because you have a radically antiquated perception of what a computer is/is capable of now, compared to what “computer” comes to define when this possibility is realized. The line between real and virtual is getting thinner.

Leave a comment