Comments on: The Challenge of Self-Replication http://www.acceleratingfuture.com/michael/blog/2008/11/the-challenge-of-self-replication/ Transhumanism, AI, nanotechnology, the Singularity, and extinction risk. Mon, 13 Feb 2012 02:08:02 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: pest control corona http://www.acceleratingfuture.com/michael/blog/2008/11/the-challenge-of-self-replication/#comment-134500 pest control corona Wed, 04 Aug 2010 08:40:33 +0000 http://www.acceleratingfuture.com/michael/blog/?p=886#comment-134500 1 of the toughest issues that administration organizations deal with is pest contaminations. Most buildings possess at minimum one unit who refuses to allow insect control experts inside or denies to maintain the unit in great sanitary situation. This a single unit can preserve pests contaminations returning back again time and again, costing condo management time, labor and money. 1 of the toughest issues that administration organizations deal with is pest contaminations. Most buildings possess at minimum one unit who refuses to allow insect control experts inside or denies to maintain the unit in great sanitary situation. This a single unit can preserve pests contaminations returning back again time and again, costing condo management time, labor and money.

]]> By: Michael Anissimov http://www.acceleratingfuture.com/michael/blog/2008/11/the-challenge-of-self-replication/#comment-123759 Michael Anissimov Wed, 14 Jan 2009 03:18:14 +0000 http://www.acceleratingfuture.com/michael/blog/?p=886#comment-123759 JJ, a "decentered" (you mean decentralized) superintelligence emerging on the Internet would likely acquire enough power to become a singleton. Thus, it is a singleton that leads to "the most desirable outcome" "from a stability point of view". Of course, it seems like both JJ and RQ didn't read the definition of singleton: http://www.nickbostrom.com/fut/singleton.html JJ, a “decentered” (you mean decentralized) superintelligence emerging on the Internet would likely acquire enough power to become a singleton. Thus, it is a singleton that leads to “the most desirable outcome” “from a stability point of view”.

Of course, it seems like both JJ and RQ didn’t read the definition of singleton:

http://www.nickbostrom.com/fut/singleton.html

]]> By: JJ http://www.acceleratingfuture.com/michael/blog/2008/11/the-challenge-of-self-replication/#comment-123750 JJ Mon, 12 Jan 2009 12:43:44 +0000 http://www.acceleratingfuture.com/michael/blog/?p=886#comment-123750 I tend to agree with RQ. A benevolent singleton seems to me as much of a utopian thought as the "everything will magically be allright" stance that Kurzweil is taking. Reality to me seems to be pointing at the web inevitably growing more and more capable applications that will be increasingly interconnected in combination with ubiquitous computing this should by definition lead to a decentered form of super intelligence. As from a stability point of view this is also the most desirable outcome. Furthermore it would solve the "who controls it?" issue if we can integrate it with the internet. In any scenario the opportunity for misuse is immense. Therefore I agree that society in its current form is unfit to harbour any superintelligence whatsoever. One can only hope that the initial forms of human level AI will transform humanity sufficiently to avoid disaster when ultraintelligence arrives. I tend to agree with RQ. A benevolent singleton seems to me as much of a utopian thought as the “everything will magically be allright” stance that Kurzweil is taking. Reality to me seems to be pointing at the web inevitably growing more and more capable applications that will be increasingly interconnected in combination with ubiquitous computing this should by definition lead to a decentered form of super intelligence. As from a stability point of view this is also the most desirable outcome. Furthermore it would solve the “who controls it?” issue if we can integrate it with the internet. In any scenario the opportunity for misuse is immense. Therefore I agree that society in its current form is unfit to harbour any superintelligence whatsoever. One can only hope that the initial forms of human level AI will transform humanity sufficiently to avoid disaster when ultraintelligence arrives.

]]> By: RQ http://www.acceleratingfuture.com/michael/blog/2008/11/the-challenge-of-self-replication/#comment-123528 RQ Thu, 11 Dec 2008 16:28:24 +0000 http://www.acceleratingfuture.com/michael/blog/?p=886#comment-123528 "we require a benevolent singleton. A top-level decision making entity that can act with extreme speed and reliability to put down threats before they emerge." There is no so such thing. I'm an incident responder (computer and network security), I'm far past trusting any computer. The average root kit changes a computer so it can't "self analyze". It can't look at its own files, or notice it's not looking at them. The hackers are the new owner of the computer and it does what they want. Anyway, I'm not going to go into the whole list of how a modern computer is subverted, either you know it or you don't. I'm not moving my mind onto something like that without FUNDAMENTAL changes. Far beyond. And then to say some "benevolent singleton" will have total and ultimate control over me? A singleton is the most fragile and least fault tolerant solution. One mistake and it's all over. Responses that boil down to "it won't won't make mistakes" or "my singleton will be the BEST hacker ever" aren't responses at all. “we require a benevolent singleton. A top-level
decision making entity that can act with extreme
speed and reliability to put down threats before
they emerge.”

There is no so such thing. I’m an incident responder (computer and network security), I’m far past trusting any computer. The average root kit changes a computer so it can’t “self analyze”. It can’t look at its own files, or notice it’s not looking at them. The hackers are the new owner of the computer and it does what they want. Anyway, I’m not going to go into the whole list of how a modern computer is subverted, either you know it or you don’t. I’m not moving my mind onto something like that without FUNDAMENTAL changes. Far beyond.

And then to say some “benevolent singleton” will have total and ultimate control over me? A singleton is the most fragile and least fault tolerant solution. One mistake and it’s all over.

Responses that boil down to “it won’t won’t make mistakes” or “my singleton will be the BEST hacker ever” aren’t responses at all.

]]> By: Roko http://www.acceleratingfuture.com/michael/blog/2008/11/the-challenge-of-self-replication/#comment-123408 Roko Fri, 05 Dec 2008 15:22:35 +0000 http://www.acceleratingfuture.com/michael/blog/?p=886#comment-123408 "the clear course of action seems to be to contribute to the construction of a benevolent singleton." - yes, this does seem to be the emerging consensus. i am inclined to agree with it, too. You would be wise to consider what kinds of singleton are possible, and what the relative merits and dangers of each choice are. For example, fAI, global government, etc. What do we do if self-improving intelligence emerges in a distributed form as Hanson seems to imply on OB? What do we do if the military gets hold of self-improving AI technology before academia or corporations, etc. “the clear course of action seems to be to contribute to the construction of a benevolent singleton.”

– yes, this does seem to be the emerging consensus. i am inclined to agree with it, too.

You would be wise to consider what kinds of singleton are possible, and what the relative merits and dangers of each choice are.

For example, fAI, global government, etc.

What do we do if self-improving intelligence emerges in a distributed form as Hanson seems to imply on OB?

What do we do if the military gets hold of self-improving AI technology before academia or corporations, etc.

]]>