Comments on: Eleven Ways to Avoid an Extremely Bad Singularity http://www.acceleratingfuture.com/michael/blog/2009/10/eleven-ways-to-avoid-an-extremely-bad-singularity/ Transhumanism, AI, nanotechnology, the Singularity, and extinction risk. Mon, 13 Feb 2012 02:08:02 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: DaveWBaldwin http://www.acceleratingfuture.com/michael/blog/2009/10/eleven-ways-to-avoid-an-extremely-bad-singularity/#comment-128469 DaveWBaldwin Tue, 13 Oct 2009 13:21:41 +0000 http://www.acceleratingfuture.com/michael/blog/?p=1363#comment-128469 The parameters you are discussing are interesting. Keep in mind though, the real danger is NOT the AGI, but the human. Not long ago, I relayed to Ben it comes down to the three 'B's'... booze, booty and bounty. Without going into the humorous anecdotes, any dangers will be more attached to that because of the human, not the machine. Starting with the computer, if an AGI level unit were sitting on the desk, it will not drink and/or feel the need associated with procreation. To build a bot that can drink is, well a waste of time, for the only liquid it may need is in the form of lubrication if that is the case. Both the computer and bot will not need the money, for as long as they have the power source, it can function. It is the human who has the drive that centers on the three B's and how that drive often moves toward bad outcomes. As long as resources are few among a nation, the leader will take from everyone and have his drinking and sex parties. In the case of the nation with many resources, the same thing happens, for due to the weakness and insecurities of man, we often try to have others think we have lots of bounty that pays for the booze to consume while the booty dances. The vanity associated with that leads to a lot of bounty that could be put to a useful purpose, wasted at best. My suggestion is placing debate regarding the need of a Guardian AGI more toward protecting all of us from the AGI used by the person who has bad motive, rather than the AGI coming up with having fun via pillage and plunder on it's own. The parameters you are discussing are interesting. Keep in mind though, the real danger is NOT the AGI, but the human.

Not long ago, I relayed to Ben it comes down to the three ‘B’s'… booze, booty and bounty. Without going into the humorous anecdotes, any dangers will be more attached to that because of the human, not the machine.

Starting with the computer, if an AGI level unit were sitting on the desk, it will not drink and/or feel the need associated with procreation. To build a bot that can drink is, well a waste of time, for the only liquid it may need is in the form of lubrication if that is the case.

Both the computer and bot will not need the money, for as long as they have the power source, it can function.

It is the human who has the drive that centers on the three B’s and how that drive often moves toward bad outcomes. As long as resources are few among a nation, the leader will take from everyone and have his drinking and sex parties.

In the case of the nation with many resources, the same thing happens, for due to the weakness and insecurities of man, we often try to have others think we have lots of bounty that pays for the booze to consume while the booty dances.

The vanity associated with that leads to a lot of bounty that could be put to a useful purpose, wasted at best.

My suggestion is placing debate regarding the need of a Guardian AGI more toward protecting all of us from the AGI used by the person who has bad motive, rather than the AGI coming up with having fun via pillage and plunder on it’s own.

]]> By: Michael Anissimov http://www.acceleratingfuture.com/michael/blog/2009/10/eleven-ways-to-avoid-an-extremely-bad-singularity/#comment-128464 Michael Anissimov Sat, 10 Oct 2009 17:44:50 +0000 http://www.acceleratingfuture.com/michael/blog/?p=1363#comment-128464 Hi Ben, there is no connection between my emotional appraisal of MNT and the thoughts that I use to estimate its plausibility. I weave them together in the text partially just to make it sound more interesting. MNT has the potential to enable brute-force AGI where we are able to implement intelligence without thoroughly understanding its goal system, leading to global destruction. Hi Ben, there is no connection between my emotional appraisal of MNT and the thoughts that I use to estimate its plausibility. I weave them together in the text partially just to make it sound more interesting. MNT has the potential to enable brute-force AGI where we are able to implement intelligence without thoroughly understanding its goal system, leading to global destruction.

]]> By: Benjamin Abbott http://www.acceleratingfuture.com/michael/blog/2009/10/eleven-ways-to-avoid-an-extremely-bad-singularity/#comment-128463 Benjamin Abbott Sat, 10 Oct 2009 17:12:42 +0000 http://www.acceleratingfuture.com/michael/blog/?p=1363#comment-128463 I'd be careful about dismissing molecular manufacturing, particularly if the implications make you uncomfortable. That's the same kind of reasoning (if we can call it that) used against artificial intelligence and life extension. Indeed, I see no reason to consider those two things more likely than nanofactories. Plenty of futurists seen MNT coming as quickly as AGI, if not sooner. See Drexler, Hall, Freitas, the CRN folks, Kurzweil, and so on. Each technology has the potential to enable the other. I’d be careful about dismissing molecular manufacturing, particularly if the implications make you uncomfortable. That’s the same kind of reasoning (if we can call it that) used against artificial intelligence and life extension. Indeed, I see no reason to consider those two things more likely than nanofactories. Plenty of futurists seen MNT coming as quickly as AGI, if not sooner. See Drexler, Hall, Freitas, the CRN folks, Kurzweil, and so on. Each technology has the potential to enable the other.

]]> By: Michael Anissimov http://www.acceleratingfuture.com/michael/blog/2009/10/eleven-ways-to-avoid-an-extremely-bad-singularity/#comment-128461 Michael Anissimov Sat, 10 Oct 2009 00:00:15 +0000 http://www.acceleratingfuture.com/michael/blog/?p=1363#comment-128461 Khannea, I find it trite and mocking. It's hard to evaluate over the Internet where exactly your social intuitions are. Your comments are always so long and often rambling -- can you try to be more concise? You also seem to enjoy theatrics in your comments -- why not just drill directly to the core of the issue and make the simplest statements possible? It's important to me that comments be as meaningful and thought-provoking as possible. Ben, thanks. I think the differences we are talking about are subtle enough that there has to be a programmatic distinction. Presumably one could derive a basic-order description of the person I wish I were through a questionnaire asking me about that very issue. It could thereby be precisely defined, though not at a great level of resolution. The crucial question is whether the resolution could be improved enough to lead to an AI that creates a model of my volition that comes up with interesting and surprising new conclusions that make sense. I don't think it's too fanciful to discuss extrapolations -- in my opinion, <i>anything</i> that works <i>must</i> be an extrapolation of some sort. The alternative, copying an existing human morality, seems like a bad idea. The question is not to use an extrapolation or not, but which extrapolation we should use. -MA Khannea, I find it trite and mocking. It’s hard to evaluate over the Internet where exactly your social intuitions are. Your comments are always so long and often rambling — can you try to be more concise? You also seem to enjoy theatrics in your comments — why not just drill directly to the core of the issue and make the simplest statements possible? It’s important to me that comments be as meaningful and thought-provoking as possible.

Ben, thanks. I think the differences we are talking about are subtle enough that there has to be a programmatic distinction. Presumably one could derive a basic-order description of the person I wish I were through a questionnaire asking me about that very issue. It could thereby be precisely defined, though not at a great level of resolution. The crucial question is whether the resolution could be improved enough to lead to an AI that creates a model of my volition that comes up with interesting and surprising new conclusions that make sense.

I don’t think it’s too fanciful to discuss extrapolations — in my opinion, anything that works must be an extrapolation of some sort. The alternative, copying an existing human morality, seems like a bad idea. The question is not to use an extrapolation or not, but which extrapolation we should use.

-MA

]]> By: bgoertzel http://www.acceleratingfuture.com/michael/blog/2009/10/eleven-ways-to-avoid-an-extremely-bad-singularity/#comment-128457 bgoertzel Fri, 09 Oct 2009 23:31:47 +0000 http://www.acceleratingfuture.com/michael/blog/?p=1363#comment-128457 Mike ... Yes I and most other AGI-ers think it's likely feasible to program an AGI goal framework whose structure and dynamics implicitly constitutes a (high-dimensional) space of "parameters for Friendliness acquisition." So I accept your rephrasing on that point, even though it's not exactly the way I would say it... About CEV: some obvious problems are... 1) it seems like it will require much more advanced technology than creating an AGI with, say, 2x human intelligence. So even if it were possible and sensible, it seems unlikely to happen at the stage where humans are still in control. Maybe the Guardian AGI will be the guy to perform the CEV experiment. 2) it's not clear that "the person I wish I were" is a well-defined concept. At the post-Singularity workshop, an actual professional moral philosopher was present, and he opined that it is NOT a well-defined concept, thus rendering CEV non-workable and ill-defined. It may be that minor variations in the way one defines this ill-defined concept, cause large variations in the "extrapolated volition" obtained. In fact I suspect this is the case for many people though not all. 3) different people have wildly incoherent ideas about what's good. I doubt the extrapolated Ben has compatible ideas about the world as the extrapolation of the average Islamic mullah, for example ... or even the average Walmart clerk. I'm a weird dude. IEV solves complaint 3, and partially solves complaint 1, but doesn't touch complaint 2. I'm sure these are not the only objections to the fanciful but fascinating CEV idea, but I don't have more time to search my memory or re-think the issues at the moment... -- Ben Goertzel Mike …

Yes I and most other AGI-ers think it’s likely feasible to program an AGI goal framework whose structure and dynamics implicitly constitutes a (high-dimensional) space of “parameters for Friendliness acquisition.” So I accept your rephrasing on that point, even though it’s not exactly the way I would say it…

About CEV: some obvious problems are…

1) it seems like it will require much more advanced technology than creating an AGI with, say, 2x human intelligence. So even if it were possible and sensible, it seems unlikely to happen at the stage where humans are still in control. Maybe the Guardian AGI will be the guy to perform the CEV experiment.

2) it’s not clear that “the person I wish I were” is a well-defined concept. At the post-Singularity workshop, an actual professional moral philosopher was present, and he opined that it is NOT a well-defined concept, thus rendering CEV non-workable and ill-defined. It may be that minor variations in the way one defines this ill-defined concept, cause large variations in the “extrapolated volition” obtained. In fact I suspect this is the case for many people though not all.

3) different people have wildly incoherent ideas about what’s good. I doubt the extrapolated Ben has compatible ideas about the world as the extrapolation of the average Islamic mullah, for example … or even the average Walmart clerk. I’m a weird dude.

IEV solves complaint 3, and partially solves complaint 1, but doesn’t touch complaint 2.

I’m sure these are not the only objections to the fanciful but fascinating CEV idea, but I don’t have more time to search my memory or re-think the issues at the moment…

– Ben Goertzel

]]>