Nick Bostrom on Superintelligence in Forbes Wednesday, Jun 24 2009
SIAI and singularity 11:36 am
Nick Bostrom, SIAI advisor, philosopher at the University of Oxford, and Director of the Future of Humanity Institute, has this to say in Forbes:
Whether abrupt and singular or more gradual and multipolar, the transition from human-level to superintelligence would be of pivotal significance. Superintelligence would be the last invention biological man would ever need to make, since, by definition, it would be much better at inventing than we are. All sorts of theoretically possible technologies could be developed quickly by superintelligence — advanced molecular manufacturing, medical nanotechnology, human-enhancement technologies, uploading, weapons of all kinds, lifelike virtual realities, self-replicating space-colonizing robotic probes and more. It would also be effective at creating plans and strategies, working out philosophical problems, persuading and manipulating.
It is an open question whether the consequences would be for the better or the worse. The potential upside is clearly enormous, but the downside includes existential risk. Humanity’s future might one day depend on the initial conditions we create–in particular, on whether we successfully design the system (e.g., the seed AI’s goal architecture) in such a way as to make it “human-friendly,” in the best possible interpretation of that term.
Consider donating to SIAI, the only group I am aware of that makes such a big deal about the human-friendliness issue and has made actual progress on the question.
Of the ‘big three’ super-geniuses (Bostrom, Hanson, Yudkowsky), Bostrom always struck me as the most rational.
Unfortunately, he’s merely repeating the standard Singularitarian ‘party line’ here. I am now very very confident that the reality is not at all what our three ‘super geniuses’ think it is.
If you’re smart yourself, naturally you’ll want others to worship smartness (very convenient for you high IQ folks to put yourselves at the top of the food chain). May be you even end up believing your own hype and point to intelligence as ‘the most powerful force in the universe’ aka Yudkowsky.
I offered a different view however and suggested that reflection is the most powerful force in the universe, not intelligence. You apparently did not grasp the point I was trying to make, which was that I thought that reflection is not the same thing as ‘mere intelligence’. I define full reflection as:
‘The capacity for full self-awareness combined with the capacity and motivation to safetly improve ones own cognitive structure’
The definition of intelligence may not neccesserily be general enough to capture the definition of reflection (and in fact I’m sure it isn’t).
Whilst our trio of super-geniuses not doubt fondly imagine that they are plumping the deepest secrets of thought (or least Yudkowsky does) , in fact they are stuck in the ‘paddle pool’ with their faces staring into puddles, whilst the deep end is a gazillon miles beyond their vision.
As to the idea that Bayesian probability shuffling is somehow the deepest secrets of thought, I find this utterly ludicrous Michael. It would be amazing if intelligence really was somehow reducible to an entirely non-sentient algorithm in the manner Yudkowsky claims, and I would be very very disappointed if that turned out to be the case, since thought would be reduced to mere ‘cranking of a mindless handle’, but I’m fairly sure that Godelian considerations forbid it.
Don’t get me wrong, I’m a not a Penrose fan, I’m sure intelligence is all algorithmic, it’s idea you can have intelligence without sentience I find utterly unbelievable. I think there’s always going to be a place for consciousness and creativity, and Bayes ain’t even close.
Ahh yes … recently several people laughed when this movement was depicted as “small and fringe” and would barely “fill a midwest auditorium” or “would never be taken serious”.
I then said – very soon this movement will hit the limelight. It is saturated. It cannot do much else than trickle through into the mainstream. Transhumanism will be on Oprah (or equivalent) within years. Ray and michael will be interviewed by CNN, SNL and BBC. If they haven’t already (Aubrey made it, yes?)
Even though, saying this “am I for real?” yes I am. The idea, no matter how undesirable it is, will become big and even though not popular for some, will be prevalent.
Very soon? it has started happening in mere months. Very soon housewives and the staff of office will casually talk about the merits of friendly AI.