Florian Widder, who often sends me interesting links, forwarded me to an interview that Russell Blackford recently conducted with Greg Egan. The excerpt he mentioned concerns the issue of smartness and whether qualitatively-smarter-than-human intelligence is possible:
... I think there's a limit to this process of Copernican dethronement: I believe that humans have already crossed a threshold that, in a certain sense, puts us on an equal footing with any other being who has mastered abstract reasoning. There's a notion in computing science of "Turing completeness", which says that once a computer can perform a set of quite basic operations, it can be programmed to do absolutely any calculation that any other computer can do. Other computers might be faster, or have more memory, or have multiple processors running at the same time, but my 1988 Amiga 500 really could be programmed to do anything my 2008 iMac can do â€” apart from responding to external events in real time â€” if only I had the patience to sit and swap floppy disks all day long. I suspect that something broadly similar applies to minds and the class of things they can understand: other beings might think faster than us, or have easy access to a greater store of facts, but underlying both mental processes will be the same basic set of general-purpose tools. So if we ever did encounter those billion-year-old aliens, I'm sure they'd have plenty to tell us that we didn't yet know â€” but given enough patience, and a very large notebook, I believe we'd still be able to come to grips with whatever they had to say.
I regard this as garden-variety anthropocentrism and basically the heliocentrism of cognitive science. It dovetails perfectly with theological notions of humanity. The simplest assumption is that humans are notthe center of the cognitive universe. The notion that we primitive humans are basically equal to all higher forms of intelligence, even if they are Jupiter Brains with quintillions of times greater computational capacity than us and can think individual thoughts with more Kolmogorov complexity than the entire human race, is pretty silly.
The transition from early hominids to humans produced a qualitative change in smartness -- why should we assume we're the end of the road? Just like there are optical illusions that our minds aren't sophisticated enough to see through (though I'm sure we can come up with dumb excuses), there are cognitive illusions that humans are programmed to be fooled by. There are so many of them that there is a huge field of study devoted to it -- heuristics and biases.
Without qualitative improvements to the structure of intelligence, we will just keep making the same mistakes, only faster. Experiments have shown that you cannot train humans to avoid certain measurable, predictable statistical errors in reasoning. They just keep making them again and again. In the best case, they can avoid them only when they are using a computer program set up to integrate the data without making the mistake. These basic findings prove that qualitative improvements in intelligence are possible, and that all minds are not created equal.
A person with an IQ of 100 cannot understand certain concepts that people with an IQ of 140 can understand, no matter how many time and notebooks they have. Intelligence means being able to get the right answer the first time, not after a million tries. Even if you could program a human being to ape the understanding of superintelligent thoughts, they wouldn't be able to come up with equivalent thoughts on their own or compare those thoughts to similarly complex thoughts.