Surprisingly Good Solutions
“The Singularity holds out the possibility of winning the Grand Prize, the true Utopia, the best-of-all-possible-worlds - not just freedom from pain and stress or a sterile round of endless physical pleasures, but the prospect of endless growth for every human being - growth in mind, in intelligence, in strength of personality; life without bound, without end; experiencing everything we’ve dreamed of experiencing, becoming everything we’ve ever dreamed of being; not for a billion years, or ten-to-the-billionth years, but forever… or perhaps embarking together on some still greater adventure of which we cannot even conceive.” - Eliezer Yudkowsky
This paragraph, as nice as it is, doesn’t really convey the principle inherent in Expected Creative Surprises. Dreams can be pleasant, but they can’t be surprising in some deep sense; if you already have a specific vision of Utopia, you’ve already mapped out your vision’s location in utilityspace. A superintelligently-designed Utopia should contain a large number of qualia which are surprisingly good, in the sense that we’ve never even imagined that point in utilityspace. Such qualia are relatively rare in the present world, but they do exist; imagine a starving African child eating their first chocolate cake. This, in some sense, is a more worthy goal for the future than, say, Iain Banks’s Culture series. The Culture is limited by human imagination, while superintelligently-designed utopias shouldn’t be.
roko said,
February 9, 2009 @ 7:39 pm
A superintelligently-designed Utopia should contain a large number of qualia which are surprisingly good, in the sense that we’ve never even imagined that point in utilityspace.
I’m not convinced wither way here. This topic is of great interest to me, though, as I am keen to understand whether the transhumanist movement should be selling itself on the basis of these “surprisingly good solutions”. I argued this against an OB commenter:
“As I understood, one of the points you made was about how preferences of both individual people and humanity as a whole are quite coarse-grained, and so strong optimization of environment is pointless. Beyond certain precision, the choices become arbitrary, and so continuing systematic optimization, forcing choices to be non-arbitrary from the updated perspective, basically consists in incorporating noise into preferences.”
roko said,
February 9, 2009 @ 7:41 pm
Original post on OB:
It’s also interesting to wonder whether the goals of “radical” transhumanists might be a little self-contradictory. With a limited human brain, you can (as a matter of physical fact) only entertain thoughts that constrain the future to a limited degree. Even with all technological obstacles out of the way, our imaginations might place a hard limit on how good a future we can try to build for ourselves. Anyone who tries to exceed this limit will end up (somehow) absorbing noise from their environment and incorporating it into their preferences. Not that I have anything against this - it is how we got our preferences in the first place - though it is not a strong motivator for me to fantasize about spending eternity fulfilling preferences that I don’t have yet and which I will generate at random at some point in the future when I realize that my extant preferences have “run out of juice”.
- link as above