Roko on Homo Sapiens’ Unbroken Chain of Morals and Metamorals Thursday, Oct 29 2009
See Roko on the issue that confronts us here. Human morality and metamorality is a very unique and specific thing, contingent on our complex evolutionary history and cognitive evolution. Break it in the transition to real AI, and we have a major problem. I care about every futurist issue, but there’s a reason why I focus the most on AI and intelligence enhancement — they could lead to the extinction of the human race if not handled properly. If an AI or upload comes to power without values that explicitly include the survival and properity of all other six billion human beings, it’s Game Over. An AI or upload could copy itself millions of times, seize all computing power on the planet, and quickly establish covert manufacturing facilities to fabricate advanced robotics. It would be unkillable. It’s equally scary whether projected to arrive in 2050 or 2100. We have to deal with it, starting now.
I also have some interesting comments about this on facebook, including from David Pearce, which I’ve written a detailed response to.
It could only establish manufacturing facilities in active collaboration with humans, and if its goals were fundamentally discordant with human values it might have some difficulty recruiting henchmen.
For the foreseeable human infrastructure is going to be essential to the functioning of AI systems – as it is to computer systems generally.
It’s not like it would have any of the difficulty humans do in concealing its motives, assuming it’s smart enough to know it should do so (probably ‘enough’ here is still infrahuman) and its innards are non-transparent or not inspected.
See, for instance, the scenario in section 9 (pp. 24-25) of AI as a Positive and Negative Factor in Global Risk.
Philosophers sometimes worry about the possibility of zombies – not zombies in the Hollywood sense, but beings who are physically type-identical to you or me, and who behave in exactly the same way, and yet who are insentient. Now I don’t think zombies are really feasible. The interesting question is: why aren’t they feasible? At least within the context of materialist neuroscience, no one has any idea why you and me aren’t zombies. There is nothing we know in physics that seems to necessitate the existence of consciousness.
In the context of artificial intelligence, Roko and Elizeer [if I've understood them correctly], raise the possibility of insentient superintelligence. This is advanced intelligence that has superior analogues of human competence even in empathetic understanding of others, but which is non-sentient – a kind of ultra-intelligent “super-zombie”, as it were. However [again if I'm interpreting Roko and Eliezer correctly] a super-zombie might not possess the functional analogue of any value we would recognize. Rather, the super-zombie might maximally “value” something completely unintelligible or arbitrary by our lights – e.g. paperclips, whose abundance it would seek to maximise. Now I don’t know if this scenario is really feasible. But since there is currently no evidence that a digital computer with a classical von Neumann architecture is conscious, I can’t rule it out. What I hadn’t realized before reading Roko’s recent Facebook Wall post is that he (and Elizeer?) think this final outcome is most probable. Indeed this outcome is presumably imminent if they believe something along the lines of a Singularity is going to happen this century?
I normally assume that posthuman superintelligence will involve sentient beings merging with AI and thus our becoming supersentient as well as superintelligent. But on Roko’s (and Eliezer’s?) scenario, smart zombies will most likely inherit the Earth – and beyond?
[apologies to either party if I've misinterpreted what is being argued.]
@David: What you have said is pretty much correct.
Roko’s (and Eliezer’s?) scenario, smart zombies will most likely inherit the Earth – and beyond?
– from the human point of view, the idea of an optimizing intelligence without any form of subjective experience, empathy etc is a queer one.
This is because humans grew up around other humans, and thus in our environment, optimizing-ability, subjective experience, empathy and sympathy are well correlated.
In reality, there are reasons to believe that optimizing-ability, empathy and sympathy are essentially independent. Subjective experience is a bit of a joker in the pack; we don’t know what it implies or is implied by.
Some fundamental points on this one.
To claim that ‘Morals are relative’ is one thing.
But to conclude from that, that ‘therefore an AI has to be carefully programmed in order not to get rid of us’ is the confused philosopher’s equivalent of saying: ‘Morals are not relative since our existence is preferable over our non-existence’.
independent Dubai escort…
[...]Rational Morality » Less is More – or: the sorry state of AI friendliness discourse[...]…