HyperExistential Risk
From SL4 Wiki
we're waiting to find out what it is! a risk to the universe beyond Earth's biosphere?
also see: Existential Disaster.
I've used the word to mean "Fate Worse Than Death" before, and I think it's an important thing to think about. An example of a HyperExistential disaster would be an upload transcending to superintelligence, going insane (or staying insane), and torturing everyone in personal pocket universes.
If the upload succeeded in doing this for an infinite amount of time, this would be an infinitely HyperExistential disaster--a fate infinitely worse than death. It makes some intuitive sense to me to have preventing this at all cost as a goal. Nick Bostrom introduces the Maxipok principle in his Existential Risks paper (for "maximize the probability of an OK outcome"); I think a good analogue for HyperExistential risks might be the AINU principle, for "avoid infinite negative utility".
I used to think that a hypothetical SuperIntelligence that followed this principle would probably destroy the Universe ("we had to destroy the Universe in order to save it") (this is very compatible with the Doomsday Argument), but then there's always the possibility of discovering larger bits of Universe that might cause a HyperExistential screwup.
I think all this is yet another argument for Friendly AI and a quick Singularity--I'd estimate upgraded humans and indifferent AIs to be much more HyperExistentially risky. Upgraded humans would be more likely to cause anthropomorphic hell scenarios, and indifferent AIs could create enormous suffering by simulating all possible universes, for example. Also, Friendly AI might prevent HyperExistential disasters from occurring in the rest of the Universe.
I don't know whether any of this applies to how the word is used in the interview with Eliezer, though. -- Nus
Yes, this is exactly what the word meant when it was used in the interviews. -- the person who used the word in the interviews
Here's a scary question: Is there any chance that an ASI may be developed as a weapon? This would be especially bad if the group that developed this ASI was acting out of hatred instead of greed. Another scary possibility is an ASI that was artificially evolved, and that happened to over-generalize the evolutionary zero-sum-game concept of "What's bad for you is good for me".
In reality, there are very few plausible scenarios that would result in INU without also resulting in a comparable amount of IPU somehow, and the few scenarios that are plausible are still extremely unlikely to actually happen. Also, there seem to be more scenarios that would result in IPU without resulting in INU. This may be an illusion based on wishful thinking, but scenarios involving IPU would usually be worked towards, and scenarios involving INU would usually be deliberately avoided.
Also, discussing possible HyperExistential Risks doesn't seem likely to make these risks less likely to happen, and these discussions tend to generate a lot of unnecessary fear, which most people consider as negative utility, so maybe I should stop discussing them now.
(See Controversial Pages/Net Utility Of The Multiverse for an attempt to prove that universes that have suffered a "HyperExistentially bad" fate are extremely rare, and the Net Utility Of The Multiverse is probably massively positive.)
--observer
