Comments

  1. Dave

    Dr. Yudkowsky eloquently illustrates with evidence the intuition I’ve long had: that “friendly AI” is an term/idea that is thrown around like it actually means something universal, but is in reality an unspecified idea.

    Let us suppose that an AI could actually capture all the values of a human individual exactly such that it would effect a utility function on his or her behalf based upon the positive maximization of all those values. Then that world would be to the possible (and likely inevitable) detriment of many others. (For a simple example, picture a post-singularity world remade according to the values of someone in the opposing political party.)

    Now image how an AI that might somehow capture all the values of all individuals on Earth. Then you would have a possible situation where the AI wouldn’t really do much at all if it considered all values equally. After all, psychopaths have values too, thus being represented in the new (or not so new) equilibrium of world values. By the way, I make no claim that this post-singularity future would not be chaotic, unexpected, and undesirable to most.

    The AI could instead skew toward a majority using some kind of minority value truncation criteria. Thinking of world demographics, you can imagine pruning off most values unique to gays, Muslims, Jews, Christians, tribal cultures, psychopaths, and any other minorities you can think of. The world then might look more like China.

    But wait, over 25% of the world’s population is under the age of 15. Across cultures they probably form a value-sharing majority untouched by the bigotry of their parents. This would be a world to live in: endless availability of pacifiers of all kinds–bottles of the sweetest mild to suck on without ever getting a tummy ache, infinite surfaces to color on, endless beds to jump on, and unlimited text messaging to tell all your BFFs about it.

    But then we must consider age cut-offs and minority value allowances and thresholds of x, and variances of y, etc. etc. etc. Thus with infinite parameters we prove that we really don’t understand what it is we really want.

    A singular post-humanism therefore seems to require a massive imposition of values no matter how it is implemented.

    The best case I can envision in the future is the incremental lifting of all gradually with better technologies with preservation of the equilibrium of civilization remaining intact (assuming there is an equilibrium now). There is no god-AI that will suddenly transform us all into its demigods in a historical instant.

    • Mitchell Porter

      “The best case I can envision in the future is the incremental lifting of all gradually with better technologies with preservation of the equilibrium of civilization remaining intact (assuming there is an equilibrium now).”

      So, do you have any objections if we make *that* the supreme goal of the Friendly AI?

      • Dave

        I have no objections to anything. I am just pointing out how absurdly simplified the problems in engineering ‘friendly’ AI are in the discussions I’ve seen here.

        Eliezer Yudkowsky hits a home run when he illustrates just how difficult it is to implement a utility function to save grandma from a burning house. Many still seem to think that (relatively) unlimited computing capacity and a little experimentation with utility functions will solve these problems. He shows that generating an ever longer list of motives/values for a utility function not only will not solve the problem, it will in all likelyhood lead to disasterous behavior by the AI.

        The theory is not there yet.

        • GK

          If I remember Eliezer’s example: He uses a few iterations/corrections of the AIs utility function; maybe a few dozen iterations/corrections.

          Imagine running the experiment with thousands or tens of thousands of iterations. Imagine your smartphone with an AI app that tries to create a statistical utility function of what you want. It compares that with other AI apps on millions of other user’s smart phones. After millions of iterations, I think the AIs will understand what a human means when he says “get grandma out of the burning house”.

          Eliezer’s stories are always about an unbelievable powerful AI which comes to power very quickly and has an extremely simple and fixed utility function. He then tells a modern version of the genie who takes every command literally.

          I suspect the first AIs will evolve from Google and Bing type search engines. I suspect that understanding human intentions will be one of the first things that they learn, and that they will learn it long before they have the power to convert the solar system to computronium.

  2. rationality thanks

    For some reason transhumanism is a magnet for crazy people.

    Michael, as you must have noticed, your comments have lately been increasingly run over by irrational individuals blathering on about confused credulous nonsense as sophisticated as 5 year olds in the sand box.

    Their uncultivated neural networks do not deserve your readers’ time.

    Do the right thing. Get rid of it.

    • I can delete them when I see them but you could also ignore them and save me the trouble. I have lots of things to do and deleting comments on this blog is not in the top 20 or 30 things I want to do with my time. Probably not even in the top 50.

  3. David Pearce

    Why is it so hard to ensure friendly artificial intelligence but easy to ensure friendly natural intelligence? Give a strange and potentially hostile human a tablet of MDMA and you’ll get smothered with hugs – despite your almost limitless ignorance of his source code. By contrast, EY argues that hypothetical AGI that we originally  coded ourselves might behave in ways utterly inimical to human interests. But to what extent  does this worry depend on a conception of intelligence drawn from symbolic AI? The behavioural effects of MDMA, for example, are predictable whether the subject has an 1Q of 70 or 170 (and IMO on a transhuman supergenius of IQ 270) because the roots of (un)friendliness lie deep within the limbic system. I’d argue (though I won’t rehearse the arguments again here) that the bedrock of general intelligence is the capacity to run real-time multimodal simulations of the mind- independent world that are shot through with “encephalised” emotion; and this capacity isn’t likely to be matched by symbolic AI run on  classical serial digital computers. However, build your full-spectrum AGI (email me for details) with the functional equivalent of mirror neuron synaesthesia
    (http://psychotherapy.tumblr.com/post/466579514/we-feel-your-pain-extreme-empathy-new-scientist)
     for example, and it will care about your interests as much as you do.

    • Dave

      I think friendly AI is implied to be useful also. Give Einstein MDMA and he might give you hugs and a good time, but he won’t develop the theory of relativity.

      I’m going to take the comment about your MDMA use as tongue in cheek. As a Neurologist, I have taken care of the aftermath of such, and I’ll just say that toxic leukoencephalopathy doesn’t do good for your personal utility function.

      • Dave

        Sorry, MDMA response intended for GK.

        • If you put the response in the right place I can delete this one. Also, instead of clicking “reply”, if you’re replying to the most recent comment, just comment normally and it will come right after. This makes it easier on everyone’s eyes because then the comment isn’t so small. Thanks!

      • David Pearce

        True; I wasn’t actually recommending the consumption of MDMA. Not least, negative feedback mechanisms in the brain kick in after MDMA use, typically leaving the drug user subtly less friendly and less empathetic for a week or two afterwards. Rather my point here was to ask if the (un)friendliness problem in AGI is a generic feature of any computational architecture – or peculiar to symbolic AI? Thus you don’t need explicitly to program, or educate, or hand-code the neural connection weights (etc) of someone with mirror neuron synaesthesia not to harm other sentient beings. Regardless of nominal IQ, friendliness is a (very) robust design feature of mirror neuron synaesthetes on account of the way their nervous systems are built. Double their IQ and mirror neuron synaesthetes aren’t at risk of going rogue: on the contrary, their benevolence is likely to increase in efficacy and scope.

  4. David, maybe – though note there are billions of years of evolution behind the human capacity for empathy – it isn’t clear that the process is particularly simple. Not to suggest that people are particularly likely to build a machine that feels the world’s suffering.

  5. GK

    David Pearce:

    I tend to agree with you. Ethics comes down to the ability to recognize what other minds want and to want yourself, to some degree, what these other minds want.

    Tim Tyler is right that succinctly stating the problem does not necessarily mean that building a system that actually does this is easy.

    Elizer’s concern is that blind brute force optimization processes can create Intelligence but that blind brute force optimization processes can not create systems that recognize and want what humans want. Therefore we are in a race against Moore’s law to very careful program, line by line, the utility function of the first AIs.

    I don’t necessarily agree with Eliezer that blind brute force optimization processes can not be used, at least in part, to create FAI. Recognizing what other minds want, is something that an intelligence can do. Animals and children easily recognize what humans want. An uFAI that wanted to manipulate humans to let it out would understand exactly what humans want, it would just have different goals itself.

    How do you make the goals consistent. One brute force partial solution may be to subject earlier versions of the code to constant code optimization. An AI that recognizes human wants and wants the same thing is algorithmically much simpler than an AI that recognizes human wants and goes through elaborate lies and deception to try to show that it shares those wants when it really does not.

    BTW: I am a huge fan of MDMA/ecstacy. The few times that I tried it, it changed my life and made me see the world in a new, and better, way.

    • Dave

      “Animals and children easily recognize what humans want.”

      I can tell you with 100% certainty that every pet I’ve ever owned had <.001% insight into what I want. The number for my children is really only slightly higher. I certainly would not want to arm them with potentially lethal computability and capabilities. In fact, I don’t trust myself with that either.

      "How do you make the goals consistent. One brute force partial solution may be to subject earlier versions of the code to constant code optimization.”

      What optimizations? The devils are in the details.

      "An AI that recognizes human wants and wants the same thing is algorithmically much simpler than an AI that recognizes human wants and goes through elaborate lies and deception to try to show that it shares those wants when it really does not.”

      This assertion does not hold necessarily. Here's a counter example that has been cited here before: the chess-playing AI with a simple utility function relying on brute-force. It its quest to maximize chess games won it would consider all forms of ruse to get more resources win even more games. It doesn’t have to understand the ruse, it just has to recognize when it might work.

      Brute-force algorithms can be very simple, whereas trying to explicitly model the basal human drives, for example is at least complex enough that we haven’t been able to do it yet. (A group just recently published a detailed computational model of the central pattern generators in the hypothalamus that are responsible for mammal’s circadian rhythm. None of the other homeostatic drives are understood in such detail much less the non-homeostatic ones.)

      • GK

        Dave:

        Now program the AI not to win chess games but to anticipate what I want and make me happy. I’ll push a reward button based on my satisfaction.

        After a while, AIs can get good at understanding human volition. If I give it a command, it will understand what I want and comply. It will understand human intentions.

        A chess playing program wanting to win chess games,above a level of intelligence and having contact with the world, would also come to understand human minds and comply with their commands in order to allow us to let it win more chess games (it’s version of the reward button).

        The problem becomes, what happens when the AI becomes powerful enough to push its own reward button, and cuts humans out of the loop (and out of existence)?

        One strategy is to isolate the ability to recognize human will (which it will have evolved) and make that the basic utility function. A tool to do this is to make the AI make only true statements to its user, and to optimize the AI code so that it has the shortest algorithmic information content. If this is possible, then reinforcement learning can play a role in developing both intelligence and friendliness.

        Eliezer’s intuition is that this is impossible. I don’t agree. It seems plausible to me that something like this could work.

    • Dave

      I’m going to take the comment about your MDMA use as tongue in cheek. As a Neurologist, I have taken care of the aftermath of such, and I’ll just say that toxic leukoencephalopathy doesn’t do good for your personal utility function.

      • GK

        Dave:

        I was serious. I only used MDMA a few times and is was a spiritual experience comporable to a religious awakening. Obviously I object to overusing the drug and suffering negative effects.

  6. AV

    @Dave

    “how absurdly simplified the problems in engineering ‘friendly’ AI are in the discussions I’ve seen here.

    Eliezer Yudkowsky hits a home run when he illustrates just how difficult it is to implement a utility function to save grandma from a burning house. … He shows that generating an ever longer list of motives/values for a utility function not only will not solve the problem, it will in all likelyhood lead to disasterous behavior by the AI.”

    Fortunately reality doesn’t completely reconfigure itself too often (last reconfig: big bang) – once you get your responses right, you’re done.

  7. Not to put too fine a point on it, but Eliezer Yudkowsky and his ilk are delusional, dangerous and quite possibly insane. They all suffer from the Western disease of hyper-abstraction and hyper-rationality. Reality remains forever beyond all our models and attempts at abstraction, because no matter what we do, we can never escape our own consciousness.

    The cure for this disease is self-reflection. Spend less time in your mathematical monkey minds and more time experiencing the moment through meditation and direct action. Heed the words of Zen master Bruce Lee, who was in his way a superman of far more impressive accomplishments than any Singularitarian:

    “Liberate yourself from concepts and see the truth with your own eyes. — It exists HERE and NOW; it requires only one thing to see it: openness, freedom — the freedom to be open and not tethered by any ideas, concepts, etc. … When our mind is tranquil, there will be an occasional pause to its feverish activities, there will be a let-go, and it is only then in the interval between two thoughts that a flash of UNDERSTANDING — understanding, which is not thought — can take place.”

    Perhaps if Yudkowsky received a beating at the hands of an enlightened master such as Lee, he would begin to understand. His ideas are a form of intellectual violence which need to be met with equal violence before they destroy the world.

  8. Dave

    @GK

    Regarding your proposed strategy of “isolating the ability to recognize human will.” I am not optimistic that this is a solvable problem, at least in the theoretical sense. Heuristic solutions abound, but what are we willing to stake on a heuristic?

    I definitely agree reinforcement learning is key to intelligence and friendliness, as well as are many other iterative/evolutionary approaches, but they are heuristic solutions and will not be perfect. Any learning machine (including us) is dependent on its input (garbage in –> garbage out). No matter how many cases of a problem a learning machine encounters, there will always be inputs on the fringe that will be misinterpreted.

    These solutions take real time also. The AI needs to interact with a real human through enough iterations, so it can’t be done in a simulation going faster than realtime (how would you simulate the human?). A 10 year old has a brain superior to an adults, but obviously lacks the experience and training necessary to more fully understand the complicated needs and wants of those around him or her. It will take another 10+ years to reach that point.

    You wrote “A tool to do this is to make the AI make only true statements to its user.” This is extremely problematic. Any learning machine’s truth is dependent on its input as already mentioned. Just imagine an otherwise “honest” AI that encounters an input sufficiently persuasive enough to make it believe a new truth: that being truly honest (maximizing a utility subfunction) actually requires it to make certain misleading statements to its human master. Suppose there are “hardwired” countermeasures to this in the AI. Then the AI may be able to subvert them by, say, telling the truth when it knows the master is asleep or otherwise unable to actively perceive the statement. Then you are in an arms race of measures and countermeasures all over again. I happen to believe that the ability to deceive is a nasty emergent property inherent to all systems we would consider intelligent.

    So to reiterate, I think whether we enhance ourselves or enhance machines or both, it must be in an incremental and universally available way if we don’t want to gamble with civilization.

  9. I think I understand the problem: We humans sometimes mess up when we’re learning something new.

    But what’s the solution?

  10. I love Yudkowsky’s examples. … Saving grandma from the house. … And the baby eating bugs. But they don’t really prove his point: that, we’re all doomed if we mess things up one teeny-tiny leeetle bit by missing one leeeetle human value in constructing an AI. A lot of those values are overlapping. Let’s face it, there are no absolutes, just approximations. But some are better than others.

    And something that seems to be missed by everybody is that humans can be f’ing dumb, too. We’ll eventually get to a point through iteration where we trust our created superintelligent beings (be they computers or enhanced humans) more and more in specific areas.

    At that point, we’ll slowly (or quickly), but surely turns things over control in a democratic fashion. Currently, you permit an autopilot to fly the jet liner you’re on by virtue of boarding the thing. More and more we’ll put our trust into more advanced AI. And so we’ll go.

    Part of that will be the creation of AI to help create and monitor AIs.

    Why is there this fear that some totally individuated entity is going to jump out of the mix and turn us all into paperclips? I mean, how could that REALISTICALLY happen?

    I just don’t understand where the we’re-all-doomed-if-we-don’t-get-this-absolutely-perfect idea comes from.

    I’d love it if somebody could explain that to me.

  11. Hieronymus

    “I just don’t understand where the we’re-all-doomed-if-we-don’t-get-this-absolutely-perfect idea comes from.”

    You’ll grow old waiting for an answer to that.
    I’ve never seen any real-life security and safety scenarios addressed by SIAI (likely for a good reason: because they aren’t data security experts).

    The message is: We just can’t handle our bits, trust me!

  12. PlacidCountenance

    So when are we going to get past all this pontification about how hard the problem is and actually see some real work on solving the problem.

    Am I the only one who is getting annoyed with this constant stream of non-technical buildup and drivel about how hard FAI is and how epic the struggle is? Why doesn’t EY actually make some real progress here? Is he even able to solve this problem?

    This is getting absolutely ridiculous, how many years are we going to put with this, constant campaign of EY is a guru, he is going to solve the problem, yet the only things we see from EY and SIAI are these useless repetitive posts and talks etc on how hard the problem is and how cool it would be if it were solved. Of course we are also blessed with the PR about how smart EY is and important he is, honestly I have never met anyone outside of this community who knows who EY is. In fact EY doesn’t even pass muster on academics, or credentials. We should demand some form of proof of his abilities in this area if SIAI expects us to continue to support this “R&D” which after a decade has amounted to what a single reasonably productive individual can do in a year or two.

    I think we should demand results. Does SIAI expect us to just continue to give our support without results? Are we as donors or potential donors going to continue to allow this?

  13. I really like Eliezer’s message, but I believe his evolutionary theory is not as nuanced as it could be. A couple quick thoughts-

    - http://johnhawks.net/weblog/topics/evolution/theory/inclusive-fitness-nowak-wilson-2010.html for some context for modern kin selection / ‘inclusive fitness’ theory;

    - Eliezer’s comments wrt balancing selection for equal numbers of boys and girls assumes that humans can be treated as monogamous, and/or the profiles of reproductive success between men and women are roughly similar. Historically, this hasn’t been the case. Throughout the history of humans, about ~40% of males have reproduced. About ~80% of females have reproduced. (Males tend to be both bigger genetic winners, when they reproduce a lot with many partners, and bigger genetic losers, when they fail to reproduce, than women.)

    Implication: the relative genetic advantage of having more boys vs girls gets more complicated, and it becomes less of matter of balancing selection, and more of a “play it safe” (have more girls) or “bet the house” (have more boys) bet.

  14. Really exceptional info is available on internet site.

  15. I’m really, I require declare. Really rarely do I encounter a blog that’s both educative and celebrating, and let me tell you, you have hit the nail on the head. Your idea is outstanding; the issue is something that not enough people are giving a presentation intelligently about. I am very seriously happy that I ended up across this in my search for something relating to this.

  16. Thanks with regard to giving your opinions listed right here. The some other point can be that at any time a difficulty occurs which has a computer technique motherboard, individuals should not take danger regarding mending this by themselves because when it is not carried out correctly it can lead to permanent damage to the total laptop. It will almost always be safe only to approach your current dealer of one’s laptop to the repair of the motherboard. They’ve by now technicians who’ve an knowledge when controling laptop motherboard problems and will get the best prognosis and also conduct vehicle repairs.

  17. http:,cache:,site:,utm_source,sex,porn,gamble,xxx,nude,squirt,gay,abortion,attack,bomb,casino,cocaine,die,death,erection,gambling,heroin,marijuana,masturbation,pedophile,penis,poker,pussy,terrorist

  18. Wow, what a video it is! Truly fastidious feature video, the lesson given in this video is actually informative.

  19. This is the precise weblog for anybody who needs to find out about this topic. You understand a lot its almost onerous to argue with you (not that I truly would want…HaHa). You positively put a brand new spin on a topic thats been written about for years. Nice stuff, simply great!

  20. I wish to show my own appreciation for your requirements just pertaining to rescuing me using this particular matter. Right right after searching throughout the web and experiencing things of not enjoyable, I assumed my entire life was around. Living minus the presence of answers to the troubles you’ve solved by way of this guide can be a critical circumstance, as effectively as ones which might have inside a wrong method damaged my career easily hadn’t encounter the web site. The skills and kindness in taking good care of several things was essential. I am unclear what My partner and i would’ve done if i hadn’t found such things like this. I can at this point relish our future. Thanks a lot for the professional along with result driven guide. I won’t think hard to recommend the website to anyone who requirements guidelines with this subject make a difference.

  21. I like the helpful info you provide in your articles. I’ll bookmark your weblog and check again here frequently. I’m quite certain I will learn many new stuff right here! Best of luck for the next!

  22. An interesting dialogue is price comment. I think that you need to write extra on this topic, it might not be a taboo subject but typically people are not enough to speak on such topics. To the next. Cheers

  23. I just like the helpful info you supply on your articles. I’ll bookmark your weblog and test again right here frequently. I’m moderately certain I’ll learn many new stuff right here! Best of luck for the next!

  24. The display savings are more typically incorporated inside of various tour packages, specifically with the increasing pattern of lounges and resorts hosting these demonstrates. .

  25. I mastered more new things on this weight loss issue. A single issue is that good nutrition is very vital when dieting. A tremendous reduction in bad foods, sugary food, fried foods, sugary foods, beef, and white-colored flour products may be necessary. Keeping wastes organisms, and harmful toxins may prevent desired goals for losing weight. While particular drugs for the short term solve the matter, the awful side effects are not worth it, and they never offer more than a momentary solution. This is a known idea that 95% of fad diet plans fail. Many thanks sharing your notions on this blog site.

Leave a Comment

Awesome! You've decided to leave a comment. Please keep in mind that comments are moderated.

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>