Accelerating Future Transhumanism, AI, nanotech, the Singularity, and extinction risk.


Why We Need Friendly AI

An article I often point people to is "Why We Need Friendly AI", an older (2004) article by Eliezer Yudkowsky on the challenge of Friendly AI:

There are certain important things that evolution created. We don't know that evolution reliably creates these things, but we know that it happened at least once. A sense of fun, the love of beauty, taking joy in helping others, the ability to be swayed by moral argument, the wish to be better people. Call these things humaneness, the parts of ourselves that we treasure – our ideals, our inclinations to alleviate suffering. If human is what we are, then humane is what we wish we were. Tribalism and hatred, prejudice and revenge, these things are also part of human nature. They are not humane, but they are human. They are a part of me; not by my choice, but by evolution's design, and the heritage of three and half billion years of lethal combat. Nature, bloody in tooth and claw, inscribed each base of my DNA. That is the tragedy of the human condition, that we are not what we wish we were. Humans were not designed by humans, humans were designed by evolution, which is a physical process devoid of conscience and compassion. And yet we have conscience. We have compassion. How did these things evolve? That's a real question with a real answer, which you can find in the field of evolutionary psychology. But for whatever reason, our humane tendencies are now a part of human nature.

We need to develop our conception of "good" to mean certain cognitive features built by evolution, rather than some metaphysical miasma floating around.

Comments (30) Trackbacks (0)
  1. (This is a serious post, please do not delete!)

    Here’s my idea for giving these issues more public exposure: A TV show about the adventures of the “the Singularity Institute” (probably need a more heroic name – the Singularity Corps?), an organization of young geniuses funded by a shadowy billionaire who save the world each week from mad scientists, cyber-criminals, evil roboticists, renegade Chinese generals and other megalomaniacs who are trying to take over the world via AI, cyberwar, robot armies, etc. SI has a super-slick control center from which they monitor global networks for signs of trouble. One of the main characters is a Spock-like AI theorist with a fanatical devotion to rationality who likes to invoke Bayesian probability when making decisions. The team includes “ninjas” skilled at physically infiltrating secure compounds “Mission Impossible” style. Their arch-enemies are “the Omega League”, a diabolical group of anonymous technologists who are constantly attempting to take over computer systems and implement their nefarious “borgification” plans for humanity.

    That’s all I have so far — what do you folks think?

  2. I found this bit interesting:

    it’s no longer about computers but NETWORKS of computers- and the CLOUD is ALREADY an exaflop computer [there are 10^9 active mobile devices with 10^9 flops each for 10^18 flops- also there are 10^8 PCs with 10^10 flops each- another exaflop]! it is just in the process of compiling- it is transitioning from sharing data to sharing computation resources
    consider this- the HUMAN network of memes and culture and conscious memory is a biological network of 10^9 brains each with about 10^16 ops- although the bandwidth between brains is very low: symbolic language- the whole neural supernetwork is about 10^25 ops

    now consider that by the late 2020s our mobile devices will each be hitting the same power of the brain: 10^16 flops – petaflop mobiles and devices-

    but AT THE SAME TIME at least basic BCI with more bandwidth than symbolic language between humans [at least basic audiovisual AR/VR and natural language recognition/basic brainwave reading] will for the first time allow the biological network of humans to cybernetically connect with the machine network/cloud at the level of each human/device

    but when the two networks connect- they will be EQUIVALENT- in size/complexity the human network of memes and neurons is 10^25 ops- and as they connect the machine network will be 10^25 flops [10^16 flops times 10^9 devices]!

    the fact that all indications show that the PLANETARY networks – the biological and the machine- will MERGE into a cybernetic whole JUST as they are EQUAL is just too big a coincidence my friends

    this is the synchronicity of synchronicities- a clear sign of a teleology at work- a planetary self-organizing principle

  3. Luke Mehlhauser SIAIs star on the rise?

    I always find it odd what the robot cultists to borrow dales term come up with when it comes to these kind of money spending endeavors. They want to hire yet another in a long list of completely unremarkable individuals to fill the ranks of their robot cultist stronghold. The bastion from which they can call all the non-believers idiots.

    It is amazing to look at the research staff as it stands and consider the abysmal state which it is in:
    Eliezer Yudkowsky: can’t pass muster as an academic and doesn’t have a highschool diploma. He has apparently published his Timeless Decision Theory paper on the website and this is supposed to be a substitute for real academic publuication? Try resume padding.

    Anna Salamon: (insufficient data)

    Vladimir Nesov: one of the few individuals who actually holds a degree that is above the average of SIAI. The unfortunate part is that while an MS is better than a BS in science getting an MS is considered a waste unless you also get a PhD since an MS only proves you are capable of learning known facts on your own but gives little evidence of much ability for independent thought. We can add the lack of real publication as far as I can tell and we have a recipe for another disappointment.

    Daniel Dewey: this is a complete joke a BS degree in a rapidly changing field proves that you can learn but nothing else. This combined with an absence of any real achievement leaves one wondering what possible contribution he will make?

    Steve Rayhawk: (insufficient data)

    Peter de Blanc: another one of the MS holding people with little potential at SIAI. The real issue here is that one of his “publications” is considered published even though it is only ARXIV (resume padding).

    Overall an unimpressive bunch. What is needed is some real academics and scholars what is missing some real academics and scholars.

    Now we come to questionable decision of hiring Luke with the meager funds SIAI has. Well let us consider the following:
    Luke has:
    No degree
    Is an IT Consultant
    No academic publications
    Speaks of his LW writings as being published a clear case of resume padding and woeful ignorance of publication
    A website devoted to his many examples of his own failure to contribute to the actual techno-scientific discourse. See the list of “books” he is going to write but never did or gave up on:
    TITLE: “Machine ethics for superintelligence: some challenges” [in progress]
    TITLE: Ethics and Superintelligence [in progress]
    TITLE: “The sources of normativity and friendly artificial intelligence”
    TITLE: “Machine ethics keeps moral philosophy honest”
    TITLE: “Machine ethics and Railton’s moral reductionism”
    These are just the top of the list there are many more. What I want to point out is that ALL not some ALL of these appear to be going nowhere. This is all posted as a monument to his lack of accomplishment on his own website.

    This is the kind of person SIAI wants to hire. This is what they want us as donors to pay for. This is how they reward our interest. They go and hire people who cannot pass muster. Is this what we want to call good business? Is this a positive step? I think the fact that on the current challenge SIAI has only managed to raise a mere ~$39,000 of the needed $125,000 shows that the confidence in SIAI is waning. No surprise they continue to assure us that they are doing good but can’t be bothered to prove it with their public actions of hiring or by the accomplishments they present.

    They should consider reducing staff and money intake and focus on the Singularity Summit only. Then and only then will it be safe to donate. Lets put an end to the waffle and the gimmicks, lets demand results and lets demand accountability.

  4. As someone who is thinking of donating – perhaps substantially – I would like to read a report, maybe with some graphs and an executive summary on the progress towards AGI and other goals of SIAI if there are any.

    What has been accomplished?
    What remains to be done?
    What questions have to be answered until the next step can be taken?
    A time line with estimates.

    Every project/research program I’ve worked on generates such data and it is usually very detailed and its predictions are fairly accurate, much better than any outside observer’s.

  5. Wow. Tough crowd.

    I still don’t understand where the risk is?
    Why worry that an AI genie is going to pop out of an AI genie bottle and kill us all?

    We’ll create friendly AI by crowd sourcing the problem, just as we are doing right now, all humans being the crowd. First we’ll have imitation AI cobbled out of voice synthesizers, search engines and wikis. The intelligence of super AI will be the product of all humanity, via our search queries and responses to search results. We’ll develop better and better information synthesizers. So, our interactions will be more like conversations with somebody with knowledge, rather than someone simply reading off a page. Then our imitation AIs will get better and better to the point where we start dating and marrying our robots and cyborgs.

    Incidentally, EY says that we have a chance to succeed because “because we are the first. And we have a chance to fail, because we are the first.” Why think that? Maybe there are plenty of ultra-intelligent civilizations out there and they simply don’t care to let us know about them.

  6. Not a tough crowd. Exaggerated promises. Reality diverging from stated plans.

    The response we’ve seen is reasonable and overdue. The program needs to be put back on track. SIAI needs management.

    • I’m not familiar with the inner workings of SIAI. Granted, when it comes to fundraising, people want to see something useful being done with their contributions. And people will have more confidence in lettered individuals at the helm of charitable institutions.

      But, ultimately, I couldn’t care less about credentials. Either somebody makes sense or they don’t. I’m all for the mavericks.

  7. Hi Mike. Big fan.

    If human is what we are, then humane is what we wish we were. […] That is the tragedy of the human condition, that we are not what we wish we were.

    That we are not what we wish we were is not the tragedy of the human condition; the human condition is not a tragedy. To paint it as such is unsubstantiated commentary. It is an old school, highly magnetic characterization though, sure to attract supporters.

    Is it not, by exactly the same (stunning) reasoning, the brilliance of the human condition that we are not what we wish we weren’t?

    It is accurate (enough) to say that we humans are the only species stupid enough, or if you prefer, “in possession of the appropriate operational architecture”, to construct almost-palpable images of what we wish we were; to piece together an ideal, picturesque configuration of stored memories and imaginarily isolated (and intelligently extrapolated) experiences… and then feel discontented – feel that life in its current state measures up poorly (read, tragically) alongside this conceivable-but-sadly-not-real fantasy.

    This, for the person imagining, is indeed a bit tragic.

    Maybe that’s what lies behind Eliezer’s overcommunication here. Or maybe it’s not.

    • Firstly, brilliance is not the antonym of tragedy.

      Secondly, ‘we are not what we wish we weren’t’ is incoherent as it stands.

      Perhaps you mean ‘we are relieved not to be the kind of beings which conceivably we might have been?

      in any case it is not the reverse corollary of Eliezer’s statement as you seem to think.

  8. I’ve heard that this article, along with another by Kaj Sotala, will be included in an upcoming (2011) edited volume by Springer.

    Both of my submissions for the Springer volume were rejected, I’m afraid.

  9. I precisely needed to thank you so much once more. I am not sure the things that I would have made to happen in the absence of the entire tactics documented by you directly on such a situation. It previously was a real frightful problem in my opinion, but finding out your well-written mode you processed the issue took me to leap with gladness. I am just happier for this advice and then hope you really know what a great job you were providing educating men and women all through your webpage. I am certain you haven’t encountered all of us.

  10. Spot lets start on this particular create-upward, I critically think this excellent website wants much more thing to consider. I’ll apt to be once more to review far more, many thanks for that info.

  11. Fantastic blog you have here but I was curious if you knew of any user discussion forums that cover the same topics talked about in this article? I’d really love to be a part of community where I can get suggestions from other knowledgeable individuals that share the same interest. If you have any recommendations, please let me know. Appreciate it!

  12. It’s in truth a winsome and utilitarian piee of info. Im exultant thay you shared this tidings with us. Gratify, qualification sharing swain like this. Point the finger at you again.

  13. I have observed that during the period of making a relationship with real estate proprietors, you’ll get them to comprehend that, in each and every real property purchase, a payment amount is paid. In the final, FSBO dealers don’t conserve the fee. Rather, they attempt to win your commission through doing a strong agent’s career. In completing this, they make investments their funds and time to complete, as best they can, the responsibilities associated with an realtor. Those assignments include disclosing your home via marketing, showing your home to inclined buyers, developing an awareness of buyer emergency in order to prompt the offer, preparing all about home inspections, taking upon qualification examinations with the loan company, supervising upkeep tasks, and supporting the closing in the deal.

  14. hey there and thank you for your information – I have definitely picked up anything new from right here. I did however expertise several technical issues using this website, since I experienced to reload the web site lots of times previous to I could get it to load correctly. I had been wondering if your web hosting is OK? Not that I’m complaining, but sluggish loading instances times will sometimes affect your placement in google and could damage your high-quality score if ads and marketing with Adwords. Well I am adding this RSS to my email and can look out for a lot more of your respective exciting content. Make sure you update this again soon..

  15. I have mastered some points through your website post. One other subject I would like to state is that there are many games in the marketplace designed specially for toddler age little ones. They include pattern acceptance, colors, creatures, and forms. These commonly focus on familiarization as an alternative to memorization. This keeps little children occupied without feeling like they are studying. Thanks

  16. We’re a group of volunteers and opening a new scheme in our community. Your web site offered us with valuable info to work on. You have done a formidable job and our whole community will be thankful to you.

  17. What’s up, all right brother there are sure multiple blogging blogs, but I advise you to use Google’s free of charge blogging services.

  18. I like what you guys are up also. Such intelligent work and reporting! Keep up the excellent works guys I’ve incorporated you guys to my blogroll. I think it will improve the value of my site :).

  19. Hello, Neat post. There’s an issue together with your website in internet explorer, would check this… IE nonetheless is the marketplace chief and a large section of other folks will pass over your magnificent writing because of this problem.

  20. I¡¦m no longer positive the place you are getting your information, but great topic. I must spend a while learning much more or working out more. Thanks for magnificent information I was looking for this info for my mission.

  21. Undeniably believe that that you stated. Your favorite justification seemed to be at the web the easiest thing to consider of. I say to you, I certainly get annoyed while folks think about concerns that they just do not realize about. You managed to hit the nail upon the top as well as outlined out the entire thing without having side effect , folks can take a signal. Will likely be again to get more. Thanks!

  22. I like this blog very much, Its a real nice place to read and incur info. “That which has been believed by everyone, always and everywhere, has every chance of being false.” by Paul Valery.

  23. I like this site very much, Its a rattling nice billet to read and get info . “You have to lead people gently toward what they already know is right.” by Philip.

  24. You are my inhalation, I own few web logs and often run out from post :). “To die for a religion is easier than to live it absolutely.” by Jorge Luis Borges.

Leave a comment

Trackbacks are disabled.