Accelerating Future Transhumanism, AI, nanotech, the Singularity, and extinction risk.


A Boring Disagreement?

My disagreement with Dale Carrico, Mike Treder, James Hughes, Ray Kurzweil, Richard Jones, Charles Stross, Kevin Kelly, Max More, David Brin, and many others is relatively boring and straightforward, I think. It is simply this: I believe that it is possible that a being more powerful than the entirety of humanity could emerge in a relatively covert and quick way, and they don't. A singleton, a Maximillian, an unrivaled superintelligence, a transcending upload, whatever you want to call it.

If you believe that such a being could be created and become unrivaled, then it is obvious that you would want to have some impact on its motivations. If you don't, then clearly you would see such preparations to be silly and misguided.

Why do people make this more complex than it needs to be? It has nothing to do with politics. It has everything to do with our estimated probabilities of the likelihood of a more-powerful-than-humanity being emerging quickly. I am practically willing to concede all other points, because I think that this is the crux of the argument. Boring and simple, if I am indeed correct.

I am fairly confident that, at this point in history, superintelligence is the MacGuffin -- the key element that determines how the story of humanity will go. I could be entirely wrong, of course, but that is my current position, and it is derived from cogsci and economics-based arguments about takeoff curves, not political nonsense. If it is wrong, it should be entirely simple to refute the hard takeoff hypothesis at the locus of cogsci and economics-based arguments rather than political or sociological arguments. Particularly, I think that James Hughes, as a sociologist, seems to have a desire to search for a "sociological" (social signaling/subcultural) explanation for other people's beliefs, rather than looking at the economics/cogsci side of the arguments, which is their entire substance. You have to note that the people that believe in hard takeoff hypotheses are mostly subculturally isolated from one another, and barely even come into geographical contact. What wins us over are abstract arguments like, "humans are qualitatively smarter than chimps and have a huge advantage over them; why couldn't there exist a superintelligence that has a similar qualitative advantage over us?"

Comments (96) Trackbacks (0)
  1. for me, its not probability but possibility that matters. if its even remotely possible quick, repeated self-modification could occur than the implications, if true, are so large they are worthy of massive amounts attention from everybody.

    its like even if there’s a 1% probability of a nuke being detonated in NYC, the fact that such destruction is possible compels everybody to agree with massive expenditures towards preventing that sort of terrorism.

    i think you should push that sort of 1% argument, rather than get into an argument over what is more likely. because as i see it, even preparing for the rapid-scenario encompasses preparation for the slow takeoff scenario aswell.

  2. I think that Hughes probably takes the sociological ground assumptions of assuming that people’s beliefs should be described exclusively in terms of their sociological causes, not as the results of rational arguments. The idea of truth causing beliefs is mistakently seens as superstitious because the sociologists don’t understand the mechanisms by which map entangles territory but they do see memetic conflicts. Object level arguments also win with Robin, but he also ignores them, which is less defensible from an econ perspective. In any event, almost everyone generalizes from history and concludes that in practice people won’t be honestly swayed by object level arguments so they fear technocracy. They aren’t open to object level arguments for why this time will be different because there were supposedly such arguments last time.

  3. Microsoft Windows arose “relatively quickly” – and spread its DNA all over the planet – mostly obliterating its rivals. Machine Intelligence will probably be a bit like that.

    On the other hand, computers have been developing for a hundred years or so now. Their rise has happened quickly on geological scales – but has not been *that* rapid on a human timescale.

  4. Google popped up fairly quickly too – and it clearly represent a self-improving ecosystem. If either they or Microsoft are responsible for the first superintelligent machine, we should presumably have started the clock ticking back when those companies were formed – right?

  5. > Why do people make this more complex than it needs to be?

    Because the conclusions that an honest analysis would lead to are rarely the ones that would be most convenient to the person doing the analysis.

  6. Machine intelligence has taken some 50 years to “emerge” so far – at least. So: we know it takes at least that long.

    Didn’t start the clock ticking back then? Don’t even have the the clock ticking yet?!? When is it going to get started, then?

  7. Boring indeed! What is NOT boring are the techniques how it will be done, one sunny day in the quite near future.

    But I wouldn’t discuss that. Who will?

  8. It is indeed a boring disagreement because even with some small but significant probability of a super-intelligent agent emerging with surprising quickness, the important question is not the *narrow* one of how to ensure that the first such singleton is “friendly” (however you might attempt to define it), but rather, how best to promote a *broad* network of robust and resilient greater-than-human-level intelligence reflecting the evolving values of humanity.

  9. jefallbright,

    When you take as the starting point of your analysis the scenario of a super-intelligent agent emerging into a position of power with surprising quickness, any hypothetical “broad network” representing positive values will have to *include* the first such superintelligence. If the first superintelligence achieving great power doesn’t represent positive values, it doesn’t matter if you later attempt to launch a broader network, since the first one has already won (according to the specification of the scenario under analysis).

    So what you suggest isn’t actually an alternative to what Michael has suggested. Instead, a solution to what Michael sees as the key problem is *included* in the requirements of your approach as well.

  10. “economics-based arguments about takeoff curves”

    What does economics have to do with it?

  11. Aleksei,

    It sounds like you assume that superintelligences cannot be result from augmented human minds. Something like this, while probably not capable of the rapid self-improvement by leaps and bounds that Michael speaks of, would only need to be slightly more intelligent than any other human to be considered a super-human intelligence.

    A world containing a substantial number of enhanced people who have become, say, 50% smarter than Newton or Von Neumann or Terry Tao [or your favorite genius here] would go a long way toward improving our world, assuming their emotional maturity and moral reasoning is at least as strong as the best of today. (And I sincerely hope we are moving toward this.) They could continue to improve themselves, and make it easier for others to enter on the same or similar paths. And one of the things they likely would guard against is a malicious or amoral AGI.

    I wonder if this is not too dissimilar from what Jeff has in mind above.

    (By the way, it’s a safe bet that these minds could answer with greater confidence than ours the question of “how probable is the emergence of a hidden, quick-bootstrapping AGI?” Of course, that doesn’t help us to answer it at the moment…)

  12. Why do you call a Maximilian a Maximilian? Where did you originate the term? I’m just curious.

  13. Mitchell, because simple economic arguments about the productivity and costs of AI reproduction contribute a lot of probability mass to hard takeoffs. A preliminary modeler of that idea is here.

    Kingraven, Maximillian means “the greatest” in Latin, and a truly smarter-than-human being would indeed be the greatest entity on the Earth in terms of its power. I also like the term because it refers to historical rulers and I think that the world under a human-derived superintelligence would plausibly be much more like past monarchies than an AI-derived superintelligence would be. A human-derived superintelligence could still want a lot of human typical things, like social power, recognition, human status, etc. (Of course, it might not, if it transcended those desires.) I doubt that an AI would be programmed to specifically value these things. A Maximillian could also just be a plain old human that has a contained superintelligence under his or her control, though that scenario might not last long.

  14. Thank you for explaining, that was enlightening and very well put.

  15. Michael – OK. I thought you meant arguments from economic theory, not from Moore’s law.

  16. It’s not from Moore’s law. It is from the economic tradition of making very simple assumptions and seeing how they play out in a model.

  17. These templates glance wonderful. We have a fantastic idea for a single of them.

  18. Wow, this piece of writing is good, my sister is analyzing these kinds of things, therefore I am going to inform her.

  19. I enjoy what you guys tend to be up too. This kind of clever work and exposure! Keep up the great works guys I’ve incorporated you guys to our blogroll.

  20. The page isn’t fully loading for some reason.

  21. What i don’t realize is actually how you are not really much more neatly-preferred than you might be now. You’re so intelligent. You understand thus considerably in the case of this topic, produced me for my part believe it from numerous numerous angles. Its like women and men don’t seem to be interested until it is something to do with Lady gaga! Your own stuffs nice. At all times care for it up!

  22. I feel other sorts of internet business operators really need to look at this amazing homepage as a model. Fantastically clean and intuitive style and design, and moreover great information! You are a guru with regard to this amazing topic area :)

  23. One a lot additional thing is that after evaluating an excellent on-line electronic policy shop, glare with regard to web shops which have been incessantly up-to-date, keeping up-to-date most copious in up-to-date harvest, the finest deals, and effective tips by services. This will bring in confident you are employing a shop that accurately stays atop the competition and supply you what you need to compose well-informed, well-informed electronic devices buys. Gratitude for your momentous tips I have especially learned by means of the blog.

  24. Thank you for the good writeup. It actually was once a amusement account it. Look complex to far brought agreeable from you! By the way, how could we keep up a correspondence?

  25. I discovered more interesting things on this weight reduction issue. 1 issue is that good nutrition is tremendously vital if dieting. A massive reduction in bad foods, sugary foods, fried foods, sugary foods, beef, and white flour products may be necessary. Holding wastes unwanted organisms, and toxins may prevent goals for losing fat. While a number of drugs briefly solve the matter, the nasty side effects are not worth it, and they also never provide more than a short-term solution. It’s a known proven fact that 95% of fad diet plans fail. Many thanks for sharing your opinions on this web site.

  26. You made some decent points there. I appeared on the internet for the issue and found most individuals will go along with with your website.

  27. Oh my goodness! a tremendous article dude. Thank you However I am experiencing difficulty with ur rss . Don’t know why Unable to subscribe to it. Is there anyone getting similar rss drawback? Anyone who is aware of kindly respond. Thnkx k9uq31s

  28. It’s arduous to search out knowledgeable folks on this subject, but you sound like you know what you’re speaking about! Thanks k9uq31s

  29. I think the bigger picture is being over looked in this article. We know without a doubt that cigarette smoking and tobacco use causes cancer. We think that maybe one ingredient in Electronic Cigarettes may be linked to some isolated cases of cancer. Yet, we’re trying to remove electronic cigarettes from the market. Do you see the irony I’m trying to suss out here?

  30. In this great pattern of things you actually secure an A+ with regard to effort. Where exactly you actually lost me personally was in your particulars. As people say, the devil is in the details… And that could not be much more true here. Having said that, permit me tell you precisely what did give good results. The writing is certainly quite persuasive and this is possibly the reason why I am making the effort in order to comment. I do not really make it a regular habit of doing that. Next, despite the fact that I can see the leaps in reasoning you come up with, I am not necessarily sure of exactly how you seem to unite the ideas which inturn make the actual conclusion. For right now I will, no doubt subscribe to your issue however trust in the future you link your facts better.

  31. Hello there! I know this is kinda off topic however , I’d figured I’d ask. Would you be interested in trading links or maybe guest authoring a blog post or vice-versa? My blog addresses a lot of the same topics as yours and I think we could greatly benefit from each other. If you’re interested feel free to send me an email. I look forward to hearing from you! Excellent blog by the way! jdei726ya

  32. It adds to your graceful appearance the same way the other Lady Datejust collections do

  33. Vey good write-up. I ertainly appreiate this webste.
    Keep t up!

  34. For reasons unknown, the words are broken on this blog

  35. This new Zune mobile phone can be shockingly fantastic, however not as good as a iPod’s. It does work very well, however isn’t as swiftly mainly because Ie, and features a clunkier vent. Any time you now and again plan on together with the internet browser that was not a huge concern, in case you plan to be able to investigate the website alot from the PMP then a iPod’s greater tv screen far better internet browser could possibly be fundamental.

  36. whoah this blog is fantastic i really like studying your articles. Keep up the great work! You recognize, lots of individuals are searching round for this info, you can help them greatly.

  37. Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something. I think that you could do with a few pics to drive the message home a little bit, but instead of that, this is excellent blog. A fantastic read. I’ll definitely be back.

  38. Thanks for the post, I’ve put your link on my blog.

  39. Subsequently, after spending many hours on the internet at last We’ve uncovered an individual that definitely does know what they are discussing many thanks a great deal wonderful post

  40. Youre so cool! I dont suppose Ive read anything like this before. So nice to search out any individual with some original thoughts on this subject. realy thank you for starting this up. this website is one thing that’s wanted on the web, somebody with a bit of originality. useful job for bringing something new to the internet!

  41. Just after representation whatsoever of the blogposts I moldiness say i pioneer this particularised 1 to typically be best nick. I mortal a weblog also and necessary to repost a number of shear of your articles on my personal diary tract. Should it be alright if I use this as daylong I private reference your web diary or produce a incoming linkage to your article I procured the snipping from? If not I make and could not do it with no having your tolerance . I hump collection starred this article to cheep and zynga calculate motivated for publication. Anyway revalue it either way!

  42. After examine a number of of the blog posts on your website now, and I really like your way of blogging. I bookmarked it to my bookmark web site record and will likely be checking again soon. Pls take a look at my web site as properly and let me know what you think.

  43. Are you able to add a Blackberry template? This web page is tricky to read otherwise for all those of us browsing with cell phones. Otherwise, in the event you can put a RSS link up, that might be good also.

  44. Advantageously, typically the submit is really the very best about this laudable theme. To be sure with all your a conclusion and will thirstily await the following revisions. Really stating cheers won’t only end up being suitable, to your good readability within your creating. I may at once seize a person’s rss to sleep in abreast of virtually any upgrades. Fine job and much success within your organization business!

  45. I would like to thnkx for the efforts you’ve put in writing this site. I’m hoping the same high-grade web site post from you in the upcoming as well. Actually your creative writing abilities has inspired me to get my own site now. Really the blogging is spreading its wings quickly. Your write up is a good example of it.

  46. Another thing I’ve noticed is the fact that for many people, low credit score is the reaction of circumstances beyond their control. By way of example they may be actually saddled with an illness and as a consequence they have substantial bills going to collections. It could be due to a work loss or inability to work. Sometimes separation and divorce can really send the financial situation in the undesired direction. Thanks for sharing your thinking on this weblog.

  47. Hi there, just changed into aware of your blog via Google, and located that it’s truly informative. I’m going to be careful for brussels. I’ll be grateful if you happen to continue this in future. Many other folks will likely be benefited from your writing. Cheers!

  48. Whats up are using WordPress for your blog platform? I’m new to the blog world but I’m trying to get started and create my own. Do you need any coding expertise to make your own blog? Any help would be really appreciated!

  49. Hmm it appears like your website ate my first comment (it was super long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I as well am an aspiring blog blogger but I’m still new to the whole thing. Do you have any tips and hints for beginner blog writers? I’d genuinely appreciate it.

  50. obviously like your web-site but you need to check the spelling on quite a few of your posts. Several of them are rife with spelling issues and I find it very bothersome to tell the truth nevertheless I’ll surely come back again.

  51. Hi There, Great information on marketing from home on your website. Thanks, and I will returning for updates whenever I can.Many thanks,

  52. Hello Guru, what tempt you to post an article. This article was exceedingly fascinating, especially since I was searching for thoughts on this subject last Thursday

  53. I just cant stop reading this. Its so cool, so choked with info that I simply didn’t know.

  54. If you are interested to learn Web optimization techniques then you have to read this paragraph, I am sure you will take much more from this piece of writing on the topic of Web optimization.

  55. Outstanding placement, you have incisive out some admirable points, I also imagine this is a fantastically admirable website.

  56. Fantastic web site. A lot of useful information here. I am sending it to some friends ans also sharing in delicious. And obviously, thanks for your sweat!

  57. Thanks for your publication. I also think that laptop computers have grown to be more and more popular these days, and now are often the only type of computer found in a household. This is because at the same time potentially they are becoming more and more reasonably priced, their computing power keeps growing to the point where they are as effective as desktop computers from just a few in years past.

  58. I have discovered some new elements from your web site about desktops. Another thing I’ve always presumed is that computers have become a specific thing that each residence must have for a lot of reasons. They supply you with convenient ways in which to organize households, pay bills, go shopping, study, listen to music and in many cases watch television shows. An innovative strategy to complete most of these tasks is a notebook computer. These computer systems are mobile ones, small, highly effective and convenient.

  59. From my research, shopping for electronic devices online may be easily expensive, nonetheless there are some tricks and tips that you can use to help you get the best discounts. There are always ways to come across discount discounts that could help make one to come across the best consumer electronics products at the cheapest prices. Thanks for your blog post.

  60. Excellent site. Lots of useful info here. I’m sending it to a few friends ans additionally sharing in delicious. And of course, thank you to your sweat!

  61. Hey! This is kind of off topic but I need some advice from an established blog. Is it difficult to set up your own blog? I’m not very techincal but I can figure things out pretty fast. I’m thinking about making my own but I’m not sure where to start. Do you have any ideas or suggestions? Cheers

  62. Thank you for the good writeup. It actually was once a leisure account it. Glance complicated to far delivered agreeable from you! By the way, how can we keep in touch?

  63. Howdy! Someone in my Myspace group shared this website with us so I came to give it a look. I’m definitely enjoying the information. I’m bookmarking and will be tweeting this to my followers! Terrific blog and great design.

  64. Thanks for this excellent article. One other thing is that nearly all digital cameras come equipped with any zoom lens that allows more or less of the scene to get included by way of ‘zooming’ in and out. These kinds of changes in target length usually are reflected within the viewfinder and on big display screen at the back of the actual camera.

  65. Admiring the dedication you put into your blog and detailed information you present. It’s nice to come across a blog every once in a while that isn’t the same unwanted rehashed material. Great read! I’ve bookmarked your site and I’m including your RSS feeds to my Google account.

  66. Hi there! I know this is kinda off topic but I was wondering which blog platform are you using for this website? I’m getting sick and tired of WordPress because I’ve had problems with hackers and I’m looking at alternatives for another platform. I would be awesome if you could point me in the direction of a good platform.

  67. I absolutely love your blog and find nearly all of your post’s to be exactly I’m looking for. Do you offer guest writers to write content for you personally? I wouldn’t mind composing a post or elaborating on a few of the subjects you write in relation to here. Again, awesome site!

  68. Wow, this article is fastidious, my sister is analyzing these things, therefore I am going to tell her.|

  69. We are a bunch of volunteers and opening a new scheme in our community. Your website provided us with useful information to work on. You’ve done a formidable process and our whole community will probably be thankful to you.

  70. I don’t leave a ton of comments, however i did a few searching and wound up here %BLOG_TITLE%. And I do have 2 questions for you if you usually do not mind. Is it just me or does it give the impression like some of the comments appear as if they are written by brain dead visitors? :-P And, if you are writing on additional social sites, I’d like to keep up with anything new you have to post. Would you make a list of the complete urls of all your shared sites like your twitter feed, Facebook page or linkedin profile?

  71. Someone essentially assist to make seriously posts I might state. This is the very first time I frequented your web page and up to now? I surprised with the research you made to make this actual submit incredible. Fantastic process!

  72. I enjoy what you guys are usually up too. This type of clever work and reporting! Keep up the superb works guys I’ve included you guys to my personal blogroll.|

  73. Its good funny YouTube video, I every time go to pay a quick visit YouTube web site in favor of humorous videos, as there is much more data available.

  74. Hi there everyone, it’s my first pay a visit at this website, and article is really fruitful in support of me, keep up posting these posts.

  75. I simply had to thank you incredibly significantly once again. I’m not certain what I could possibly have followed within the absence with the type of ideas contributed by you concerning my theme. It truly was a frightful difficulty for me, nevertheless finding out the pretty professional manner you resolved the challenge forced me to jump more than joy. Now i’m thankful for the work and even hope that you’re conscious of a great job you’re carrying out training persons these days thru your weblog post. I know that you’ve got never got to know any of us.

  76. The Star Trek score is great also, Kirk’s theme is cool, the title theme is beautiful, while I missed the fanfare.

  77. Your blog is astounding dude. i love to visit it daily. extremely great layout and content ,

  78. What a lovely story! The tale in this YouTube video that is posted at this place is truly a fastidious one with having fastidious picture quality.

  79. I just added this web-site to my feed reader, great stuff. Can not get sufficient!

  80. Simply want to say your article is as astonishing. The clarity in
    your post is just spectacular and i could assume you’re an expert on this subject. Well with your permission allow me to grab your feed to keep updated with forthcoming post. Thanks a million and please keep up the rewarding work.

  81. I enjoy what you guys are usually up too. Such clever work and exposure! Keep up the terrific works guys I’ve incorporated you guys to my personal blogroll.

  82. Throughout the great pattern of things you actually receive an A for effort and hard work. Exactly where you actually lost me was on the facts. As as the maxim goes, details make or break the argument.. And that could not be more correct right here. Having said that, let me tell you just what did do the job. The text is certainly incredibly engaging and this is possibly why I am making an effort to comment. I do not really make it a regular habit of doing that. Next, while I can certainly notice a leaps in reasoning you come up with, I am definitely not convinced of how you appear to unite the points which in turn make the actual conclusion. For the moment I will, no doubt yield to your issue however trust in the future you link your dots better.

  83. Very good post. I will be facing some of these issues as well..

  84. Hello Professional explained content. Please post regularly and make us learn more.

  85. I love your blog.. very nice colors & theme. Did you design this website yourself or did you hire someone to do it for you? Plz reply as I’m looking to construct my own blog and would like to know where u got this from. kudos

  86. You made some clear points there. I did a search on the issue and found most people will approve with your blog.

  87. I have been exploring for a bit for any high quality articles or blog posts on this sort of area . Exploring in Yahoo I at last stumbled upon this site. Reading this information So i’m happy to convey that I have a very good uncanny feeling I discovered just what I needed. I most certainly will make certain to don’t forget this website and give it a glance on a constant basis.

  88. Hello my friend! I wish to say that this post is awesome, nice written and include approximately all important infos. I’d like to see more posts like this.

  89. Hello there, just became aware of your blog through Google, and found that it’s really informative. I’m going to watch out for brussels. I will be grateful if you continue this in future. Lots of people will be benefited from your writing. Cheers!

  90. After looking over a number of the articles on your blog, I really like your technique of blogging. I saved it to my bookmark webpage list and will be checking back soon. Please visit my web site too and tell me what you think.Also check out my latest Youtube video right here and please let me know what you think about it.

  91. Great – I should definitely pronounce, impressed with your website. I had no trouble navigating through all tabs as well as related info ended up being truly simple to do to access. I recently found what I hoped for before you know it at all. Quite unusual. Is likely to appreciate it for those who add forums or something, website theme . a tones way for your customer to communicate. Excellent task..

  92. For the reason that the YouTube movies are posted at this place same like I also embed YouTube video code at my own website, as it is easy to get embedded code.

  93. Hello there I am so happy I found your webpage, I really found you by accident, while I was looking on Aol for something else, Nonetheless
    I am here now and would just like to say kudos for a marvelous
    post and a all round entertaining blog (I also love the theme/design), I don’t have time to read it
    all at the minute but I have book-marked it and also added in your RSS
    feeds, so when I have time I will be back to read much more,
    Please do keep up the awesome b.

    Also visit my website …

  94. Excellent publish. I became verifying consistently this particular site with this particular prompted! Helpful details precisely one more cycle :) I look after these kinds of data a whole lot. I’d been looking for this particular data to get a really prolonged moment. Thanks a lot as well as involving chance.

  95. Experts are the dumbest people on planet. Always making unrealistic predictions and senseless assumptions. Understanding Apple, even a little kid might know that Apple doesn’t make cheap! Yet these guys create artificial increases and downgrades, just earning money somewhere I think!

  96. Hi there. Today using windows live messenger. It is deemed an highly well crafted document. I will you should definitely bookmark that and come back in find out more of the useful information. Basically posting. I will surely go back.

Leave a comment

No trackbacks yet.