Foresight 2010: the Synergy of Molecular Manufacturing and AGI
I will be speaking at Foresight 2010 this weekend in Palo Alto. My presentation, "Don't Fear the Singularity, but Be Careful: Friendly AI Design" will be both exciting and awesome. You can register here.
If you can't make it out to Palo Alto, the whole thing will be streamed live by TechZulu, which did the same for the recent H+ Summit. Here is their Alexa data for reference.
Hod Lipson of "computer program self-discovers the laws of physics" fame will be there, along with familiar faces and names such as Rob Freitas, Ralph Merkle, Robin Hanson, Paul Saffo, David Friedman, Brian Wang, and Monica Anderson. Salim Ismail, Executive Director of Singularity University, will speak late on Sunday.
Ten Things You Need to Stop Tweeting About
Here's the link. Twitter is the logical conclusion of the modern trend of shortening attention spans and Facebook-style addictive online socializing.
For a possible antidote, read Andrew Keen's Cult of the Amateur: How Today’s Internet is Killing Our Culture. Ben Goertzel's "The Global Nincompoop Awakens" is another informative piece.
Social media: always encouraging us to say less and less to more and more people.
End-of-Year Letter from Michael Vassar to Singularity Institute Supporters
I forgot to post this back on 11/24/09 when it was originally sent out to our mailing list.
Dear Friends of SIAI,
I’d like to share with you my picture of SIAI’s most recent year, of the global situation we are addressing, and of the work we can do with your help.
To put it bluntly, we are a small group of intelligent, ambitious, but as yet mostly inexperienced people who are working to increase the odds of an eventual positive Singularity and to decrease the odds of human extinction. Given our size and the size of the world, we’ve been reaching out to larger bodies of capable people. 2009 saw a number of successes in this regard: our Singularity Summit drew 800 attendees, including top academics who have remained in conversation about AI risks; Eliezer Yudkowsky finished a long-running series of posts laying out the concepts needed for thinking about AI risks, and started the successful online community Less Wrong; and we established a Visiting Fellows Program, which brought 14 researchers to Silicon Valley to do research and writing around AI risks and seeded ongoing focused full-time work.
Successes are useful in themselves. They are also indicators of what else can be achieved. In light of recent bridge-building to outside researchers and actors, we are fairly confident that we can turn even relatively small increases in donations into greater odds of an eventual positive singularity (basically: by continuing to do research and outreach in contact with capable people who may care). We wrote up many of the details of how we can do that on a new grant proposals site -- a site that lets you see exactly what proposed projects we’re thinking of, and how many dollars would let us do what to increase the odds of a positive long-run future.
If you care about humanity’s long-run future, and if you think SIAI is one of the more promising groups on the scene today, you’ll probably want to check out our proposed projects. Donate if you find a project worth funding or, if you don’t, share your thoughts on what else it is that would be worth funding. SIAI’s mission is ambitious but far from impossible; and spread across our staff and donor network we may have the seeds of a winning strategy.
Also, a Challenge Campaign is starting today, thanks to the generous support of Edwin Evans, Rolf Nelson, Henrik Jonsson, Jason Joachim, and Robert Lecnik. Until February 28, 2010, each dollar donated to SIAI will be matched on a 1:1 basis, up to $100,000, for a total of $200,000. In this context, especially, every donation counts.
Please take a look, and consider putting your dollars into the effort that, as far as I can tell, can turn your dollars into more good than any other today.
Best regards,
Michael Vassar,
President, SIAI
P.S.: I’m not kidding when I say I’d love to hear your thoughts. Do email me.
P.P.S.: We’re still taking applications for our Visiting Fellows program. If you’re interested in working directly with SIAI, check out our program description, and send in an application.
Bob Mottram Objections to 2010 Singularity Research Challenge and Response
Bob Mottram isn't impressed by the Singularity Institute's grant proposals for our $100,000 Singularity Research Challenge:
It's kind of sad how SIAI seems to have become obsessed with "AI risks" and human extinction. Perhaps they always were from the beginning, but it's just my perception of them that was at fault. There's certainly a place for some group, existing independently from academia, who actively promote AI related R&D in a direction which has positive value to society and addresses problems which are highly relevant. This applies especially to the work which is less glamorous, more ambitious stuff which requires an expenditure of effort on a longer time scale than a typical PhD thesis or DARPA/X-prize contest.
The list of grant proposals for the Singularity Research Challenge seems incredibly disappointing, and focused on spurious notions of risk which, in my opinion, would have no beneficial impact on AI even if it were to be funded in its entirety.
To clarify what is happening, what Bob Mottram considers "spurious notions of risk", we consider "deadly serious notions of risk", so this is the main source of disagreement. Here was my response:
Our Uncertain Future project is pioneering probabilistic futurism in AI and WBE studies, and has received thumbs-up from several academics including Bela Nagy, who manages the Santa Fe Institute Performance Curve Database.
A hard takeoff from a human-indifferent AI is not a fallacious risk. It is quite real. Because human moral values are complex, creating a machine that does what we would consider "nice" or "common sense" is much more difficult than creating a machine with human-level intelligence but insufficiently complex and specific values. See the Fun Theory sequence on Less Wrong, for instance.SIAI believes that AGI is an extremely difficult endeavor and deserves far more theory-level work than programming in the dark or working towards narrow AI tasks that drain away our attention at the expense of the Singularity itself.
Basically, if you consider an intelligence explosion plausible, SIAI's activities make sense, and if you don't, they don't. It's not a matter of marketing, just disagreement on which tasks are the most important for humanity to face right now. We consider clarifying decision theory and creating a reflective decision theory to be a major priority, for instance, and spend time on that accordingly.
To clarify further, in 2009 SIAI grew large enough to break into several loose divisions. This is excellent, because the Singularity Institute is one of the most important organizations on the planet and is one of the only barriers standing between humanity and extinction from unFriendly AI. However, it makes the task of explaining what we do all the more complicated. It so happens that I am paid to explain it, but sometimes I get discouraged because I discuss the organization constantly on this blog, occasionally several posts per day, and there is still a great amount of confusion about what our organization does and believes. Perhaps I ought to run an SIAI Video Q&A in the vein of Eliezer's recent Less Wrong Q&A.
What happened in the last year is that Anna Salamon and Steve Rayhawk joined us and created the Visiting Fellows program, under Anna's leadership. (Anna, Steve, and myself were only recently added to our staff page.) This entity is only peripherally related to SIAI's central AI project, which was more or less put on hold for two years while Eliezer Yudkowsky wrote the Less Wrong sequences. As our 2009 accomplishments document states, Eliezer worked with Marcello Herreshoff (his profile can be found here) on Friendly AI over the Summer.
So, think of SIAI as having three branches -- 1. Administrative/PR, which consists of President Michael Vassar, Media Director Michael Anissimov (aka me), and Chief Compliance Officer Amy Willey, 2. AGI research that constitutes serious progress towards seed AI, which makes up years of Eliezer's past work, Eliezer's future work after he finishes putting together his rationality book, Marcello Herreshoff's intermittent work, and contributions from Peter de Blanc, Nick Hay, and others (since 2006), including Anna Salamon and Steve Rayhawk, and 3. the Visiting Fellows Program, which including Visiting Fellows and various volunteers.
The goal of the Visiting Fellows Program is to put together extremely smart people concerned about reducing existential risk and have them pursue academic projects that make the best use of their respective strengths. Branch #3 also serves as a filtering mechanism for 2. The thing is, starting a true AGI project would be very expensive, not so much in money but in terms of intelligence, philosophy, computer science, and math knowledge required. Consolidating the necessary personnel will not be easy.
Why are we concerned about "AI risks" and human extinction? Well, this is why, among other reasons. SIAI is not about pursuing intermediate AI commercial benefits -- our organization only exists to pursue the Singularity and minimize AI risk. Writing illustrating this point has been produced in substantial quantities since our founding in 2000. SIAI is mostly a bunch of utilitarians.
Would readers be interested in a Lulu book putting together a lot of information about the Singularity Institute in one place? Only about 0.5% of my blog readers ever comment, so I feel like I'm talking to a vast sea of silent lurkers all the time. Seriously, it's weird.
In general, our approach turns off people like Bob Mottram, but inspires praise from people like Alan Darwst. In particular, Mr. Darwst writes:
Among the utilitarians I've met over the years, a sizable fraction have come to the conclusion that the optimal destination for utilitarian funding is organizations that research speculative futuristic scenarios and the philosophical / scientific / methodological questions that such research requires. In particular, many of these utilitarians have named the Singularity Institute for Artificial Intelligence (SIAI) as a good example of such an organization, so I'll focus on it here, but the discussion can apply more broadly.
The way the Singularity goes is a matter of life and death for humanity. Unfriendly AI programmed to value anything besides a very specific set of Homo sapiens-characteristic values will probably overlook our material preservation. From the perspective of most possible minds, humans are just another arrangement of atoms. We don't have any inherent moral value. "Moral value" is an "imaginary" thing that only exists among the tiny space of minds-in-general with explicit moral philosophies.
If we had the ability to build AGI today, our planet would not last the year, because we haven't solved Friendly AI. If we could build a seed AI now, we wouldn't know how to specify its goals in a way that doesn't eliminate us all completely. We are clueless. We can't create a utility function that is consistent under reflection and preserves individual humans when a tremendous amount of optimization pressure is applied to fulfilling it. We need a mathematical model of value that leaves us alive even when the unimaginable power of superintelligence is channeled into it. I think that Coherent Extrapolated Volition is a good enough solution that it would work, but it needs to be specified in much more detail. That's exactly what one of our grant proposals is about.
These grant proposals deserve funding now. We are about to walk into a minefield and we don't even have a map. We need to throw everything at the problem -- people, money, attention, everything.
Armin Krishnan on Killer Robots
Via the Moral Machines blog, Armin Krishnan, Visiting Professor for Security Studies at the University of Texas at El Paso and author of Killer Robots: Legality and Ethicality of Autonomous Robots was interviewed by Gerhard Dabringer.
In your recent book “Killer Robots: The Legality and Ethicality of Autonomous Weapons†you explore the ethical and legal challenges of the use of unmanned systems by the military. What would be your main findings?
The legal and ethical issues involved are very complex. I found that the existing legal and moral framework for war as defined by the laws of armed conflict and Just War Theory is utterly unprepared for dealing with many aspects of robotic warfare. I think it would be difficult to argue that robotic or autonomous weapons are already outlawed by international law. What does international law actually require? It requires that noncombatants are protected and that force is used proportionately and only directed against legitimate targets. Current autonomous weapons are not capable of generally distinguishing between legitimate and illegitimate targets, but does this mean that the technology could not be used discriminatively at all, or that the technology will not improve to an extent that it is as good or even better in deciding which targets to attack than a human? Obviously not. How flawless would the technology be required to work, anyway? Should we demand a hundred percent accuracy in targeting decisions, which would be absurd only looking at the most recent Western interventions in Kosovo, Afghanistan and Iraq, where large numbers of civilians died as a result of bad human decisions and flawed conventional weapons that are perfectly legal. Could not weapons that are more precise and intelligent than present ones represent a progress in terms of humanizing war?
I don’t think that there is at the moment any serious legal barrier for armed forces to introduce robotic weapons, even weapons that are highly automated and capable of making own targeting decisions. It would depend on the particular case when they are used to determine whether this particular use violated international law, or not. The development and possession of autonomous weapons is clearly not in principle illegal and more than 40 states are developing such weapons, indicating some confidence that legal issues and concerns could be resolved in some way. More interesting are ethical questions that go beyond the formal legality. For sure, legality is important, but it is not everything. Many things or behaviors that are legal are certainly not ethical. So one could ask, if autonomous weapons can be legal would it also be ethical to use them in war, even if they were better at making targeting decisions than humans? While the legal debate on military robotics focuses mostly on existing or likely future technological capabilities, the ethical debate should focus on a very different issue, namely the question of fairness and ethical appropriateness. I am aware that “fairness†is not a requirement of the laws of armed conflict and it may seem odd to bring up that point at all. Political and military decision-makers who are primarily concerned about protecting the lives of soldiers they are responsible for clearly do not want a fair fight. This is a completely different matter for the soldiers who are tasked with fighting wars and who have to take lives when necessary. Unless somebody is a psychopath, killing without risk is psychologically very difficult. Teleoperators of the armed Predator UAVs actually seem to suffer from higher levels of stress than jet pilots who fly combat missions. Remote controlling or rather supervising robotic weapons is not a job well suited for humans or a job soldiers would particularly like to do. So why not just leave tactical targeting decisions to an automated system (provided it is reliable enough) and avoid this psychological problem? This brings the problem of emotional disengagement from what is happening on the battlefield and the problem of moral responsibility, which I think is not the same as legal responsibility. Autonomous weapons are devices rather than tools. They are placed on the battlefield and do whatever they are supposed to do (if we are lucky). The soldiers who deploy these weapons are reduced to the role of managers of violence, who will find it difficult to ascribe individual moral responsibility to what these devices do on the battlefield. Even if the devices function perfectly and only kill combatants and only attack legitimate targets, we will not feel ethically very comfortable if the result is a one-sided massacre. Any attack by autonomous weapons that results in death could look like a massacre and ethically difficult to justify, even if the target somehow deserved it. No doubt, it will be ethically very challenging to find acceptable roles and missions for military robots, especially for the more autonomous ones. In the worst case, warfare could indeed develop into something in which humans only figure as targets and victims and not as fighters and deciders. In the best case, military robotics could limit violence and fewer people will have to suffer from war and its consequences. In the long term, the use of robots and robotic devices by the military and society will most likely force us to rethink our relationship with the technology we use to achieve our ends. Robots are not ordinary tools, but they have the potential for exhibiting genuine agency and intelligence. At some point soon, society will need to consider the question of what are ethically acceptable uses of robots. Though “robot rights†still look like a fantasy, soldiers and other people working with robots are already responding emotionally to these machines. They bond with them and they sometimes attribute to the robots the ability to suffer. There could be surprising ethical implications and consequences for military uses of robots.
You can read the rest here.
PhysOrg: TrueCompanion takes wraps off robot girlfriend
From PhysOrg, news about the world's first sex robot "complete with artificial intelligence and flesh-like synthetic skin".
When it comes to news like this, simpletons just laugh (yes, it's funny, but this is obvious), while more sophisticated analysts carefully consider the near and long-term implications and trends.
Humanity Plus, IEET, and Other Transhumanist Community Stuff
My current boss and long-time associate and conversation-partner, Michael Vassar, is running for the board of the organization Humanity Plus. In a recent phone call telling me about his decision, he said that he wanted to make transhumanism more effective by encouraging better cooperation among its many organizations and groups, and spending more time pursuing brand clarification. Read his candidate statement and consider voting for him.
Recently, a bunch of people stepped down from the board of the organization, namely Nick Bostrom, James Hughes, and Mike Treder. James Hughes was a strong figure in the organization, formerly known as the World Transhumanist Association, for many years. For those not in the know, Nick, James, and Mike are now part of the IEET, which is more of a formal think tank than Humanity Plus, the organization formerly known as WTA.
I am a member of Humanity Plus. I used to encourage the IEET to reproduce select blog posts of mine as it wished, but then in mid-year I politely asked them to cease doing so in response to Managing Director Mike Treder's frequent attacks on my moral philosophy of choice, Singularitarianism. See Roko's coverage. Mr. Treder compared Singularitarianism to "birtherism, creationism, and climate science denialism". He wrote, " In each case, arguments are marshaled that seem to resemble scientific or legal reasoning but that end up as speculative assertions intended to support fanciful, ideological, or faith-based positions."
I invite you to read documents like "Basic AI Drives" by Stephen Omohundro and decide for yourself whether the arguments seem primarily faith-based or scientific.
Now I've gotten into conflict already, but that's not my intention. I just wanted to review and mention the IEET. Even though my articles don't appear there any more and I no longer mention them in my bio, I think that IEET could be worth supporting if you can't bring yourself to support SIAI. As a center-left type guy (leaning center-right on economics), I find much of Treder's writing to be stereotypical SWPL leftism, but there are many other authors on the site. James Hughes recently had a good piece up that directly addresses Less Wrong, Eliezer Yudkowsky's rationality project, and the like. The IEET, in general, seems to be making some inroads to the mainstream media, and is indeed the public face of "democratic transhumanism".
Humanity Plus is more of a blank slate. They had a successful recent conference, and could still go a number of different directions. This board runoff will help determine which. Join the organization now if you wish to vote next week.
Our Friend Gadolinium
Brian Wang directs our attention to one important part of Rob Freitas' radionuclide page:
The mass of the alpha-particle is ~7000 times greater than that of an electron, so the velocity and hence the range of a-particles in matter is considerably less than for beta-particles of equal energy. Consequently the optimum radionuclide for medical nanorobots is predominantly an alpha emitter.
Among all gamma-free alpha-only emitters with t1/2 > 106 sec, the highest volumetric power density is available using Gd148 (gadolinium) which a-decays directly to Sm144 (samarium), a stable rare-earth isotope. A solid sphere of pure Gd148 (~7900 kg/m3) of radius r = 95 microns surrounded by a 5-micron thick platinum shield (total device radius R = 100 microns) and a thin polished silver coating of emissivity er = 0.02 suspended in vacuo would initially maintain a constant temperature T2 (far from a surface held at T1 = 310 K)
75-year half-life, initially generating 17 microwatts of thermal power which can be converted to 8 microwatts of mechanical power by a Stirling engine operating at ~50% efficiency. (Smaller spheres of Gd148 run cooler.) While probably too large for most individual nanorobot designs, such spheres could be an ideal long-term energy source for a swallowable or implantable "power pill" (Chapter 26) or dedicated energy organ (Section 6.4.4). A ~0.2 kg block of pure Gd148 (~1 inch3) initially yields ~120 watts, sufficient in theory to meet the complete basal power needs of an entire human body for ~1 century (given suitable nucleochemical energy conversion and load buffering mechanisms, and a sufficiently well-divided structure).
The last part is the punchline, of course. Freitas acknowledges future design challenges such as energy conversion, load buffering, and division of structure. If these challenges are overcome, a large block of Gd148 (or simply gadolinite ready to be processed into pure gadolinium) could supply nutrition to millions of people for millennia. Gadolinium has a half-life of 75 years, so you'd need double as much for each 75-year period you wish to avoiding refueling for, but storing gadolinium in its stable gadolinite form seems avoid this problem. Unfortunately, gadolinite is fairly rare and gadolinium itself is only found in the Earth's crust at a 6.2 ppm level. By comparison, the abundance of gold in the Earth's crust is only 0.0011 ppm. According to this page, annual production of gadolinium is 200 tons.
Just to throw some numbers out there, if one cubic inch is enough per person per century, a million people would require a million cubic inches. That can fit in a cube 9 x 9 x 9 ft large. According to Freitas' numbers, this would weigh about 200,000 kg, or 200 metric tonnes, which is on par with today's annual production. If demand for gadolinium grew, it seems plausible that its cost would fall greatly -- after all, gold is about 6,000 times rarer and our annual production is 2,800 tons. Feeding ten billion people with gadolinium, if that were possible, would require about 2,000,000 metric tonnes for the first century. At an extraction rate of 200,000 metric tonnes per year, it could be done in a decade. This would require increasing current production by a factor of 1,000. According to this book, gadolinite can contain 40% rare earth oxides, 5% of which consists of gadolinium itself. That means that gadolinium makes up about 2% of the total. (Wrong: see comments.) Processing ten million metric tonnes of the ore annually would yield the required amount. For comparison, we extract 1.2 billion tons of iron from the Earth's crust annually.
Update: all of the above is wrong for one reason or another, as pointed out in the comments, but at least I had fun. I was confusing chemical stability with nuclear stability and made the mistake that I thought gadolinium-148 would be nuclear-stable in its gadolinite form, which is wrong. The atomic number of gadolinium is 64 meaning that gadolinium-148 contains 20 extra neutrons above neutron-proton parity. It seems to me that we'd eventually have to find a less safe and cheaper isotope to make this work on a large level if it's suitable in practice and we ever want to.
Radioisotopic Food Nanobots: Freitas Response
I recently wrote to Rob Freitas about his radioisotope-powered food nanorobot idea that, if it works, could allow people to eat at severely reduced levels for as long as a century or more. As far as I can tell, food would still be needed due to cell loss from shedding skin cells and the like, but this would likely be relatively little. As Roko pointed out, the gadolinium-powered nanobots could reconstitute ATP from waste products like urea. The gadolinium would just provide the energy for running the chemical reactions needed to produce fresh ATP.
Here is the email I wrote to Rob Freitas:
Hi Robert, I saw an idea of yours posted at the World Future Society, and blogged it. Me and my readers weren't clear on some of the details, and a few google searches turned up nothing. All of us would appreciate if you would weigh in on the thread and answer our burning questions.
Thanks, and I'm always impressed by all the ideas you come up with.
Best,
Michael
Here is the response (posted with permission):
Hi, Michael.
The 148Gd power source proposal was described in NMI (1999) at http://www.nanomedicine.com/NMI/6.3.7.1.htm. The semiconductor shell structure crudely illustrated in Fig. 6.7 is intended to be an atomically precise structure. The radioactive 148Gd is kept permanently encapsulated while inside the body. The minimum radius for this powerplant is on the order of ~11 microns, so it is clearly intended for fixed-site multi-nodal (not bloodborne) use.
I haven't yet published any detailed scaling studies specifically describing dietary-related nanorobotic systems. These proposals now exist only in rough form in my long (across 2 decades!) accumulated notes for Chapter 26 in Vol. III of my Nanomedicine book series. I hope to find time to publish NMIII sometime in this decade.Best wishes,
Rob Freitas
I read the page that Freitas linked. Here's one of the core specs:
A (1 micron)3 cube of Gd148 produces ~5 a-particles/sec, yielding an output current of ~1 picoampere at ~3 volts (e.g., ~3 pW).
Interesting! The page also points out that the cost of Gd148 must be brought down significantly before it becomes a feasible power source, because in 1998 it cost about a dollar per two cubic microns(!) This is expensive stuff. The number of nanobots that might be used would need to be on the order of a hundred trillion (not a billion, as I wrote previously), each with a cubic micron-sized power core, though 11 microns across due to shielding. Given the 1998 cost of Gd148, a full system would cost about $50 trillion for the fuel alone! Near the top of the page it says, "Selection of an optimum radioactive fuel is guided primarily by safety criteria".
An interesting idea, and food for nanotechnological thought.
Nuclear Threat Initiative 2009 Accomplishments
Here is the link. 2009 was a good year for nuclear threat mitigation due to the direct support of President Obama. Before, Bush was paying lip service to containing the nuclear threat but wasn't really doing anything about it. Here's a piece of news worth repeating:
Early next year, NTI will release “Nuclear Tipping Point,†a documentary film featuring former Secretaries Shultz, Kissinger, and Perry and Senator Nunn as they share the personal experiences that led them to create the Nuclear Security Project. Introduced by General Colin Powell, the film is narrated by actor Michael Douglas and includes interviews with former Soviet President Mikhail Gorbachev and California Governor Arnold Schwarzenegger.
Read the rest. The NTI isn't only focused on the nuclear risk -- it also takes actions to contain chemical and biological weapons as well.