Funding the Mitigation of Extinction Risks Wednesday, Sep 17 2008
risks 11:18 pm
Insufficient resources are currently devoted to the mitigation of extinction risks. This is the argument of organizations such as the Lifeboat Foundation, the Singularity Institute, and the Oxford Future of Humanity Institute. In the web edition of the Bulletin of the Atomic Scientists, Sandberg, Matheny, and Cirkovic ask, “How can we reduce the risk of human extinction?” They offer many valuable suggestions, including expansion into space, developing secure bunkers and interdisciplinary research in quantitative risk assessment, probability theory, and technological forecasting. I’m frequently engaged in the latter, most recently by participating in a summer project funded by a grant from SIAI, and in an ongoing basis by encouraging donations to the Lifeboat Foundation.
Fundraising progress has occurred but is largely disappointing. The Future of Humanity Institute does not publish its annual budget, but judging by the staff, I’d guess their budget is about $600,000 a year, funded by James Martin. According to guidestar.org, the Singularity Institute’s revenue in 2006 was about $460,000, and the Lifeboat Foundation’s was far less, about $10,000. All these organizations could use a lot more funding, in the tens of millions.
Why is the situation so poor? My guess is not lack of wealth or will to address the problem, but lack of ideas for low-hanging fruit to pick that helps the cause. If we had better actionable ideas, individuals and foundations would be more inclined to step forward to fund them. Of course, there is also the problem of the “silly factor” regarding extinction risks, and the vagueness of tangibility of such a broad venture.
More reasons: prevention of extinction risk is not a positive goal, it’s a negative goal. People prefer to fund positive goals. Another possible reason is that many philanthropists are on the older side, and inclined to worry more about their own demise than the hypothetical extinction of the human race.
Another challenge is that the most prominent scientists that fear human extinction, Martin Rees and Stephen Hawking, haven’t done enough to address the problem. Hawking encourages development of space, but what about other measures? Little activity there. Same with Sir Rees. After a blast of activity around his recent book, little else follows. An absence of specific initiatives and public proclamations of support. Little of the entrepreneurial spirit we see in business.
My preferred initiative to counteract extinction risk is Friendly AI research, but I know that this idea is not popular to everyone in the risk mitigation community. How about research into self-sufficient closed systems, human intelligence enhancement, building safeguards into gene synthesis equipment, or pursuing other avenues that Sandberg et al present in their recent article? Perhaps a prominent scientist needs to pick one of these ideas and ask their wealthy friends to fund them. After all, it’s only our future.
September 18th, 2008 at 6:29 am
Perhaps those focused on the “mitigation of extinction risks” could engage in coordinated lobbying of governments. They could pick one objective that wouldn’t cost governments more than, say, $50 million and hire a lobbyist to help achieve the objective.
September 18th, 2008 at 2:55 pm
Lack of ideas for low-hanging fruit is an important limit to my willingness to contribute.
Asteroid detection deserves a bit more funding. Seasteading should provide some resilience against climate change (especially a sudden drop in temperature due to a large asteroid or volcano) since marine aquaculture should be less weather sensitive than land based crops, but starting seastead-based aquaculture isn’t cheap.
If I saw clear signs that there was a good way to spend money on reducing AGI risks, I’d devote a fair amount of money to that. But I’m sufficiently unimpressed by the opportunities I currently see that I think it’s better to save my money and hope to find a better choice later.
September 19th, 2008 at 9:55 am
It occurs to me that one early way of simplifying the problem of Friendliness is to engineer towards a goal of increased absolute utility while maintaining the relative utility of all individuals and groups within the human race.
That is to say, give the AGI the goal of increasing the utility of every individual person, but do so in a way that causes each and every person to maintain the same relative utility compared to other people. This might seem extremely un-egalitarian, but do keep in mind that it would allow individuals to modulate their relative utility compared to others, so long as this had no impetus from the AGI.
Just a thought.
September 19th, 2008 at 5:16 pm
Ian, thanks for the thought, here’s my opinion.
We don’t want to give the AGI a fixed goal of *anything*. We want to create an AGI that computes an extrapolation of human values and then creates an arbitrary object that embodies those values. The initial AGI goes away right after it creates that embodiment.
In the old version, the AGI was never given any fixed goal, but an evolving set of attractors that simulated human moral progress. Never at any point was there the idea to give the AGI a fixed set of goals or commandments, which seems to be what you’re getting at here.
Can you at least read CFAI and integrate the insights from that into your FAI ideas?
September 19th, 2008 at 5:52 pm
I’ve looked over the CFAI document on more than one occasion, at varying levels of depth of attention-span.
I was just thinking in text, basically — there wasn’t anything solid to the thought. It just seemed to me that one of the major problems in attempting to create a Friendly AGI is defining what Friendliness is. And, of course, that’s going to be a relatively fixed answer — or at least, our ability to define it for a superintelligence is going to be somewhat limited.
The one thing that is guaranteed with the concept I was looking at before, I think, is that it will not result in the destruction of the human race, nor would it abrogate the individual rights of any individual person. Which, of course, is important. I mean, what if the extrapolation of human values turns out that we don’t value existing? Or something particularly screwy along those lines as well. Meanwhile, the concept I discussed would at the very least guarantee that we could /benefit/ from even a so-limited sAGI; and it would even be possible to benefit sufficiently from it that a second attempt at creating a recursively improving superintelligence with another reliably ’safe’ goal architecture could be created.
Another thing to look at, I suspect, is just how fixed in definition “utility” really is. That’s got to be the biggest flaw in my concept. (You’ll notice I’m being unusually “hedgy” with my ‘bets’ here.) I know better than to think I’m a qualified theoretician here. What I do know is that *I* cannot see how a fixed-relative-utility goal for the effects of a seed AGI would result in the destruction of the human race. It would, of course, be free to develop and refine any of its other goals — including how it evaluates and implements the “core” goal, of course — as it would very assuredly be better at this than we would be.
And, of course, if nothing else — it could always take an “Asimovian” approach and create a ’second generation’ entity which fails to share its “goal-restricted” qualities. (Sorry to include fiction here, but it’s apt in this case!)
September 21st, 2008 at 10:27 am
Michael: “My preferred initiative to counteract extinction risk is Friendly AI research,”
- that’s interesting: since I’m now doing an AI master’s, I guess I can wake up every morning and say that I’m going to work to save the world?
On a serious note: what drew you to this conclusion? My own view is that we are in a state of ignorance as to what the fastest route to a smarter than human intelligence will be, so I’d advocate betting our resources pretty much evenly on all of the plausible avenues: AI, IA, uploading. Having read Hanson’s material on uploading, I seriously worry about the latter possibility.