Accelerating Future

It’s called the Nuclear Tipping Point, and I wish they’d market it more aggressively to youngsters as well as the older set, but unfortunately their marketing crew is just too damn old. I mean no disrespect to older folks (after all, I plan to live for hundreds of thousands of years), it’s just that younger folks seem to get more rallied up and enthusiastic over causes, and we need that here. I don’t think that a grim academic perspective is very useful for real action either. Important causes start in academia, but can’t stay there.

The last movie this laudable organization put out was called Last Best Chance. It’s funny because it’s overly self-serious and Fred Thompson is President in it.

This latest movie has even more famous old men in it than the last one, and it’s totally free. Famous old men featured include Colin Powell, Arnie, and everyone’s favorite controversial-as-hell Secretary of State, Mr. Realpolitik himself, Henry.

It’s sad how the people who invented the nuclear bomb and spent their careers dealing with the threat of it are now screaming about the risk of terrorist nuclear weapons, and no one under the age of 40 is listening. Few people over 40 are listening either, but the numbers seem better there. (Obama, most notably.) Perhaps it will take a nuke going off in one of our major cities before people wake up. There’s this thing called a boat that lets you bring a payload right up to the coast without too much trouble.

My generation is too interested in webcomics, MMOs, perpetual left-right political warfare, and gosh-wow technologies to care about the real risks right in front of us.

Here’s Colin Powell.

Here’s the blurb:

The film is introduced by General Colin Powell, narrated by Michael Douglas and includes interviews with California Governor Arnold Schwarzenegger and former Soviet President Mikhail Gorbachev. “Nuclear Tipping Point” was written and directed by Ben Goddard and produced by the Nuclear Security Project in an effort to raise awareness about nuclear threats and to help build support for the urgent actions needed to reduce nuclear dangers.

For those concerned about existential risks, try to first see if you can get anyone interested in plausible non-existential risks. That can set a baseline for the level of success you expect to achieve for existential, longer-term threats like AGI.

4 Responses »

The chatter in the background dies down after a couple minutes.

“The people at Oak Ridge were frustrated. They were very fine scientists. They did this work… they tested all these procedures against real nuclear weapons, but the American people weren’t getting any defense.”

Here is the file. Unfortunately, the image scans are crap.

One Response »

From yesterday’s list of links, I particularly want to call attention to the rotifer link. This press release is interesting because it shows how animals can survive even when they are exact genetic copies of one another. Instead of outcompeting parasites through mutation, they run away by going into cryptobiosis. I predict that a form of asexual multicellular synthetic life will be created by 2030 that can defend against parasites through aggressive defense, say silica spines, so that running away isn’t even necessary. These organisms will just sit around and reproduce. The primary method to get rid of them at first will be dessication, but this will eventually prove useless as they disperse too widely to target.

What many humans don’t realize is that we are surrounded by quintillions of organisms with very little genetic diversity that dominate us in terms of biomass and persistence. They are the status quo — we are the aberration. These are organisms that have survived every mass extinction. Culprits include the tardigrades (which can survive outer space), nematodes (absolutely ubiquitous; it is estimated there are between 10¹⁸ (one quintillion) and 10²¹ (one sextillion) nematodes worldwide, and they are crawling all over you right now), chaetognaths (considered useful models of basal bilaterans, there are a lot of them in the oceans, really a lot), and so on.

The only reason that these organisms aren’t ripping us all to shreds right now is because there have been no synthetic biologists to push them out of evolutionary minima and give them more sensible strategies for total domination. Sorry to be alarmist, but I studied evolutionary biology for a couple years and that is my opinion. Evolution is terribly poor at transversing local minima to reach a global optima, and that is really the only saving grace for fragile macroscale multicellular agglomerations like ourselves. Interesting and low-energy-cost evolutionary innovations are rarely combined because they require several working parts to come together which are maladaptive individually but adaptive in cooperation.

The reason why rotifers are interesting is that their lack of genetic diversity makes them a good model for self-replicating machines. The ability to switch into a dormant, armored state (cryptobiosis) seems characteristic of a variety of small organisms, and we can expect this ability to be exploited to the fullest by human-engineered microscale replicators. The ability to distribute many of these replicators across a wide area will eventually create a “viral load” scenario analogous to the one faced by aging humans — so many diverse beings build up in our body that the workload faced the immune system to combat nascent infections eventually becomes prohibitive and the system breaks down.

Some scientists have laughed at the idea that human-engineered organisms could dominate microbes that have evolved for billions of years, but I find this ridiculous. Human-engineered artifacts have already outperformed everything created by evolution in terms of energy density, speed, mass, acceleration, local dominance, and so on. The key point is that evolution is radically dumb (but it has many trials available) and humans are very smart. Let’s discuss some of the ways to engineer microorganisms that cannot be defeated by the legacy biota.

1. Broad-spectrum biocides: natural organisms use a variety of biocides, but observe that humans have created thousands of highly effective synthetic antibiotics and biocides that evolution never discovered even after four billion years of experimentation.

2. Phage-immune bacteria, for instance bacteria that use genetic programs incompatible with malicious code injection by phages. Phages are the main bacteria-curtailing force on the planet and we depend on them for our survival.

3. Bacteria specifically engineered for immunity to broad-spectrum antibiotics which produce and secrete these antibiotics as a biofilm. There is even the possibility of release-and-shield, where microbes release the biocide then shield themselves from it for long enough for the competitors to be defeated, at which point the shield is raised.

4. Sucking them in: microorganisms could coat themselves in a gel shield which absorbs and dissolves both nutrients, phages, and rival microbes. For instance, the extracellular matrix of animal tissues is much stronger than the slime layer used by bacteria. Cooperative colonial bacteria could create stronger extracellular shields depending on how well-established the colonial region is, devoting stronger shields to the colonial center and weaker shields to the exploratory fringes.

5. Incubation-then-release: many evolutionary minima involve colonial organisms that are evolutionarily strong in larger colonies but evolutionarily weak in small colonies. By sterilizing a large area, filling it with nutrients, and allowing a founder population to develop (a “mega petri dish”), an important evolutionary minima could be hopped.

6. Quorum computing: evolution has developed a variety of means for microbes to communicate with one another on a crude level: quorum sensing. One of the interesting evolutionary innovations of the last billion years was to produce multicellular organisms that survive against many uncooperative microbes. By creating microbial superorganisms that effectively cooperate and compute using biocomputation, it may be possible to beat multicellular life at its own game by creating “organisms” miles across that effectively cooperate to defeat all rivals. This is definitely not a near-term risk but it could be a risk within the lifetimes of many alive today, given no singleton that guards us at a low level.

7. The last point in particular opens up a very large space for experimentation. For a colony that knows how to differentiate its perimeter members from interior members, it can activate all sorts of interesting genes in the perimeter members to make life miserable for organisms next to them. Bacteria already do this in a rudimentary way with quorum sensing. As long as a suitable barrier can be erected, the production of a variety of poisons is possible and safe for the majority of the colony.

Even natural selection in hospitals is enough to create killer bacteria immune to many antibiotics. What about bacteria specifically engineered by smart humans for reproduction and survival?

5 Responses »

Via the Moral Machines blog, Armin Krishnan, Visiting Professor for Security Studies at the University of Texas at El Paso and author of Killer Robots: Legality and Ethicality of Autonomous Robots was interviewed by Gerhard Dabringer.

In your recent book “Killer Robots: The Legality and Ethicality of Autonomous Weapons” you explore the ethical and legal challenges of the use of unmanned systems by the military. What would be your main findings?

The legal and ethical issues involved are very complex. I found that the existing legal and moral framework for war as defined by the laws of armed conflict and Just War Theory is utterly unprepared for dealing with many aspects of robotic warfare. I think it would be difficult to argue that robotic or autonomous weapons are already outlawed by international law. What does international law actually require? It requires that noncombatants are protected and that force is used proportionately and only directed against legitimate targets. Current autonomous weapons are not capable of generally distinguishing between legitimate and illegitimate targets, but does this mean that the technology could not be used discriminatively at all, or that the technology will not improve to an extent that it is as good or even better in deciding which targets to attack than a human? Obviously not. How flawless would the technology be required to work, anyway? Should we demand a hundred percent accuracy in targeting decisions, which would be absurd only looking at the most recent Western interventions in Kosovo, Afghanistan and Iraq, where large numbers of civilians died as a result of bad human decisions and flawed conventional weapons that are perfectly legal. Could not weapons that are more precise and intelligent than present ones represent a progress in terms of humanizing war?

I don’t think that there is at the moment any serious legal barrier for armed forces to introduce robotic weapons, even weapons that are highly automated and capable of making own targeting decisions. It would depend on the particular case when they are used to determine whether this particular use violated international law, or not. The development and possession of autonomous weapons is clearly not in principle illegal and more than 40 states are developing such weapons, indicating some confidence that legal issues and concerns could be resolved in some way. More interesting are ethical questions that go beyond the formal legality. For sure, legality is important, but it is not everything. Many things or behaviors that are legal are certainly not ethical. So one could ask, if autonomous weapons can be legal would it also be ethical to use them in war, even if they were better at making targeting decisions than humans? While the legal debate on military robotics focuses mostly on existing or likely future technological capabilities, the ethical debate should focus on a very different issue, namely the question of fairness and ethical appropriateness. I am aware that “fairness” is not a requirement of the laws of armed conflict and it may seem odd to bring up that point at all. Political and military decision-makers who are primarily concerned about protecting the lives of soldiers they are responsible for clearly do not want a fair fight. This is a completely different matter for the soldiers who are tasked with fighting wars and who have to take lives when necessary. Unless somebody is a psychopath, killing without risk is psychologically very difficult. Teleoperators of the armed Predator UAVs actually seem to suffer from higher levels of stress than jet pilots who fly combat missions. Remote controlling or rather supervising robotic weapons is not a job well suited for humans or a job soldiers would particularly like to do. So why not just leave tactical targeting decisions to an automated system (provided it is reliable enough) and avoid this psychological problem? This brings the problem of emotional disengagement from what is happening on the battlefield and the problem of moral responsibility, which I think is not the same as legal responsibility. Autonomous weapons are devices rather than tools. They are placed on the battlefield and do whatever they are supposed to do (if we are lucky). The soldiers who deploy these weapons are reduced to the role of managers of violence, who will find it difficult to ascribe individual moral responsibility to what these devices do on the battlefield. Even if the devices function perfectly and only kill combatants and only attack legitimate targets, we will not feel ethically very comfortable if the result is a one-sided massacre. Any attack by autonomous weapons that results in death could look like a massacre and ethically difficult to justify, even if the target somehow deserved it. No doubt, it will be ethically very challenging to find acceptable roles and missions for military robots, especially for the more autonomous ones. In the worst case, warfare could indeed develop into something in which humans only figure as targets and victims and not as fighters and deciders. In the best case, military robotics could limit violence and fewer people will have to suffer from war and its consequences. In the long term, the use of robots and robotic devices by the military and society will most likely force us to rethink our relationship with the technology we use to achieve our ends. Robots are not ordinary tools, but they have the potential for exhibiting genuine agency and intelligence. At some point soon, society will need to consider the question of what are ethically acceptable uses of robots. Though “robot rights” still look like a fantasy, soldiers and other people working with robots are already responding emotionally to these machines. They bond with them and they sometimes attribute to the robots the ability to suffer. There could be surprising ethical implications and consequences for military uses of robots.

You can read the rest here.

One Response »

Here is the link. 2009 was a good year for nuclear threat mitigation due to the direct support of President Obama. Before, Bush was paying lip service to containing the nuclear threat but wasn’t really doing anything about it. Here’s a piece of news worth repeating:

Early next year, NTI will release “Nuclear Tipping Point,” a documentary film featuring former Secretaries Shultz, Kissinger, and Perry and Senator Nunn as they share the personal experiences that led them to create the Nuclear Security Project. Introduced by General Colin Powell, the film is narrated by actor Michael Douglas and includes interviews with former Soviet President Mikhail Gorbachev and California Governor Arnold Schwarzenegger.

Read the rest. The NTI isn’t only focused on the nuclear risk — it also takes actions to contain chemical and biological weapons as well.

No Responses »

I just found this by Googling “existential risk”, but I’ve heard of this project for a little while already…

Existential Risk Reduction Career Network

Here is the text on the front page:

This is a career network for those interested in reducing existential risks, an existential risk being one of several events and trends that would destroy or permanently handicap all of humanity. A number of such dangerous possibilities have been with us throughout our history, primarily those of exceptionally powerful natural disasters. Obviously, no such disaster has yet occurred, though a monumental volcanic eruption approximately 70,000 years ago may have come perilously close. More pressing are the dangers brought about by our increasing technological prowess. While technological advancement brings widespread improvements in our quality of life, it also continually enhances our capabilities for destruction. If and when such capabilities become sufficiently powerful, it is imperative that they be handled responsibly. The great danger of existential risk is that one mistake is too many, and there are no second chances.

The concept of existential risk is a relatively new one, and a variety of dangers are just recently becoming apparent. The impact of any individual to existential risk reduction can be very significant, especially at these early stages where the amount of work done in this area is still drastically less than justified by the universal importance of the subject. This network is for those interested in donating substantial amounts (relative to income) to non-profit organizations focused on the reduction of existential risk. If you have such an interest, feel free to contact us at contact(at)xrisknetwork.com.

One Response »

In the context of our 2010 Singularity Research Challenge, Steven over at Black Belt Bayesian has a collection of “reasons to invest in reducing existential risk that you might not have considered before”.

No Responses »

H/t to Joshua Fox for the link.

I have been watching some of Tim’s videos over the last few months, but I definitely haven’t seen them all. This one is nice because it summarizes a poignant feeling of concern.

In this video, he builds a model of AGI development using construction paper and Post-It notes.

4 Responses »

In February 2009, the President of the American Association for Artificial Intelligence, Eric Horvitz, convened a panel on “long-term AI futures” which explicitly delved into issues around the Singularity and intelligence explosion. Horvitz has told me (and the New York Times) that the reason he convened the panel was not due to personal interest or concern in the issue but in response to the public interest and concern in the issue.

In the New York Times article covering the meeting, Horvitz was quoted as saying, “My sense was that sooner or later we would have to make some sort of statement or assessment, given the rising voice of the technorati and people very concerned about the rise of intelligent machines”. In August, they released an interim report that said:

Popular perspectives on the outcomes of AI research include expectation that there will be one or more disruptive outcomes. These include that notion that the research will somehow lead to the advent of utopia or catastrophe. The utopian perspective is perhaps best captured in the writings of Ray Kurzweil and others, who speak of a forthcoming “technological singularity.” At the other end of the spectrum, some people are concerned about the “rise of intelligent machines,” fueled by popular novels and movies, that tell stories of the loss of control of robots. Whether forecasting utopian or catastrophic outcomes, the radical perspectives are frightening to people in that they highlight some form of radical change on the horizon—often founded on a notion of the loss of control of the computational intelligences that we create.

The panel of experts was overall skeptical of the radical views expressed by futurists and science-fiction authors.

To me, this was a disappointing result. The phrasing is also disappointing. It is not just the opinion of “popular perspectives” that AI will “somehow” lead to the advent of utopia or catastrophe. Many academics (including AI researchers) have presented views that AI would be highly disruptive, including Ray Solomonoff, Nick Bostrom, Shane Legg, Matt Mahoney, I.J. Good, Bill Gates, Hans Moravec, Marvin Minsky, and many others. Solomonoff, Moravec, and Minsky have all been leaders in AI for decades, so it seems like a deliberate choice of focus to attribute “radical views” to the public rather than AI experts. It provides the AAAS panel with a comfortable level of removal from the claims, a level of removal they could not easily obtain if they cited Solomonoff, Moravec, and Minsky as the sources of Singularity views.

It is remarkable for the panel to suggest that AI will probably not result in disruptive outcomes — if you can turn a pile of sand into a thinking intelligence in the time it takes you to fabricate a computer chip and transfer files to it, then that wouldn’t be disruptive? In my view, it is the degree of disruption that is up for debate — I don’t take people very seriously if they imply there will be little or no disruption whatsoever.

In wondering why the panel came up with this result, Eliezer Yudkowsky suggested “snap consideration and snap judgment”. However, Steve Rayhawk offered a more detailed analysis, which I will post in its entirety here, with a few formatting changes to ensure successful reposting. The first two sentences are a quote that Rayhawk is responding to. Everything that follows from this point on (except for the last line and the quote) was posted by Steve Rayhawk to Less Wrong.

Roughly, what I expect to happen by default is no modular analysis at all - just snap consideration and snap judgment. I feel little need to explain such.

You, or somebody anyway, could still offer a modular causal model of that snap consideration and snap judgment. For example:

1. What cached models of the planning abilities of future machine intelligences did the academics have available when they made the snap judgment?
1.1 What fraction of the academics are aware of any current published AI architectures which could reliably reason over plans at the level of abstraction of “implement a proxy intelligence”?
1.1.1 What fraction of them have thought carefully about when there might be future practical AI architectures that could do this?
1.1.2 What fraction use a process for answering questions about the category distinctions that will be known in the future, which uses as an unconscious default the category distinctions known in the present?

2. What false claims have been made about AI in the past? What decision rules might academics have learned to use, to protect themselves from losing prestige for being associated with false claims like those?
2.1 How much do those decision rules refer to modular causal analyses of the object of a claim and of the fact that people are making the claim?
2.2 How much do those decision rules refer to intuitions about other peoples’ states of mind and social category memberships?
2.3 How much do those decision rules refer to intuitions about other peoples’ intuitive decision rules?
2.4 Historically, have peoples’ own abilities to do modular causal analyses been good enough to make them reliably safe from losing prestige by being associated with false claims? What fraction of academics have the intuitive impression that their own ability to do analysis isn’t good enough to make them reliably safe from losing prestige by association with a false claim, so that they can only be safe if they use intuitions about the states of mind and social category memberships of a claim’s proponents?

3. Of those AI academics who believe that a machine intelligence could exist which could outmaneuver humans if motivated, how do they think about the possible motivations of a machine intelligence?
3.1 What fraction of them think about AI design in terms of a formalism such as approximating optimal sequential decision theory under a utility function? How easy would it be for them to substitute anthropomorphic intuitions for correct technical predictions?
3.2 What fraction of them think about AI design in terms of intuitively justified decision heuristics? How easy would it be for them to substitute anthropomorphic intuitions for correct technical predictions?
3.3 What fraction of them understand enough evolutionary psychology and/or cognitive psychology to recognize moral evaluations as algorithmically caused, so that they can reject the default intuitive explanation of the cause of moral evaluations, which seems to be: “there are intrinsic moral qualities attached to objects in the world, and when any intelligent agent apprehends an object with a moral quality, the action of the moral quality on the agent’s intelligence is to cause the agent to experience a moral evaluation”?
3.3.1 What combination of specializations in AI, moral philosophy, and cognitive psychology would an academic need to have, to be an “expert” whose disagreements about the material causes and implementation of moral evaluations were significant?

4. On the question of takeoff speeds, what fraction of the AI academics have a good enough intuitive understanding of decision theory to see that a point estimate or default scenario should not be substituted for a marginal posterior distribution, even in a situation where it would be socially costly in the default scenario to take actions which prevent large losses in one tail of the distribution?
4.1 What fraction recognized that they had a prior belief distribution over possible takeoff speeds at all?
4.2 What fraction understood that, regarding a variable which is underconstrained by evidence, “other people would disapprove of my belief distribution about this variable” is not an indicator for “my belief distribution about this variable puts mass in the wrong places”, except insofar as there is some causal reason to expect that disapproval would be somehow correlated with falsehood?

5 What other popular concerns have academics historically needed to dismiss? What decision rules have they learned to decide whether they need to dismiss a current popular concern?
5.1 After they make a decision to dismiss a popular concern, what kinds of causal explanations of the existence of that concern do they make reference to, when arguing to other people that they should agree with the decision?
5.2 How much do the true decision rules depend on those causal explanations?
5.3 How much do the decision rules depend on intuitions about the concerned peoples’ states of mind and social category memberships?
5.4 How much do the causal explanations use concepts which are implicitly defined by reference to hidden intuitions about states of mind and social category memberships?
5.4.1 Can these intuitively defined concepts carry the full weight of the causal explanations they are used to support, or does their power to cause agreement come from their ability to activate social intuitions?

6. Which people are the AI academics aware of, who have argued that intelligence explosion is a concern? What social categories do they intuit those people to be members of? What arguments are they aware of? What states of mind do they intuit those arguments to be indicators of (e.g. as in intuitively computed separating equilibria)?
6.1 What people and arguments did the AI academics think the other AI academics were thinking of? If only a few of the academics were thinking of people and arguments who they intuited to come from credible social categories and rational states of mind, would they have been able to communicate this to the others?

7. When the AI academics made the decision to dismiss concern about an intelligence explosion, what kinds of causal explanations of the existence of that concern did they intuitively expect that they would be able make reference to, if they later had to argue to other people that they should agree with the decision?

It is also possible to model the social process in the panel:

8. Are there factors that might make a joint statement by a panel of AI academics reflect different conclusions than they would have individually reached if they had been outsiders to the AI profession with the same AI expertise?
8.1 One salient consideration would be that agreeing with popular concern about an intelligence explosion would result in their funding being cut. What effects would this have had?
8.1.1 Would it have affected the order in which they became consciously aware of lines of argument that might make an intelligence explosion seem less or more deserving of concern?
8.1.2 Would it have made them associate concern about an intelligence explosion with unpopularity? In doubtful situations, unpopularity of an argument is one cue for its unjustifiability. Would they associate unpopularity with logical unjustifiability, and then lose willingness to support logically justifiable lines of argument that made an intelligence explosion seem deserving of concern, just as if they had felt those lines of argument to be logically unjustifiable, but without any actual unjustifiability?
8.2 There are social norms to justify taking prestige away from people who push a claim that an argument is justifiable while knowing that other prestigious people think the argument to to be a marker of a non-credible social category or state of mind. How would this have affected the discussion?
8.3 If there were panelists who personally thought the intelligence explosion argument was plausible, and they were in the minority, would the authors of the panel’s report mention it?
8.3.1 Would the authors know about it?
8.3.2 If the authors knew about it, would they feel any justification or need to mention those opinions in the report, given that the other panelists may have imposed on the authors an implicit social obligation to not write a report that would “unfairly” associate them with anything they think will cause them to lose prestige?
8.3.3 If panelists in such a minority knew that the report would not mention their opinions, would they feel any need or justification to object, given the existence of that same implicit social obligation?

9. How good are groups of people at making judgments about arguments that unprecedented things will have grave consequences?
9.1 How common is a reflective, causal understanding of the intuitions people use when judging popular concerns and arguments about unprecedented things, of the sort that would be needed to compute conditional probabilities like “Pr( we would decide that concern is not justified | we made our decision according to intuition X ∧ concern was justified )”?
9.2 How common is the ability to communicate the epistemic implications of that understanding in real-time while a discussion is happening, to keep it from going wrong?

A great breakdown, worth thinking carefully about.

12 Responses »

Anders Sandberg directs us to two new lectures by Oxford philosophers.

“Global Catastrophic Risks” by Nick Bostrom
“Human Enhancement: Bioliberation or Biothreat?” by Julian Savulescu

Scroll down a bit to see the controls if you don’t see them at first. The custom flash interface has some cool features, like simultaneously showing the slides and speaker. You can even click a button near the bottom to expand the slides or the speaker window.

In his talk, Savulescu mentions the cognitive enhancement value of iodine in salt. He says that about a billion IQ points are lost each year due to iodine deficiency. If you’re a pregnant woman and you don’t get iodine in your salt during pregnancy, your child loses about 10-15 IQ points. It would cost 2 cents per person per year to iodize salt. 4 billion people lack adequate iodine.

No Responses »

This time, a bird dropped a piece of bread into outdoor machinery which led to part of the accelerator circuit overheating. That’s 2 unrelated shutdown incidents. According to Anders Sandberg, we have to experience 30 unrelated technical failures before we can chalk up the failure of the LHC to anthropic selection. 2 down, 28 to go.

4 Responses »

Shane Legg has worked for more groups pursuing AGI than anyone else in our community. He has worked for Ben Goertzel’s Webmind, Peter Voss’ A2I2, before returning to academia under the well-known AGI theorist Marcus Hutter. Therefore, when he lays down a scary scenario about AI in a 2-hour talk on Halloween, you should listen. David Wood’s summary is a good place to get the main ideas.

No Responses »