Gregor Wolbring:

The start of a nano arms race, and the lack of willingness to regulate potential synthetic biology through the modification of existing treaties or the application of existing treaties or the development of new regulations is short sighted.

Nano or synthetic biology weapons will diffuse into hands other than the inventor and first user, and it is easier to reverse engineer nano or synthetic biology military products than to make a nuclear weapon. Once they exist they can be copied, and diffusion of the resulting products will make local and global security nearly impossible. Security would come with a hefty price tag — not just in financial terms, but in changes to societal interactions.

Yes, and we have to be willing to make these sacrifices. Not sure about the nuclear weapon thing. I am almost done with a paper that goes over which nanoproducts I should think be regulated or outright banned. Basically, out of all the products that will become possible almost overnight with molecular nanotechnology (MNT), my recommendation is that only a tiny portion should be permitted. Products below a certain size, above a certain size, above a certain energy consumption level (2 MW/person should be appropriate), etc etc etc would all have to go. This shouldn’t be a problem, as even the remaining products would offer orders of magnitude better performance than the products we are used to. However…

The unfortunate fact of the matter is that a long-term, human-only nanotech society is not viable. We are not pacificistic enough, or restrained enough, or wise enough not to do horrible things (mostly by accident, though deliberate misuse is also a problem) with a technology that lets you build products of the quality, quantity, and sheer size that MNT will permit. It doesn’t really matter if the technology arrives in 2015 or 2035 (though obviously I think the former is more likely) - it’s suicidal within a matter of years either way. A central reason is that a billion times present-day computing capacity is guaranteed to make human-level, autonomous AI much, much easier than it is today. This will lead to MNT-equipped AI, which offers such an entity “optimization power” far in excess of multiple human races. Optimization power will be thrown behind a utility function, or supergoal, or whatever you choose to call it, which will be initially specified in terms of computer code by human programmers. The problem is that most utility functions we can currently imagine lack the stability, complexity, precision, and moral sophistication to permit the existence of information theoretically-unique, astonishingly specific matter-structures called “human beings pursuing their own goals”. We are used to inhabiting a world where each agent has roughly the same quantity of optimization power, and nearly the same goals (though selfish goals cause divergent interests, the information content of the utility function itself is the same thing), leading to equilibria that involve the majority surviving and reproducing indefinitely. Not for very much longer. MNT alone introduces power differentials of heretofore unseen proportions, far greater than those introduced by mere nuclear-tipped ICBMs. AI would produce power differentials literally larger than we can imagine.

Back to nanotech, a committee of the NATO Parliamentary Assembly says:

Additional lengthy and scrupulous studies of military NT are urgently needed. The prospects of molecular NT should be assessed with particular attention, as this is the most controversial aspect of NT and would present extremely grave consequences if its feasibility is confirmed.

Grave indeed. Several independent groups have confirmed the feasibility of molecular manufacturing. Although MNT is a favorite speculative technology of many techies, few understand the details, and thus lack the means to determine the difficulty or developmental timeframes except based on intuitive guesses based on projected consequences in popular articles, “inspiration” from video games and novels, or hearsay. For example, if a techie hears that MNT is likely to lead to immortality, they will put it hundreds of years in the future. If they hear that MNT will do little more than make clean water available to 3rd world countries, and allow immersive VR goggles, then they might put its development in 2030. This “forecasting based on consequences” is often concealed using subtle language, but easily destroyed by knowledgable persons asking prodding questions, like “are you familar with concept of a Stewart platform, prior work on mechanosynthetic tooltips, the evolution of the scientific community’s opinion of MNT over the past ten years?”, etc.

In the last few years, I’ve realized that a necessary prerequisite of grasping the consequences of AI is understanding MNT in a non-trivial way. This is likely because those unfamiliar with MNT find it impossible to imagine how an AI would get “out of the box”, even with human help. Because true MNT understanding is still extremely hard to come by, the foundation for AI understanding is simply not there. However, I am hopeful for improvements in MNT understanding, because the technology, when discussed in conservative and straightforward terms, is not radically difficult to grasp, however, there are such deep biases in human modeling of nonhuman minds that explaining advanced AI issues can be near-impossible.

Regards to Chris Peterson for the links.