Crossing the line to superintelligence is of vastly greater significance than the invention of fire, the Internet, or landing on the Moon. All these accomplishments came from human-level intelligence. Boost the underlying intelligence itself, and you’ve done something far deeper than create some new external product of human-level intelligence.

For example, consider the world from the viewpoint of a Homo erectus. They had tools - handaxes. These tools were of various types - pointed, cordate, ovate, ficron and bout-coupé shapes, cleavers, retouched flakes, scrapers, and segmental chopping tools. Flint, basalt, chalcedony, quartzite, andesite, sandstone, chert and shale were all used as raw materials to build these axes. Some were very large and probably just ornamental. Some were discus-shaped and possibly used as hunting weapons. It is thought they also had a social role, with enterprising Homo erectuses fashioning better tools for greater peer approval. From the viewpoint of one of these guys, they had command over a remarkable number of handaxe forms and designs, and put them to use for a variety of different purposes.

From the viewpoint of an intelligence smarter than us in the way that we’re smarter than Homo erectus, all our technology, from planes to trains to lamps to sinks to nanotubes to satellites to linear accelerators, probably look just like variants on the same basic handaxe. Our descendants or future selves will not look back on us admiringly, and say, “golly gee, these guys were so clever that no leap in intelligence ever happened that bested the difference between them and their immediate predecessors!” They will not be genuinely impressed with what we are doing, any more than we are genuinely impressed by a pre-Neolithic hand axe. If we were to show them our greatest technological achievements, they might pretend to be genuinely impressed, so as not to hurt our feelings, but really, they’d probably be daydreaming on the side about mechanisms of such complexity that no aggregation of human beings, no matter how numerous or intelligent, could ever make sense of it all.

I believe that a lot of Singularity skepticism derives from people who don’t get that we’re not the highest form of intelligence that the universe permits to exist. Being a computer science poindexter sometimes hurts more than it helps, because such people are accustomed to being the smartest ones in the room, making it all the more difficult to imagine an intelligence that not only blows them out of the water quantitatively, but can think thoughts they can’t think, even in principle. When people say, “oh, we’ll be able to fight the superintelligent AIs with our rebel guerilla group!”, or “we’ll nuke it to smithereens if it disobeys!”, they don’t get that, once it’s smarter than you, you’ve already lost. Once you’re dealing with something genuinely smarter than human, you have to rely on the hope that it doesn’t want to hurt you, not the assumption that your crappy “foolproof safeguards” will do a lick of good against a true superintelligence. Eliezer came up with the AI Box game to help hammer this point into the collective consciousness.

This is why I raise an eyebrow when people tell me they believe in a slow or incremental AI or IA takeoff. Once you cross the line, you’re quite simply not in Kansas anymore. The scale-up to the point of human-equivalent intelligence may be a slow process, but once you go beyond it, we as humans lose our privilege to say what this new mind can do. Our license to put down limits is permanently revoked. This doesn’t mean that we can’t predict anything - Vinge was wrong when he said that the Singularity is a point of absolute unknowability. If a selfish entity is the first to cross the line, it doesn’t bode as well for humanity than if a selfless and benevolent entity crosses the line. The method of line-crossing (BCI, AI, IA) will influence the play-out of post-line activity, at least until a degree of progress is reached where the post-human point of origin becomes moot. But methods and motivations aside, when we’re talking solely about ability, the prudent assumption to make is that superintelligence is sufficient to shatter most barriers we can conceive of. Self-improving superintelligence all the more so.