The Danger of Powerful Computing Wednesday, Jan 23 2008
AI 4:02 pm
Later this year, Intel will roll out their next-generation, 45-nanometer Nehalem architecture. In terms of clock rates, this means 3.0 GHz and above, but remember that these numbers cannot be used for straightforward extrapolations of processing power. Essentially, these processors will have significantly higher performance than the last generation.
Semiconductor experts believe we have 12 to 15 years of Moore’s law until chips hit the 16 - 11 nm mark and the conventional silicon-based paradigm becomes obsolete. Obviously, hundreds of millions of investor dollars will be put towards developing post-silicon or three dimensional architectures before then.
The Intel Core 2, a chip released in 2006, operates at between 533 and 1333 million instructions per second. By comparison, the human brain has about 100 billion neurons firing 200 times per second. If we assume that each unique firing event represents a functionally meaningful computational operation (doubtful), then the computing power of the human brain is about 20 trillion operations per second, roughly 20,000 times faster than the Intel Core 2.
If the computing power of a typical computer chip keeps doubling every two years or so, as it has for at least 50 years, then we will have home computers with computing power equivalent to the human brain in 2038. Of course, there is no guarantee that the doubling trend will hold. It could stall, or accelerate beyond what we expect.
Human brain-equivalent computing power is something that humanity has never had access to, but it’s just around the corner, historically speaking. It makes you wonder how humanity might utilize computers with such power. If recent news is any indication, such computers will likely be used to implement artificial intelligence (AI) systems for military purposes.
For a while, these artificial intelligence systems will be too stupid to be any real threat — except to those on the business end of military hardware controlled by them. But eventually, they will be cleverer than human beings, and pose a serious threat to humanity in general. People are already hooking up artificial intelligence systems to deadly weapons.
I’m not worried about artificial intelligences suddenly deciding they dislike being bossed around, then turning on their creators. This is anthropomorphic thinking. An AI programmed to obey its creators will not spontaneously turn on them, because it will lack the self-promoting tendencies of creatures sculpted by Darwinian selection. An AI could be exceptional at working out missile trajectories, analyzing enemy troop movements and intelligence data, but incredibly poor at acknowledging itself as a entity worthy of any special entitlements. Casual thinkers fail to realize that our Darwinian instincts correspond to specific neural structures, incredibly unlikely to arise in artificial minds by sheer chance.
The problem arises not when an AI rebels against its programmers, but when it interprets its own programmed goals in ways the programmers did not intend. There are innumerable conceivable examples, but imagine an AI programmed to destroy people against the government of its home specific country. This AI might end up massacring millions of the country’s own citizens with objections to their own government. Surely, if George W. Bush had a pet AI with control over nuclear facilities with such programming, most Americans would be dead by now.
It is not hard to imagine how such an AI might become a threat to humanity in general. An AI programmed to eliminate violence may come to the conclusion that the best way to reduce violence permanently would be to destroy all humanity, judging the immediate violence to be a necessary compromise for avoiding long-term violence over thousands of years. Given access to its own manufacturing hardware, this scenario is feasible.
Even though the notion of a human-equivalent artificial intelligence may seem fantastic to some, I expect to see it within my own lifetime, and within the lifetime of many of my readers here. I have considered several solutions to the challenge of rogue AI, but before I mention them, do you yourself have any? If so, please describe them in the comments.
Try to avoid mentions of fictional scenarios. Think for yourself: Hollywood is not an appropriate intellectual crutch.

January 23rd, 2008 at 4:51 pm
Welcome back.
I’m more concerned about humans with powerful technologies that want to use them intentionally for destructive purposes in the present than I am about bugs (unintended drawbacks) in tomorrow’s techniques.
Show me solutions for existing “rogue nations” or tiny bioterrorist organizations, and we might have something we can extend to threats we don’t actually have, yet.
I still think the real threat of AI isn’t on the battlefield, or even in the jailhouse, but in the labor market. Employment dislocations due to the sudden automation of jobs aren’t going to be universally identified as “bugs” when you consider that companies will be causing them intentionally to make a greater profit in competitive marketplaces.
The economic and cultural “technologies” we already have have bugs to begin with - and I think they’ll probably be very similar bugs to the ones that will arise and be exacerbated by new tech.
January 23rd, 2008 at 7:29 pm
“If we assume that each unique firing event represents a functionally meaningful computational operation (doubtful), then the computing power of the human brain is about 20 trillion operations per second, roughly 20,000 times faster than the Intel Core 2.”
Most computer scientists (or non-biologists) make this assumption to calculate the power of human brain. However, they fail to recognize that the processing power of brain is not simply neuronal firings per second, or the number of connections they make. These obviously add to the incredible complexity, however, there probably hundreds of other parameters such as differential secretion of neurotransmitters, strength of signals,which create combinatorial effects and can modify and change the state of the information that is being stored or transmitted. Not to mention that biological systems do not operate as digital computers and they have inherent flexibility of storing and processing information. Even a single cell probably have more computation power than an Intel core 2 duo chip.
In order to create human level intelligence, the amount of computation power required is possibly trillions of times more than the fastest computer chips of today.
I wouldn’t underestimate the power of brains Michael
January 23rd, 2008 at 8:18 pm
“These obviously add to the incredible complexity, however, there probably hundreds of other parameters such as differential secretion of neurotransmitters, strength of signals,which create combinatorial effects”
Neuron firing is discrete, on-or-off. To quote:
“If the aggregate input is greater than the axon hillock’s threshold value, then the neuron fires, and an output signal is transmitted down the axon. The strength of the output is constant, regardless of whether the input was just above the threshold, or a hundred times as great. The output strength is unaffected by the many divisions in the axon; it reaches each terminal button with the same intensity it had at the axon hillock. This uniformity is critical in an analogue device such as a brain where small errors can snowball, and where error correction is more difficult than in a digital system.”
- http://www.virtualventures.ca/~neil/neural/neuron-a.html
“Also, when the threshold level is reached, an action potential of a fixed sized will always fire…for any given neuron, the size of the action potential is always the same. There are no big or small action potentials in one nerve cell - all action potentials are the same size. Therefore, the neuron either does not reach the threshold or a full action potential is fired - this is the “ALL OR NONE” principle.”
- http://faculty.washington.edu/chudler/ap.html
“Not to mention that biological systems do not operate as digital computers”
We switched to digital for a reason. Every time you copy analog information, it degrades a little. With enough copying, the information becomes useless. Hence, our troubles with memory.
“Even a single cell probably have more computation power than an Intel core 2 duo chip.”
73% of all statistics are made up, including this one. Please provide sources, especially when you contradict a long list of eminent scientists.
“Employment dislocations due to the sudden automation of jobs…”
has been happening for hundreds of years, and yet the number of jobs has steadily increased over the long term. There seems to be some kind of economic mechanism in place to create jobs, even when they aren’t really necessary.
January 23rd, 2008 at 8:50 pm
Is anissimov a luddist??
January 23rd, 2008 at 8:56 pm
There’s always the question whether we’ll have made significant advances in mapping a human brain at a neuronal level by that time. This could entail attempts at recreating a virtual, functioning model of it.
But frankly, I’m more concerned over us not having advanced AI than having it.
Good post.
January 23rd, 2008 at 9:44 pm
I’m not so sure that an AI would be “programmed” as is typically done now. Artificial life and genetic programming techniques are available. Their behavior (almost by definition) can also be influenced by “Darwinian” imperatives.
In fact, for some software projects, such evolutionary development techniques may be the only feasible way to make something working that’s really big and complex.
January 23rd, 2008 at 10:25 pm
People have already written programs with competitive, and territorial drives; for instance, programs for playing online poker. They have also written programs that use Darwinian processes to evolve new programs to solve a specific problem. A future programmer might program a computer to compete against other computers, and to evolve even more complex programs and computers. His motivation might be to gain more money and power for himself, or just for fun. We have seen the beginnings of this in the evolving virus programs that have already been written.
January 23rd, 2008 at 10:44 pm
I generally like your posts but throw-away cuties like “Surely, if George W. Bush had a pet AI with control over nuclear facilities with such programming, most Americans would be dead by now.” make it seem like emotional douchery.
As for concerns about AI taking over the labor market, supposing market competition is also still present in the future, would that not drive prices down? Sounds kind of like that book ‘Player Piano’ where everything is basically free and the biggest problem is finding a cure for the boredom.
January 23rd, 2008 at 11:52 pm
I see the safest scenario as “turtles all the way down”– essentially, not “one big AI” that might decide to complete its goals in undesirable ways, but rather a hierarchy (or ecosystem) of AIs, *each with its own safeguards*.
The basic problem with getting undesired results from a Strong AI’s decision making process is that it’s a monoculture. One bug crops up, and poof. If we build decisional co-dependence into a hierarchy or ecosystem of AIs it might complicate things quite a lot and perhaps lead to its own set of feedback-driven bugs, but it would also significantly ease the critical monoculture/all-powerful-AI-where-a-bug-means-death problem.
Obviously it’s hard to talk about the possible future forms of something we don’t know how to build yet. But my preference would be toward distributed, rather than monolithic, decision making.
January 24th, 2008 at 1:50 am
I think humans lack free-will. That about says it all. Hawkins is an idiot with his “I have noticed even people who claim everything is predestined, and that we can do nothing to change it, look before they cross the road.” HELLO STEVEN, we’re programmed to live and veer into the best of scenarios. Getting hit by a car isn’t exactly a great survival technique. This is relevant. You all should know that with your Darwinian this and Darwinian that. It’s all futile. Physics makes us all it’s bitches. Quantum mechanics doesn’t change shite. But you don’t have a choice so do as you will.
January 24th, 2008 at 3:18 am
It’s odd. I suggested to regard extreme disparity of affluence between people as an existential risk to the lifeboat foundation, Mike.
Yet you affirm it isn’t AI systems being an existential danger, *initially* and you clearly acknowledge an idiot like Bush could, by sheer incompetence or corrupt ideals, trigger the extermination of millions of his own people (and probably a lot more of furriners).
When are you guys at the Lifeboat Foundation going to acknowledge the fact that the ballooning wealth of an upper class, especially when coupled with them being able to collect unprecedented technological tools, is one of the biggest existential risks humanity faces?
We cant do much about the technologies and the growth of computing power. But we sure as hell can point to the potential of an unaccontablle power elite to screw over the world, with whatever means they have.
UNLESS of course such a position would be politically impossible. But if thats the case, just go ahead and admit it is.
January 24th, 2008 at 4:04 am
Hi all, thanks for all the comments. I’ll try to respond to them all. I appreciate the responses.
Nato, thanks for the welcome. We’ve brainstormed many possible solutions for rogue nations, of course, and we’ve seen some unfolding on the world stage in the last decade, as well. In your comment, you seem to imply that the threat of rogue AGI is essentially zero, which I find hard to believe. Humans may be the only agents with the ability to kill millions today, but will it always be that way?
Snowcrash, when you say that a single brain cell has more computational power than an Intel Core 2, you stretch your credulity. Brain power may be greater than the number I gave, but I have to say, your estimate reminds me a little bit of that one paper which came out a couple years ago which asserted human brain processing power was around 10^10^10^10 operations per second. Anyway, feel free to provide a reference for your claim.
Fred, no, I’m not a Luddist, but I advocate selective development of technology. We need to develop protective technologies before potentially dangerous technologies.
hthth, the “how could anything possibly go wrong?” attitude is going to get us in trouble this century.
Horus, maybe you’re right. Still, I doubt programmers would create a human-level AI without having some confidence about which goals it has.
Ken, true, but the computational costs for human-level AGI are likely to be so large that evolving them in a community would be computationally prohibitive. I doubt AI will be evolved — all the enthusiasm about neural networks has already proven misplaced, and engineered approaches are speeding ahead of these methods.
Jim, sorry, but to even have the vague possibility of my posts reaching far beyond preexisting readership, I have to throw in silly things like references to GWB once in a while. People are accustomed to pop culture/political references being tossed out every two seconds. Sometimes my throw-away lines will fail, yes.
Mike, the problem is that the first AI will cost so much that there will be only one. How do you protect humanity in the interval between one powerful AI and an ecology of them?
Yours truly: ????
Khannea, when I think of existential risks, I like to consider specific technologies, rather than humans or human motivations behind them. Otherwise it gets way, way too complicated. If dangerous technologies are developed in a proper order, or suitably regulated (with the help and support of the elite), then even rich people will lack access to them. This is all we can hope for. Enforcing a global wealth distribution scheme and socialist takeoff seems unlikely in the next few decades, and even if it were possible, I doubt it would be very desirable. Look at the lessons of the Soviet Union and China, and the ongoing nationalization of industries in Venezuela. In the latter case, their currency is falling through the floor. I can’t speak for the entire Lifeboat Foundation, but as a general rule, I think we try to avoid politics, or around half of the people there (whichever side is offended by our political positions) would likely leave. If you have socialist politics, then the best way to support those positions would be to vote for socialist leaders and encourage others to do the same.
For another answer, you might want to ask the Lifeboat Foundation’s International Spokesperson, Philippe van Nedervelde.
January 24th, 2008 at 8:24 am
Khannea wrote “When are you guys at the Lifeboat Foundation going to acknowledge the fact that the ballooning wealth of an upper class, especially when coupled with them being able to collect unprecedented technological tools, is one of the biggest existential risks humanity faces?”
There is little risk from these new technologies in the hands of the “power elite”. After all, who has more motivation to maintain the status quo than the people at the top of the system?
No, the real risk is the democratization of technology. As knowledge spreads, the resources of a nation-state or giant corporation are no longer needed to “screw over the world”. Now small fanatical groups can do it, and their motivations are quite different from those who currently hold power. Perhaps they wish to redistribute wealth, or establish a worldwide theocracy. Their cause doesn’t matter - it all boils down to the desire to impose their beliefs on others.
January 24th, 2008 at 10:09 am
Has anyone seen the movie ‘ghost in the shell’? What do you have to say about the concept of “life” being diluted?
Just imagine if an AI after further evolving itself becomes so damn intelligent that it incorporates elements of excellent algorithms and programs which further help it evolve? An AI robot injecting it’s evolved and merged code into other machines would simply imply it is “reproducing”.
What would the concept of life imply then? Reproduce? metabolism? awareness? an AI robot would seemingly possess all of these.
An AI robot would then simply be an electronic replication of the natural organic life systems already in place, would it not?
what do you say?
January 24th, 2008 at 12:12 pm
I agree with many aspects of what has been said so far. For the foreseeable future it’s human intelligence greatly amplified by technology which is going to pose the main threat. The dictators and despots of the future will have access to technologies for surveillance and control of their populations which 20th century tyrants could only dream of. Dictators usually surround themselves by a cult of personality, but when augmented by brain implants and other technologies such individuals could really achieve deity-like status compared to unmodified humans.
Advances in automation still significantly below the radar of human-level intellect will continue to cause economic and societal disruptions. The introduction of new forms of automated labour will drive a wedge between the “haves” and “have nots”, massively increasing disparities in personal income far beyond the levels we see today. There always comes a point where no human labour - however cheap - can compete against an automated system.
I can foresee a new luddite movement, whose membership will be drawn from the ranks of those with a variety of concerns about the rising tide of technology. Some will be reacting against losing their jobs to automation. Others will object to increasing societal pressures for self augmentation, wanting to remain unmodified and resenting those who gain advantage in the employment market and other areas of life though using artificial implants, drugs or bioengineering.
Once SAI or AGI arrives then everything changes. Companies can be run by AGIs with no conventional human labour, making gigantic profits for a handful of ultra-powerful individuals. Such companies will rapidly out-compete those using traditional forms of labour, then battle against each other in a darwinian arms race of speed, intelligence and accelerated economics. Quaint notions of AI friendliness will be forgotten once it becomes obvious how much revenue fully automated companies can generate for their new owners.
January 24th, 2008 at 2:49 pm
I can understand that.
Perhaps significant segmentation of function and diversification of the decision making process can happen, even within a single AI. With our neural architectures, decision making is distributed across multiple brain regions, and if one region makes a bad call it’s often outvoted by other regions. Each brain region also has its own safeguards against error.
I realize that AI architecture may be a lot different than brain architecture– and in a way I’m merely stating the obvious– but I think insofar as we can spread decision making out, and also segment function into modular pieces, we can build in safeguards. Each individual safeguard might not mean much, but in aggregate they might save us from some of the accidental excesses of strong AI.
January 24th, 2008 at 2:57 pm
The best way to reduce the risk of rogue AI is for us to greatly increase our own intelligence before we write the code for a smart AGI. Perhaps we should tax computer hardware makers and use the money to subsidize manufacturers of cognitive enhancing drugs.
January 24th, 2008 at 8:41 pm
Mike: in addition to what Michael said, the first AI would have a good chance of rapidly making itself smarter until, well, it can take over the world (the “intelligence explosion” definition of the Singularity). Even diversification within a single AI won’t help you with this unless its goals don’t make it want to do that, which appears surprisingly hard to ensure while still making the AI do anything useful.
January 24th, 2008 at 11:13 pm
“Later this year, Intel will roll out their next-generation, 45-nanometer Penryn processors.”
Penryn are shipping now. Yorkfield (quad core) Feb/Mar and Wolfsdale (dual core) Jan.
Perhaps you are thinking of Nehalem?
January 25th, 2008 at 12:32 pm
Hi Michael,
I’m curious to know where you get your 100 billion neurons number from. Not saying it’s not accurate, it’s just that I’ve heard 30 billion as the estimate from a few sources (Jeff Hawkins, among others).
Thanks.
January 26th, 2008 at 5:36 pm
Michael Anissimov on Rogue AI…
Michael Anissimov has a nice post up about Rouge AI over on Accelerating Future. He ended his post as follows:
“I have considered several solutions to the challenge of rogue AI, but ……
January 26th, 2008 at 6:37 pm
James, I do agree. Unfortunately there may be other risks from intelligence enhancement, though likely not as severe as AGI development.
As for taxing, that’s quite the economist’s solution. Yet the body politic will not take AGI seriously long before it happens, making this plan infeasible.
Michael, I’ve just heard it repeated a bunch of times. If you track down an authoritative source let me know. In science, numbers like this are often off by an order of magnitude or more.
January 27th, 2008 at 1:05 am
Michael, Snowcrash’s assertion that a single cell has more computing power than an Intel Core 2 due may have been unhelpfully rhetorical, but his central point is spot-on - it’s a gross category error to suppose that the basic unit of computing in a brain is a neuron or a synapse. The unit of biological computing - the equivalent of a transistor - is the molecule, through the logic operations done by an allosteric protein. Nervous systems are a late refinement of the information processing functions carried out by individual cells; there’s a recent paper in PRL, for example, demonstrating behaviour influenced by memory in slime moulds. The credibility of arguments of the type you’re making would be greatly increased if those making them demonstrated a willingness to engage with the real complexities of biological information processing. I can recommend Christof Koch’s book “The Biophysics of Computation: information processing in single neurons” as a good place to start.
January 27th, 2008 at 6:30 am
“it’s a gross category error to suppose that the basic unit of computing in a brain is a neuron or a synapse”
It’s perhaps an even worse category error to assume that there is such a thing as a basic unit of computing in the human brain at all. I would also question the usefulness of the making the software/hardware distinction for evolved brains, so I don’t think you can usefully talk about “the computing power of the human brain”. Evolved systems are messier than you think.
My intuition tells me that the machine I am typing this message on could, if programmed correctly, run software which, for some canonical and useful definition of “intelligence”, would be much much more intelligent than me.
However, no-one has written that software yet, and it is fairly clear that for each order of magnitude increase in FLOPS that silicon computers get, it will be a little bit easier to write said software. How much easier? Again, this is a tough question. My intuition tells me that we’ve probably already reached a point of diminishing returns, but hey, who knows.
January 27th, 2008 at 6:44 am
So, further to my comment above, I would say that computer power is not as dangerous as the fact that programmers will keep tinkering, trying to make better software. Even if computer power is fixed, computer science and computer engineering will continue to advance steadily, and I think that this component is probably much more important than Moore’s law, etc. Who will those programmers be working for? The military? A slightly-less-than-responsible corporation?
This has been said many times, but I’ll say it again: “raw processing power” is probably NOT what is lacking for the development of interesting/dangerous AIs.
January 27th, 2008 at 6:55 pm
Unfortunately there may be other risks from intelligence enhancement, though likely not as severe as AGI development.
What sort of risks?
Significantly slowing down AGI development does, indeed, seem unfeasible in the forseeable future.
Intelligence enhancement may be a double-edged sword, in that it will speed up UFAI as well as FAI… but, if intelligence enhancement led more people to conclude that FAI is desirable, it would be a net bonus.
But, is significant intelligence enhancement a lot *easier* than building an FAI? If not, then such a roundabout mechanism is a poor use of our resources. Instead, we should keep on doing what we’re doing wrt FAI; if the marketplace invents intelligence-enhancement (or any other generic breakthrough technology in the meantime) then we can use it (just as we adopt better hardware and generic software as it gets invented), but it wouldn’t alter our overall game-plan.
February 2nd, 2008 at 10:12 am
Michael, you write smart articles and ask tough questions. And there are many smart people reading you and posting comments.
But you know as well as I do that no one has come up with a clear and definitive answer to these problems. How could we solve these mysteries? Maybe we’re not smart enough to find the answers.
I was just thinking: maybe a little (early-stage) intelligence enhancement could help us grasp what’s in store for us when the subsequent generations of huge intelligence enhancements arrive.
We believe that once humans build smarter-than-human intelligence, it will engage a positive feedback loop and enhance itself, probably at an accelerating or even explosive rhythm. But right before it happens it would be nice if that smarter-than-human intelligence could join us, participate at SIAI, LifeBoat Foundation & other places and help us get ready to face rogue AIs.
May 8th, 2008 at 9:40 am
[…] 10 Futuristic Materials Five Ways You Can Help Transhumanism Now Five Futuristic Forms of Air Travel Feasibility Arguments for Molecular Nanotechnology A Challenger Appears! Interview with Future Blogger Transhumanist Blogs Negative Utilitarianism Seven Influential Transhumanists Is Star Trek a Fascist Society? Brain-Computer Interfaces for Manipulating Dreams Boston Dynamics Big Dog Gimme Some of That Flying Car Shit Top 10 Excuses for Dying Cognitive Enhancement Strategies Vatican Takes Official Anti-Transhumanist Stance Response to Amor Mundi on Transhumanism Nuclear Terrorism High Cost of Force Protection Annalee Newitz’s Vitriolic Anti-Transhumanism Taking Global Risk and WWIII Seriously Human Arrogance Look to Inner, not Outer Space The Religion of Science Temperature Engineering Stephen Omohundro’s “Basic AI Drives” The Danger of Powerful Computing […]