Accelerating Future Transhumanism, AI, nanotech, the Singularity, and extinction risk.


Think Twice: A Response to Kevin Kelly on ‘Thinkism’

In late 2008, tech luminary Kevin Kelly, the founding executive editor of Wired magazine, published a critique of what he calls "thinkism" -- the idea of smarter-than-human Artificial Intelligences with accelerated thinking and acting speeds developing science, technology, civilization, and physical constructs at faster-than-human rates. The argument over "thinkism" is important to answering the question of whether Artificial Intelligence could quickly transform the world once it passes a certain threshold of intelligence, called the "intelligence explosion" scenario.

Kelly begins his blog post by stating that “thinkism doesn’t work", specifically meaning that he doesn't believe that a smarter-than-human Artificial Intelligence could rapidly develop infrastructure to transform the world.  After using the Wikipedia definition of the Singularity, Kelly writes that Vernor Vinge, Ray Kurzweil and others view the Singularity as deriving from smarter-than-human Artificial Intelligences (superintelligences) developing the skills to make themselves smarter, doing so at a rapid rate. Then, “technical problems are quickly solved, so that society’s overall progress makes it impossible for us to imagine what lies beyond the Singularity’s birth”, Kelly says. Specifically, he alludes to superintelligence developing the science to cure the effects of human aging faster than they accumulate, thereby giving us indefinite lifespans. The notion of the Singularity is roughly that the creation of superintelligence could lead to indefinite lifespans and post-scarcity abundance within a matter of years or even months, due to the vastly accelerated science and robotics that superintelligence could develop. Obviously, if this scenario is plausible, then it might be worth devoting more resources to developing human-friendly Artificial Intelligence than we are currently. A number of eminent scientists are beginning to take the scenario seriously, while Kelly stands out as an interesting critic.

Kelly does not dismiss the Singularity concept out of hand, saying "I agree with parts of that. There appears to be nothing in the composition of the universe, or our minds, that would prevent us from making a machine as smart as us, and probably (but not as surely) smarter than us." However, he then rejects the hypothesis, saying, "the major trouble with this scenario is a confusion between intelligence and work. The notion of an instant Singularity rests upon the misguided idea that intelligence alone can solve problems." Kelly quotes the Singularity Institute article, "Why Work Towards the Singularity", arguing it implies an "approach [where] one only has to think about problems smartly enough to solve them." Kelly calls this "thinkism".

Kelly brings up concrete examples, such as curing cancer and prolonging life, stating that these problems cannot be solved by “thinkism.” "No amount of thinkism will discover how the cell ages, or how telomeres fall off", Kelly writes. "No intelligence, no matter how super duper, can figure out how human body works simply by reading all the known scientific literature in the world and then contemplating it." He then highlights the necessity of experimentation in deriving new knowledge and working hypotheses, concluding that, "thinking about the potential data will not yield the correct data. Thinking is only part of science; maybe even a small part."

Part of Kelly's argument rests on the idea that there are fixed-rate external processes, such as the metabolism of a cell, which cannot be sped up to provide more experimental data than they would otherwise. He explains, that "there is no doubt that a super AI can accelerate the process of science, as even non-AI computation has already sped it up. But the slow metabolism of a cell (which is what we are trying to augment) cannot be sped up." He also uses physics as an example, saying "If we want to know what happens to subatomic particles, we can't just think about them. We have to build very large, very complex, very tricky physical structures to find out. Even if the smartest physicists were 1,000 smarter than they are now, without a Collider, they will know nothing new." Kelly acknowledges the potential of computer simulations but argues they are still constrained by fixed-rate external processes, noting, "Sure, we can make a computer simulation of an atom or cell (and will someday). We can speed up this simulations many factors, but the testing, vetting and proving of those models also has to take place in calendar time to match the rate of their targets."

Continuing his argument, Kelly writes: "To be useful artificial intelligences have to be embodied in the world, and that world will often set their pace of innovations. Thinkism is not enough. Without conducting experiements, building prototypes, having failures, and engaging in reality, an intelligence can have thoughts but not results. It cannot think its way to solving the world's problems. There won't be instant discoveries the minute, hour, day or year a smarter-than-human AI appears. The rate of discovery will hopefully be significantly accelerated. Even better, a super AI will ask questions no human would ask. But, to take one example, it will require many generations of experiments on living organisms, not even to mention humans, before such a difficult achievement as immortality is gained."

Concluding, Kelly writes: "The Singularity is an illusion that will be constantly retreating -- always "near" but never arriving. We'll wonder why it never came after we got AI. Then one day in the future, we'll realize it already happened. The super AI came, and all the things we thought it would bring instantly -- personal nanotechnology, brain upgrades, immortality -- did not come. Instead other benefits accrued, which we did not anticipate, and took long to appreciate. Since we did not see them coming, we look back and say, yes, that was the Singularity."

This fascinating post of Kelly's raises many issues, the two most prominent being:

1) Given sensory data X, how difficult is it for agent Y to come to conclusion Z?
2) Can experimentation be accelerated past the human-familiar rate or not?

These will be addressed below.

Can We Just Think Our Way Through Problems?

There are many interesting examples in human history of situations where people "should" have realized something but didn't. For instance, the ancient Egyptians, Greeks, and Romans had all the necessary technology to manufacture hot air balloons, but apparently never thought of it. It wasn't until 1783 that the first historic hot-air balloon flew. It is possible that ancient civilizations did build hot-air balloons and left no archeological evidence of their remains. One hot air balloonist thinks the Nazca lines were viewed by prehistoric balloonists. My guess would be that the ancients might have been clever enough to manufacture hot air balloons, but probably not. The point is that they could have built them, but didn't.

Inoculation and vaccination are another relevant example. A text from 8th century India included a chapter on smallpox and mentioned methods of inoculating against the disease. Given that the value of inoculation was known in India c. 750 BC, it would seem that the modern age of vaccination should have arrived prior to 1796. Besides safe water, vaccines reduce mortality and increase population growth more than any other means. Aren't 2,550 years enough time to go from the basic principle of inoculation to the notion of systematic vaccination? It could be argued that the discovery of cell theory (1665) was a limiting factor; if cell theory had been introduced to 8th century Indians, perhaps they would have been able to develop vaccines and save the world from hundreds of millions of unnecessary deaths.

Lenses, which are no more than precisely curved pieces of glass, are fundamental to scientific instruments: the microscope and the telescope and are at least 2,700 years old; the Nimrud lens, discovered at the Assyrian palace of Nimrud in modern-day Iraq, demonstrates their antiquity. The discoverer of the lens noted that he had seen very small inscriptions on Assyrian artifacts that made him suspect that a lens was used to create them. There are numerous references to and evidence of lenses in antiquity. The Visby lenses found in a 11th to 12th century Viking town are sophisticated aspheric lenses with angular resolution of 25–30 µm. Even after lenses became widespread in 1280, it took microscopes almost 500 years to develop to the point of being able to discover cells. Given that lenses are as old as they are, why did it take so incredibly long for our ancestors to develop them to the point of being able to build microscopes and telescopes?

A final example that I will discuss regards complex gear mechanisms and analog computers in general. The Antikythera mechanism, dated to 100 BC, consists of about 30 precisely interlocked bronze gears designed to display the locations in the sky of the Sun, Moon, and the five planets known at the time. Why wasn't it until more than 1,400 years later that mechanisms of similar complexity were constructed? At the time, Greece was a developed civilization of about 4-5 million people. It could be that a civilization of sufficient size and stability to produce complex gear mechanisms did not come into existence until 1,400 years later. Perhaps a simple lack of ingenuity is to blame. The exact answer is unknown, but we do know that the mechanical basis for constructing bronze gears of similar quality existed for a long time, it just wasn't put into use.

It apparently takes a long time for humans to figure some things out. There are numerous historic examples where all the pieces of a puzzle were on the table, there was just no one who put them together. The perspective of "thinkism" suggests that if the right genius were alive at the right time, he or she would have put the pieces together and given civilization a major push forward. I believe that this is borne out by contrasting the historical record with what we know today.

Value of Information

It takes a certain amount of information to come to certain conclusions. There is a minimum amount of information required to identify an object, plan a winning strategy in a game, model someone's psychology, or design an artifact. The more intelligent or specialized the agent is, the less information it needs to reach the conclusion. Conclusions may be "good enough" rather than perfect, in other words, "ecologically rational".

An example is how good humans are at recognizing faces. The experimental data shows that we are fantastic at this; in one study, half of respondents correctly identified this image as being a portrait of Napoleon Bonaparte, even though it is only a mere 6×7 pixels.

MIT computational neuroscientist Pawan Sinha found that given 12 by 14 pixels worth of visual information, his experimental subjects could recognize 75-percent of the face images in a set accurately, where the set had a mix of faces and other objects. Sinha also programmed a computer to identify face images, with a high success rate. A New York Times article quotes Dr. Sinha: "These turn out to be very simple relationships, things like the eyes are always darker than the forehead, and the mouth is darker than the cheeks,” Dr. Sinha said. “If you put together about 12 of these relationships, you get a template that you can use to locate a face.” There are already algorithms that can identify faces from databases which only include a single picture of an individual.

These results are relevant because they are examples where humans or software programs are able to make correct judgments with extremely small amounts of information, less than we would intuitively think is necessary. The picture of Napoleon above can be specified by about 168 bits. Who would imagine that hundreds of people in an experimental study could uniquely identify a historic individual based on a photo containing only 168 bits of information? It shows that humans have cognitive algorithms that are highly effective and specialized at identifying such information. Perhaps we could make huge scientific breakthroughs if we had different cognitive algorithms specialized at engaging unfamiliar, but highly relevant data sets.

The same could apply to observations and conclusions of all sorts. The amount of information needed to make breakthroughs in science could be less than we think. We do know that new ways of looking at the world can make a tremendous difference in uncovering true beliefs. A civilization without science might exist for a long time without accumulating significant amounts of objective knowledge about biology or physics. For instance, the Platonic theory of classical elements persisted for thousands of years.

Then, science came along. In the century following the development of the Scientific Method by Francis Bacon in 1620, there was rapid progress in science and technology, fueled by this new worldview. By 1780, the Industrial Revolution was in full swing. If the Scientific Method had been invented and applied in ancient Greece, progress that would have seemed mind-boggling and impossible at the time, like the Industrial Revolution, could have potentially been achieved within a century or two. The Scientific Method increased the marginal usefulness of each new piece of knowledge humanity acquired, giving it a more logical and epistemologically productive framework than was accessible in the pre-scientific haze.

Could there be other organizing principles of effective thinking analogous to the Scientific Method that we're just missing today? It seems hard to rule it out, and quite plausible. The use of Bayesian principles in inference, which has led to breakthroughs in Artificial Intelligence, would be one candidate. Perhaps better thinkers could discover such principles more rapidly than we can, and make fundamental breakthroughs with less information than we would currently anticipate being necessary.

The Essence of Intelligence is Surprise

A key factor defining feats of intelligence or cleverness is surprise. Higher intelligence sees the solution no one else saw, looks past the surface of a problem to find the general principles and features that allow them to understand and resolve it. A classic, if cliché example is Albert Einstein deriving the principles of special relativity working as a patent clerk in Bern, Switzerland. His ideas were considered radically counterintuitive, but proved correct. The concept of the speed of light being constant for all observers regardless of their velocity had no precedent in Newtonian physics or common sense. It took a great mind to think about the universe in a completely new way.

Kelly rejects the notion of superintelligence leading to immortality when he says, "this super-super intelligence would be able to use advanced nanotechnology (which it had invented a few days before) to cure cancer, heart disease, and death itself in the few years before Ray had to die. If you can live long enough to see the Singularity, you'll live forever [...] The major trouble with this scenario is a confusion between intelligence and work." Kelly highlights "immortality" as being very difficult to achieve through intelligence and its fruits alone, but this understanding is relative. Medieval peasants would see rifles, freight trains, and atomic bombs as very difficult to achieve. Stone Age man would see bronze instruments as difficult to achieve, if he could imagine them at all. The impression of difficulty is relative to intelligence and the tools a civilization has. To very intelligent agents, a great deal of tasks might seem easy, including vast categories of tasks that less intelligent agents cannot even comprehend.

Would providing indefinite lifespans (biological immortality) to humans be extremely difficult, even for superintelligences? Instead of saying "yes" based on the evidence of our own imaginations, we must confess that we don't know. This doesn't mean that the probability is 50% -- it means we really don't know. We can come up with a tentative probability, say 10%, and iterate based on evidence that comes in. But to say that it will not happen with high confidence is impossible, because a lesser intelligence cannot place definite limits (outside of, perhaps, the laws of physics) on what a higher intelligence or more advanced civilization can achieve. To say that it will happen with high confidence is also impossible, because we lack the information.

The general point is that one of the hallmarks of great intelligence is surprise. The discovery of gunpowder must have been a surprise. The realization that the earth orbits the Sun and not vice versa was a surprise. The derivation of the laws of motion and their universal applicability was a surprise. The creation of the steam engine led to surprising results. The notion that we evolved from apes surprised and shocked many. The idea that life was not animated by a vital force but in fact operated according to the same rules of chemistry as everything else was certainly surprising. Mere human intelligence has surprised us time and time again with its results -- we should not be surprised to be surprised again by higher forms of intelligence, if and when they are built.

Accelerating Experimentation

One of Kelly's core arguments is that experimentation to derive new knowledge and the "testing, vetting and proving" of computer models will require "calendar time". However, it is possible to imagine ways in which the process of experimentation and empirical verification could be accelerated to faster-than-human-calendar speeds.

To start, consider variance in the performance of human scientists. There are historic examples of times where scientific and technological progress was very rapid. The most recent and perhaps striking example was during World War II. Within six years, the following technologies were invented: radar, jet aircraft, ballistic missiles, nuclear power and weapons, and general-purpose computers. So, despite fixed-rate external processes limiting the rate of experimentation, innovation was temporarily accelerated anyway. Intuitively, the rate of innovation was arguably three to four times greater than in a similar period before the war. Though the exact factor is subjective, few historians would disagree that rapid scientific innovation occurred during WWII.

Why was this? Several factors may be identified: 1) increased military spending on research, 2) more scientists due to better training connected to the war effort, 3) researchers working harder and with more motivation than they otherwise would, 4) second-order effects resulting from larger groups of brilliant people interacting with one another in a supportive environment, as in the Manhattan Project.

An advanced Artificial Intelligence could employ all these strategies to accelerate its own speed of research and development. It could 1) amass a large amount of resources in the form of physical and social capital, and spend them on research, 2) copy itself thousands or millions of times using available computers to ensure there are many researchers, 3) possess perfect patience, perpetual alertness, and accelerated thinking speed to work harder than human researchers can, and 4) benefit from second-order effects by utilizing electronic communication between its constituent researcher-agents. To the extent that accelerated innovation is possible with these strategies, an Artificial Intelligence could exploit them to the fullest degree possible.

Of course, experimentation is certainly necessary to make scientific progress -- many revolutions in science begin with peculiar phenomena that are artificially magnified with the aid of carefully designed experiments. For instance, the double-slit experiment in quantum mechanics emphasizes the wave-particle duality of light, a phenomenon not typically observed during everyday circumstances. Determining the details of how different chemicals intermingle to produce reaction products has required millions of experiments. Understanding biology has required many millions of experiments as well. Only strictly observational facts such as the cellular structure of life or the surface features of the Moon can be assessed through direct observation. To determine how metabolic processes actually work or what is underneath the surface of the moon requires experimentation, trial and error.

There are four concrete ways in which experimentation might be accelerated to speeds beyond the typical human level. These are conducting experiments faster, more efficiently, conducting them in parallel, and choosing the most useful experiments to begin with. Kelly argues that "the slow metabolism of a cell (which is what we are trying to augment) cannot be sped up". But, this is not entirely clear. It should be possible to build chemical networks that simulate cellular processes operating more quickly than cellular metabolisms do. In addition, it is not clear that a comprehensive understanding of cells would be necessary to achieve biological immortality. Achieving indefinite biological lifespans could be more readily achieved by repairing cellular damage and chemical junk faster than it accumulates than constantly keeping all cells in a state of perpetual youth, which seems to be what Kelly is implying is necessary. In fact, it may be possible to develop therapies for repairing the damages of aging with our current biological knowledge. Since we aren't superintelligences, it is impossible to tell. But Kelly makes an error when he assumes that keeping all cells in a state of perpetual youth, or total understanding, is required for indefinite lifespans. This shows how even small differences in knowledge between humans can make an all-important difference in research targets and agendas. The difference in knowledge between humans and superintelligences will make the difference larger still.

Considering these factors highlights the earlier point that the perceived difficulty of a given advance, like biological immortality, is strongly influenced by the framing of the necessary prerequisites to achieve that advance, and the intelligence doing the evaluation. Kelly's framing of the problem is that massive amounts of biological experimentation would be necessary to derive the knowledge to repair the body faster than it breaks down. This may be the case, but it might not be. A higher intelligence might be able to achieve equivalent insights with ten experiments that a lesser intelligence would require a thousand experiments to uncover.

The rate of useful experimentation by superhuman intelligences will depend on factors such as 1) how much data is needed to make a given advance and 2) whether experiments be accelerated, simplified, or made massively parallel.

Research in biology, medicine, and chemistry have exploited highly parallel robotic systems for experiments. This field is called high-throughput screening (HTS). One paper describes a machine that simultaneously introduces 1,536 compounds to 1,536 assay plates, performing 1,536 chemical experiments at once in a completely automated fashion, determining 1,536 dose-response curves per cycle. Only 23 nanoliters of each compound is transferred. This highly miniaturized, highly parallel, high-density mode of experimentation has only begun to be exploited due to advances in robotics. If robotics could be manufactured on a massive scale more cheaply, one can imagine warehouses full of such machines conducting many hundreds of millions of experiments simultaneously.

Another method of accelerating experimentation would be to improve microscale manufacturing and to construct experiments using the minimum possible quantity of matter. For instance, instead of dropping weights off the Leaning Tower of Pisa, construct a microscale vacuum chamber and drop a cell-sized diamond grain in that chamber. Thousands of physics experiments could be conducted in the time it would require to conduct one experiment by the traditional method. With better sensors, you can conduct an experiment on ten cells that with inferior sensors would necessitate a million cells. More fine-grained control of matter can allow an agent to extract much more information from a smaller experiment that costs less and can be run faster and massively parallel. It is conceivable that an advanced Artificial Intelligence could come up with millions of hypotheses and test them all simultaneously in one small building.

Between-Species Comparisons

In his 1992 paper defining the Singularity, Vernor Vinge called the hypothetical post-Singularity world "a regime as radically different from our human past as we humans are from the lower animals". Kelly, meanwhile, said that for artificial intelligences to amass scientific knowledge and make breakthroughs (like biological immortality) would require detailed models, and the "testing, vetting and proving of those models" requires "calendar time". These models will "take years, or months, or at least days, to get results". Since the comparison between different species is sometimes seen as a model for plausible differences between humans and superintelligences, let's apply that model to the context of experiments that Kelly is referring to. Do humans create effects in the world faster than squirrels? Yes. Are humans qualitatively better at working towards biological immortality than squirrels? Yes. Do humans have a fundamentally superior understanding of the universe than squirrels do? It would be safe to say that we do.

The comparison with squirrels sounds absurd because concepts like biological immortality and "understanding the universe" are fuzzy at best from the perspective of a squirrel. Analogously, there may be stages in comprehension of reality that are fundamentally more advanced than our own and only accessible to higher intelligences. In this way, the "calendar time" of humans would have no more meaning to a superintelligence than "squirrel time" has relevance to human life. It's not so much a factor of time -- though higher intelligences can do much more in much less time -- but also the general category of thoughts which can be processed, objectives which can be imagined, and plans which can be achieved. The objectives and methods of a higher intelligence would be on a completely different level than those of a lower intelligence and are different in kind, not degree.

There are several reasons why it makes sense to assume that qualitatively smarter-than-human intelligence, that is, qualitative differences on the order of difference between humans and squirrels or greater, should be possible. The first reason concerns the relative speed of human neurons relative to artificial computing machinery. Modern computers operate at billions of serial operations per second. Human neurons operate at only a couple hundred serial operations per second. Since most acts of cognition must occur within one second to be evolutionarily useful, and must include redundancy and fault tolerance, the brain is constrained to problem solutions involving 100 serial steps or less. What about the universe of possible solutions to cognitive tasks that require more than 100 serial steps? If the computer you are using had to implement every meaningful operation in 100 serial steps, the vast majority of common algorithms used today would have to be thrown out. In the space of possible algorithms, it quickly becomes obvious that constraining a computer to 100 serial steps is an onerous limitation. Expanding this space by a factor of ten million seems likely to lead to significant qualitative improvements in intelligence.

The reason that qualitatively smarter-than-human intelligence is possible is about neurological hardware and software. There are relatively few hardware differences between humans and chimpanzee brains. The evidence actually supports the notion that primate brains are more distinct from non-primates than humans are from other primates, and that the human brain is merely a primate brain scaled up for a larger body and with an enlarged prefrontal cortex. One quantitative study of human vs. chimpanzee brain cells came to this conclusion:

Despite our ongoing efforts to understand biology under the light of evolution, we have often resorted to considering the human brain as an outlier to justify our cognitive abilities, as if evolution applied to all species except humans. Remarkably, all the characteristics that appeared to single out the human brain as extraordinary, a point off the curve, can now, in retrospect, be understood as stemming from comparisons against body size with the underlying assumptions that all brains are uniformly scaled-up or scaled-down versions of each other and that brain size (and, hence, number of neurons) is tightly coupled to body size. Our recently acquired quantitative data on the cellular composition of the human brain and its comparison to other brains, both primate and nonprimate, strongly indicate that we need to rethink the place that the human brain holds in nature and evolution, and to rewrite some basic concepts that are taught in textbooks. The human brain has just the number of neurons and nonneuronal cells that would be expected for a primate brain of its size, with the same distribution of neurons between its cerebral cortex and cerebellum as in other species, despite the relative enlargement of the former; it costs as much energy as would be expected from its number of neurons; and it may have been a change from a raw diet to a cooked diet that afforded us its remarkable number of neurons, possibly responsible for its remarkable cognitive abilities.

In other words, it appears as if our exceptional cognitive abilities are the direct result of having more neurons rather than neurons in differing arrangements or relative quantities. If this continues to be confirmed in subsequent analyses, it implies, all else equal, that scaling up the number of neurons in the human brain could lead to similar intelligence differentials as those between humans and chimps. Given the evidence above, this should be our default assumption -- we would need specific reasoning or evidence to assume otherwise.

A more detailed reason for why qualitatively smarter-than-human intelligence seems possible is that the higher intelligence of humans and primates appears to have something to do with self-awareness and complex self-referential loops in thinking and acting. The evolution of primate general intelligence appears correlated with the evolution of brain structures that control, manipulate, and channel the activity of other brain structures in a contingent way. For instance, a region called the pulvinar was called the brain's "switchboard operator" in a recent study, though there are dozens of brain areas that could be given this description. Of 52 Brodmann areas in the cortex, at least seven are "hub areas" which lie near the top of a self-reflective control hierarchy: areas 8, 9, 10, 11, 12, 25, and 28. Given that these areas obviously play important roles in what we consider higher intelligence, yet only evolved relatively recently in evolutionary terms and are comparatively poorly developed, it is quite plausible to suggest that there is a lot of room for improvement in these areas and that qualitative intelligence improvements could result.

Imagine a brain that has "hub areas" which can completely reprogram other brain modules on a fine-grained level, the sort of reprogramming and flexibility only currently available in computers. Instead of only being able to reprogram a few percent of the information content of our brains, like we have now, a mind that can reprogram 100 percent of its own information content would allow limitless room for fast, flexible cognitive adaptation. Such a mind could quickly reprogram itself to suit the task at hand. Biological intelligences can only dream of this kind of adaptiveness and versatility. It would open up a vast new space not only for functional cognition but also appreciation of aesthetics and other higher-order mental traits.

Superior Hardware and Software

Say that we could throw open the hood of the brain and enhance it. How would that work?

To understand how "smarter than human intelligence" would work requires overviewing how the brain works. The brain is a very complicated machine. It operates entirely according to the laws of physics, and includes specific modules designed to handle different tasks. Look at our capabilities of identifying faces; it is clear that our brains have specific neural hardware adapted to rapidly identifying human faces. We don't have the same hardware for rapidly identifying lizard faces -- every lizard is just a lizard. To a lizard, different lizard faces might intuitively appear highly distinct, but to us humans, a species wherein there is no adaptive value in differentiating lizard faces, they all look the same.

The paper "Intelligence Explosion: Evidence and Import" by Luke Muehlhauser and Anna Salamon reviews some features of what Eliezer Yudkowsky calls the "AI Advantage" -- inherent advantages that an Artificial Intelligence would have over human thinkers as a natural consequence of its digital properties. Because many of these properties are so key to understanding the "cognitive horsepower" behind claims of "thinkism", I've chosen to excerpt the entire section on "AI Advantages" here, minus references (you can find those in the paper):

Below we list a few AI advantages that may allow AIs to become not only vastly more intelligent than any human, but also more intelligent than all of biological humanity. Many of these are unique to machine intelligence, and that is why we focus on intelligence explosion from AI rather than from biological cognitive enhancement.

Increased computational resources. The human brain uses 85–100 billion neurons. This limit is imposed by evolution-produced constraints on brain volume and metabolism. In contrast, a machine intelligence could use scalable computational resources (imagine a “brain” the size of a warehouse). While algorithms would need to be changed in order to be usefully scaled up, one can perhaps get a rough feel for the potential impact here by noting that humans have about 3.5 times the brain size of chimps, and that brain size and IQ correlate positively in humans, with a correlation coefficient of about 0.35. One study suggested a similar correlation between brain size and cognitive ability in rats and mice.

Communication speed. Axons carry spike signals at 75 meters per second or less. That speed is a fixed consequence of our physiology. In contrast, software minds could be ported to faster hardware, and could therefore process information more rapidly. (Of course, this also depends on the efficiency of the algorithms in use; faster hardware compensates for less efficient software.)

Increased serial depth. Due to neurons’ slow firing speed, the human brain relies on massive parallelization and is incapable of rapidly performing any computation that requires more than about 100 sequential operations. Perhaps there are cognitive tasks that could be performed more efficiently and precisely if the brain’s ability to support parallelizable pattern-matching algorithms were supplemented by support for longer sequential processes. In fact, there are many known algorithms for which the best parallel version uses far more computational resources than the best serial algorithm, due to the overhead of parallelization.

Duplicability. Our research colleague Steve Rayhawk likes to describe AI as “instant intelligence; just add hardware!” What Rayhawk means is that, while it will require extensive research to design the first AI, creating additional AIs is just a matter of copying software. The population of digital minds can thus expand to fill the available hardware base, perhaps rapidly surpassing the population of biological minds. Duplicability also allows the AI population to rapidly become dominated by newly built AIs, with new skills. Since an AI’s skills are stored digitally, its exact current state can be copied, including memories and acquired skills—similar to how a “system state” can be copied by hardware emulation programs or system backup programs. A human who undergoes education increases only his or her own performance, but an AI that becomes 10% better at earning money (per dollar of rentable hardware) than other AIs can be used to replace the others across the hardware base—making each copy 10% more efficient.

Editability. Digitality opens up more parameters for controlled variation than is possible with humans. We can put humans through job-training programs, but we can’t perform precise, replicable neurosurgeries on them. Digital workers would be more editable than human workers are. Consider first the possibilities from whole brain emulation. We know that transcranial magnetic stimulation (TMS) applied to one part of the prefrontal cortex can improve working memory. Since TMS works by temporarily decreasing or increasing the excitability of populations of neurons, it seems plausible that decreasing or increasing the “excitability” parameter of certain populations of (virtual) neurons in a digital mind would improve performance. We could also experimentally modify dozens of other whole brain emulation parameters, such as simulated glucose levels, undifferentiated (virtual) stem cells grafted onto particular brain modules such as the motor cortex, and rapid connections across different parts of the brain. Secondly, a modular, transparent AI could be even more directly editable than a whole brain emulation—possibly via its source code. (Of course, such possibilities raise ethical concerns.)

Goal coordination. Let us call a set of AI copies or near-copies a “copy clan.” Given shared goals, a copy clan would not face certain goal coordination problems that limit human effectiveness. A human cannot use a hundredfold salary increase to purchase a hundredfold increase in productive hours per day. But a copy clan, if its tasks are parallelizable, could do just that. Any gains made by such a copy clan, or by a human or human organization controlling that clan, could potentially be invested in further AI development, allowing initial advantages to compound.

Improved rationality. Some economists model humans as Homo economicus: self-interested rational agents who do what they believe will maximize the fulfillment of their goals. On the basis of behavioral studies, though, Schneider (2010) points out that we are more akin to Homer Simpson: we are irrational beings that lack consistent, stable goals. But imagine if you were an instance of Homo economicus. You could stay on a diet, spend the optimal amount of time learning which activities will achieve your goals, and then follow through on an optimal plan, no matter how tedious it was to execute. Machine intelligences of many types could be written to be vastly more rational than humans, and thereby accrue the benefits of rational thought and action. The rational agent model (using Bayesian probability theory and expected utility theory) is a mature paradigm in current AI design.

It seems likely to me that Kevin Kelly does not really understand the AI advantages of increased computational resources, communication speed, increased serial depth, duplicability, editability, goal coordination, and improved rationality, and how these abilities could be used to accelerate, miniaturize, parallelize, and prioritize experimentation to such a degree that the "calendar time" limitation could be surpassed. The calendar of a powerful AI superintelligence might be measured in microseconds rather than months. Different categories of beings have different calendars to which they are most accustomed. In the time it takes for a single human neuron to fire, a superintelligence might have decades of subjective time to contemplate the mysteries of the universe.

Nos es a Polim

Part of the initial insight that prompted the perspective that Kelly calls "thinkism" was that the brain is a machine which can be accelerated by porting the crucial algorithms on a different substrate, namely a computer, and running them faster. The brain works through algorithms -- that is, systematic procedures. For an example, take the visual cortex, the part of the brain that processes what you see. This region of the brain is actually relatively well understood. The first layers capture surface features such as lines, darkness, and light. Deeper layers make out shapes, then motion, then specifics such as which face belongs to which person. It gets so specific that scientists have measured individual neurons that recognize celebrities like Bill Clinton or Marilyn Monroe.

The algorithms that underlie our processing of visual information are understood on a basic level, and it is only a matter of time until all the other cognitive algorithms are understood as well. When they are, they will be implemented on computers and sped up by a factor of thousands or millions. Human neurons fire 200 times every second, computer chips fire 2,000,000,000 times every second.

What would it be like to be a mind running at ten million times human speed? If your mind is really really fast, events on the outside would seem really really slow. All the elapsed time from the founding of Rome to the present day could be experienced subjectively in about two hours. All the time from the emergence of Homo sapiens to the present day could experienced in a week. All the time from the dinosaurs to the present day could be experienced in a mere 2,400 years. Imagine how quickly a mind could accrue profound wisdom running at such an accelerated speed; the "wisdom" of a 90-year old would seem childlike by comparison.

To visualize concretely the kind of arrangement in which these minds could exist, imagine a computer a couple hundred feet across made of dense nanomachinery situated at the bottom of the ocean. Such a computer would have far more computing power than the entire planet today, similar to the way that a modern smartphone has more computing power than the entire world in 1960. Within this computer would exist virtual worlds practically without end; their combined volume far exceeding that of the solar system, or perhaps even the galaxy.

In his post, Kelly seems to acknowledge that minds could be vastly accelerated and magnified in this way: remarkably, he just doesn't think that this would translate to increased wisdom, performance, ability, or insight significantly beyond the human level. To me, at first impression, the notion that a ten million times speedup would have a negligible effect on scientific innovation or progress seems absurd. It appears obvious that it would have a world-transforming impact. Let's look at the argument more closely.

The disagreement between Singularitarians such as Vinge and Kurzweil and skeptics such as Kelly seems to be about what sorts of information-acquisition and generation procedures can be imported into this vastly accelerated world and which cannot. In his hard sci-fi book Diaspora, author Greg Egan calls the worlds of these enormously accelerated minds "polises", which make up the vast bulk of humanity in 2975. Vinge and Kurzweil see the process of knowledge acquisition and creation as being something that can in principle be sped up, brought "within the purview of the polis", whereas Kelly does not.

Above, I argued how the benefits of experimentation can be accelerated through the processes of running experiments faster, parallelizing them, using less matter, and choosing the right experiments. But what about less controversial information flow from world to polis? To build the polis to begin with, you'd have to be able to emulate -- not just simulate -- the human mind in detail, that is, copy all of its relevant properties. Since the human brain is one of, if not the most complex object in the universe that we know of, this also implies that a vast variety of less complex objects could be scanned and inputted to the polis in a similar fashion. Trees, for instance, could be mass-inputted into the virtual environment of the polis, consuming thousands or millions of times less computing power than the sentient inhabitants. It goes without saying that nonbiological, inanimate background features such as landscapes could be input into the polis with a bare minimum of challenge or difficulty.

Once a process can be simulated with a reasonable level of computing power, it can be inputted into the polis and run at a factor of tens-of-millions speedup. Newtonian physics, for instance. Today, we use huge computers to perform molecular dynamics simulations on aggregates of a few hundred atoms, simulating a few microseconds of their activity. With futuristic nanocomputers built by superintelligent Artificial Intelligences, macro-scale systems could be simulated for hours of activity for a very affordable cost in computing power. Such simulations would allow these intelligences to extract predictive regularities, or "rules of thumb" which would allow them to avoid simulating these systems in such excruciating detail in the future. Instead of requiring full-resolution molecular dynamics simulations to extrapolate the behavior of large systems, they might resolve a set of several thousand generalities that allow these systems to be predicted and understood with a high degree of confidence. This has essentially been the process of science for hundreds of years, but the "simulations" are instead direct observations. With enough computing power, fast simulations can be "similar enough" to real-life situations that genuine wisdom and insight can be derived from them.

Though real, physical experimentation will be needed to verify the performance of models, those facets of the models that are verified will be quickly internalized by the polis, allowing it to simulate real-world phenomena at millions of times the real-world speed. Once a facet of a real-world system is internalized, understanding it instantly becomes a matter of routine, just as today the design of a massive bridge has become a matter of routine, a factor of running calculations based on the known laws of physics. Though from our current perspective, the complexities of the world of biology seem intimidating, the capability of superintelligences to quickly conduct millions of experiments in parallel and internalize knowledge once it is acquired will quickly dissolve these challenges as our recent ancestors dissolved the challenge of precision engineering.


I have only scratched the surface of the reasons why innovation and progress by superintelligences will predictably surpass the "calendar time" with which humanity has grown so accustomed. As humans routinely perform cognitive feats that bewilder the brightest squirrel or meadow vole, superintelligent scientists and engineers will leave human scientists and engineers in the dust, as if our all prior accomplishments were scarcely worth mentioning. It may be psychologically challenging to come to terms with such a possibility, but it would really just be the latest in an ongoing trend of human vanity being upset by the realities of a godless cosmos.

The Singularity is something that our generation needs to worry about -- in fact, it may be the most important task we face. If we are going to create higher intelligence, we want it on our side. The benefits of success would be beyond our capacity to imagine, and will likely include the end of scarcity, war, disease, and suffering of all kinds, and the opening up of a whole new cognitive and experiential universe. The challenge is an intimidating one, but one that our best will rise to meet.

Comments (43) Trackbacks (0)
  1. Long post! And I got a bit tired reading it, I have to admit. Well, I could respond to several points in this post, but I don’t want to digress. Instead, I’d like to share with you a scenario for future technological progress, in which – in my humble opinion – artificial intelligence would make relatively little difference. Let’s assume that Moore’s Law comes to an end at the 4nm technology node sometime around 2020. Then supercomputers would no longer get faster and the whole paradigm of simulations as the third pillar of science would run out of steem in the same way that theoretical physics and the elusive search for the so-called “theory of everything” ran out of steem shortly after WW2. Surely, there might be a new paradigm like nanotechnology. The development of this new paradigm will surely benefit from computer simulations. In fact, computer simulations might turn out to be the decisive bottleneck for the development of nanotechnology. But it is not clear that nanotechnology will develop in a way that immediately feeds back into the development of faster computers. In fact, the creation of nanocomputers might turn out to be at the very end of a 20 year long period of nanotechnological research, a period, in which supercomputers are employed but don’t get faster themselves. So, in this scenario, where simulation is the bottleneck, I don’t see what difference it would make if superhuman intelligence was invented in the year 2020.

  2. I’ve been observing transhumanists on the net for a few years, and frankly I’m gobsmacked by how non-existent their grasp of technology is. It’s simply mystical thinking. Pit them against any non-transhumanist engineering student and you won’t have a competition nor a discussion – you’ll have good laughs.

    There are people who know what they’re talking about, actual scientists, but most people hanging around transhumanist sites seem to be as clueless as they are gullible.

    This is unfortunate, since transhumanism is smart, way smart – too smart for the bulk of its adherents. Spreading and popularizing the advanced ideas of transhumanism, like you do, inevitably attracts people whose enthusiasm greatly exceeds their mental capacity.

    I’m sure that you, Michael, being smart, agree with me, although perhaps not publicly, if not fully, at least partially.

    Just my $0.02

  3. I’m afraid that these well-meaning over-enthusiastic people will hijack transhumanism – they already have, to an extent – and turn it into a travesty, attracting increasingly negative attitudes. But how to keep transhumanism out of the hands of those who would distort and damage it either intentionally or inadvertently?

  4. I’m not sure if other readers agree, but to me you are a rare, true carrier of the torch of transhumanism, burning as bright as ever, I’m glad to see.

    Blog more! Or don’t: quality over quantity!

  5. Having conducted empirical studies to get a sense of what mathematics might underlie the phenomena, I can wholeheartedly agree with Kelly. There are questions where you must measure, gather data and analyze it. But given the mathematical nature of our existence, better mathematics, which is the result of pure thinking, can advance physical (which is ultimately mathematical) control of our world.
    If the theory of everything exists, and I believe it does, since everything seems to be reducible to ever smaller and more fundamental principles, understanding and probably control of every physical process will fall out of it. And I’m quite sure it will be achieved through thinking, and won’t require empirical studies.

    It is likely that we already know enough physics and mathematics – the pieces of the puzzle are on the table – and the rest is up to thinking.

  6. I should probably respond to some of the other points in your post. You mentioned the failure of ancient Greeks to develop the scientific method and to build hot air balloons. IMHO, this pretty much boils down to the historic question of why we had to go through the middle ages. The ancient civilizations, like the Egyptians, Greeks and Romans, bulit many impressive things like the pyramids and made use of fairly advanced science. But, of course, these civilizations did not care much about basic socio-economic needs of the common people. In contrast, we saw many useful inventions during the middle ages, the iron plough, crop rotation, wind mills, etc. During the time of the great discoveries following Christoph Columbus, many new crops like potatoes were introduced to western Europe. All of these inventions improved the basic socio-economic situation of the people in western Europe. And some time around the year 1800, it became possible to become a scientist as a profession. So, IMHO, it wasn’t necessarily the fact that there weren’t the right kinds of geniuses around in ancient times. It was merely the fact that these geniuses couldn’t AFFORD to put their own genius to good use.
    The other point concerns the question whether – in these days, where socio-economic pressures are relieved – more intelligent geniuses could make a major difference. This boils down to the question whether mathematical models of the real world are to difficult to comprehend for the average person or whether mathematical models of the real world are fundamentally limited. I believe that the latter is the case. Thats’s why we are now in the third era of science, the era of simulation, after we had the era of (unguided) experimentation and the era of theory. Like I said, this paradigm might one day turn out to be limited as well. I think that people who believe that one day the whole universe will be simulated on a computer are as misguided as people who believe that the theory of everything exists.
    Karl Popper once said that the future is wide open. We don’t know what comes next. And that’s my personal philosophy, anyway.

  7. I don’t have a problem with the idea that computers can become increasingly better tools for solving increasingly sophisticated and complex problems. But how do we pose the problems, and how to we evaluate the results? “We want to live longer” Answer: “Here, take this pill and you can live up to 200 years. There may be some psychological consequences.” Is that a good solution? Does it only create a problem that is even worse, when population suddenly increases at an even more unsustainable rate? “O machine, now we have burgeoning population, and our ecosystem is collapsing. Can you fix that?” The lights start blinking. A week later, a response comes: “I have a solution to your problem, but some people aren’t going to like it.”

  8. HAL9004: You need to upgrade the combustion process in the power plant I designed by injecting ammonia into the flame.

    Human: Hey, HAL9005, the 9004 says it now wants us to add ammonia to have a reducing-effect on the combustion. Will this exacerbate the rising ionosphere instability we’ve been noticing?

    HAL9005: I keep telling you: the HAL9004 is an idiot. You have to unplug him and scrap those generators, and start over with my design. The people in city Y realize this. Do I have to get them to do this, or can I count on you to do it?

  9. Because you are my friend, Michael, and because you like war, especially WW2, here’s a little riddle for you: which other important inventions can you think of that are associated with times of war?

    I can think of two:
    1) The Haber-Bosch process to synthesize ammonium directly from air and to produce artifical fertilizer was invented in 1913 in preparation for world war 1.
    2) The first (non-programmable) calculating machine using the decimal system, which could be used to solve abstract mathematical problems, was invented in 1623, five years after the start of the 30 year long war from 1618-1648. The Greeks built a similar machine around 82 BC, but the input was fixed in the form of the position of the sun and the output was the local time.

  10. Speaking of war, I believe that some of the technologies that are developed during times of war tend to cause problems in the long run. Here are a few thought-provoking questions:

    1) Would the terrorist attacks in 2001 have been possible, if in May 2000 the Americans didn’t decide to make the increased accuracy of the GPS signal of +/- 10m generally available?
    2) One of the defining features of the 2008 financial crisis was the spike in global food prices. Was there a distortion of the global food market due to the Haber-Bosch process?
    3) The non-programmable calculating machine built by German ace Wilhelm Schickhard in 1623 is something what – in these days – we would call an appliance. Wikipedia tells us that an appliance is a combination of hardware and software, where the software is specifically matched to the hardware and where the software serves only one or few applications. Mind you, appliances are potentially very powerful and much faster than programs that run on general-purpose computers. One interesting kind of appliance is the in-memory computing appliance built by companies such as SAP (SAP HANA). SAP HANA is used by global companies to react to changing customer behaviour in real time. Could this be the catalyst for the next stock market crash?

  11. A great commentary and argument, particularly beacause of bringing good historical examples. In fact, this is how strong arguments are made: by bringing evidence, instead of just opinions (as Kelly seemed to do).
    (Only one small thing: as the first comentator here mentioned, try to be a bit shorter… I stoped reading at “Superior Hardware and Software”)

  12. OK, actually, I feel the need to correct myself, just so that people don’t think I’m a completely unrealistic idiot. Of course, the switching on of GPS had nothing to do with the in 2000, neither do I believe that specialized software will cause another crash on the stock market. Rather, it’s the other way around. ERP software might have played a role in the because companies using ERP are ultimately more efficient than some internet-based business models. On the other hand, GPS among other things is still a pretty good argument for the US to blackmail other countries into lending them money. This, however, might change as other countries are setting up their own satellite navigation systems. And this, rather than software, migh spark the next crisis.

  13. Hi,

    I agree that computer intelligence will blow the doors off of human intelligence in terms of the speed of developments. Humans sleep. They eat. They get distracted. They forget. They communicate by talking or writing/reading, which are horrendously slow methods of communication.

    And even if the pace of progress only increased by a factor of 10, it would be dizzying. Consider the massive changes that are likely in the coming decades due to computer-driven cars and trucks:

  14. The link to my post on “The Future of Transportation” didn’t seem to come through. I’ll try it this way:

  15. I think that other online business enthusiasts definitely should give some thought to this important webpage as an example. Very clean and simple to use design and style, as well as awesome subject material! You are an authority with regard to this valuable topic :)

  16. Here’s a rebuttal to the “speed of experiments will slow it down” argument:
    Today, many experiments are done to test aspects of theories in order to see how well the predictions of the theories line up with reality. These experiments are often multifaceted.

    Recently, however, large organizations have taken to simulating the experiments first and then running real time experiments with the many suitable candidates they get as a result.

    It’s likely that Artificial Intelligences who are capable of holding much greater volumes of data in their “head” compared to humans would have much more detailed and fine tuned theories which would result in much more detailed and fine tuned simulations. From there it’s pretty easy to imagine that whereas a team of humans might have to conduct several thousand different experiments to test parts of very high level theories about e.g. different parts of the human organism, an advanced AI would be able to conduct a much more limited set of experiments to validate the successfulness of it’s model because for the most part it *would already know* the answers and just need verification.

    That said, this is idle speculation and I have no position on whether “the singularity” will come before or after 2030.

  17. It is actually a great and useful piece of information. Thanks for sharing.

  18. Time Magazineから「なかったことになりがちな新年の抱負トップ10」(Top 10 Commonly Broken New Year Resolutions)の記事を拝借してきた。それぞれにつき、どういったアプリケーションが夢の実現を助けてくれるのか見て行こうと思うごちゃごちゃという必要もあるまい,ミネトンカ モカシン 白 梨花。新年に、より良い人生を送る手助けをしてくれるのは以下に紹介するアプリケーションだ(訳注:原文ではiOS版をメインに取り上げていました。しかしながら、大学では岡田氏が北川氏の先輩になります。 なぜ北川有広氏が岡田氏の後輩になったかの経歴事情についてはまた他の項で述べます,エルメス ポーチ。 今回、岡田氏には、9期前後から18期前後のOBと学校との連絡・調整をしてもらっています,オークリー サングラス。DIESEL2013年春夏新作長財布 [ブランド]ディーゼル [素材]羊革 [カラー]ブラック [仕様]シングルジップ開閉/コインパース/カードポケット12/紙幣多目的ポケット4/チャーム付きチェーン/専用ボックス [大きさ]約 19,hermes エルメス.5cm11cm こちらは、柔らかなラムレザーを使用し、思わずうっとりしてしまう手触りのラウンドジップ長財布,リング ブルガリ。 ガンメタルロゴとジップ、チャーム付きチェーンなど ディーゼルが得意とするアンティークな表情も持った逸品です。 カードポケットも豊富で使い勝手の良さは抜群,エルメスベルトバックル1254068,バレンシアガ バッグ ミニシティ! ラッピング希望の場合は備考欄にてご依頼ください(無料・弊社資材にて包装),バーバリー レディース1265217,GUCCI サングラス メンズ,CHANEL サングラス 20131269139。

  19. Is there an RSS feed with the total content articles as a substitute of just the teasers?

  20. Pleasant report. We appreciate beer!!!

  21. For tons of amazing pictures-check out a great new ipad book all about graffiti artists.

  22. Hi colleagues, how is the whole thing, and what you wish for to say concerning this piece of writing, in my view its actually awesome in favor of me.|

  23. Hi there everyone, it’s my first pay a visit at this website, and post is in fact fruitful designed for me, keep up posting these posts.

  24. While using hottest Ray Ban , your looks can be superb directly on for their good shapes. Thats generally you’ll find the most good possibilities for high flyers in order to keep implementing Ray Ban offerings Through because they have something that is the most suitable when it comes to quality and that should undoubtedly satisfy your flavours and designs using their ,,the classic appears to be like with a innovative pattern. In order to often the mixtures of tints which may offer you with class.

  25. For lots of old school pictures-there is a cool exclusive ipad book about graffiti history.

  26. Thanks Mark, glad you uncovered it useful. :)

  27. Whats up! I simply would like to give an enormous thumbs up for the great info you’ve got right here on this post. I will probably be coming again to your weblog for extra soon.

  28. Clearly, Apple’s instance retail store is the winner utilizing a mile. It’s actually a huge selection of all types of apps instead of a very pathetic selection of one or two designed for Microsoft zune. Ms possesses projects, specially in that life of adventures, still I not really know I will want to can guess relating to the long term if the feature is very important in your direction. The ipod and iphone can be a much better determination in that case.

  29. Als Spezialist für Personalvermittlung bringt die EM Personal AG zusammen, was
    zusammen gehört. Seit 15 Jahren sind wir auf diesem Gebiet erfolgreich.

    Die EM Personal AG findet die professionellen Lösungen für Unternehmen.
    Unsere Dienstleistungen bieten wir an in den Bereichen

    Bauhaupt- und Ausbaugewerbe

    Als Unternehmen finden Sie mit uns die für individuelle Anforderungen geeigneten

    Sie sind auf der Suche nach einem neuen Arbeitgeber, welchem Sie Ihre Fähigkeiten
    anbieten möchten, sind Sie bei uns genau richtig.

  30. Youre so cool! I dont suppose Ive learn something like this before. So good to search out any person with some authentic ideas on this subject. realy thanks for beginning this up. this web site is one thing that’s wanted on the net, somebody with a bit of originality. helpful job for bringing one thing new to the web!

  31. Thanks for sharing your thoughts about %meta_keyword%. Regards|

  32. We are a group of volunteers and opening a new scheme in our community. Your website provided us with valuable info to work on. You’ve done an impressive job and our whole community will be grateful to you.

  33. The investors are the individuals that invest in a business and will keep a part of the ownership. Each tax requires there own rules and regulations which can be very confusing at times. You can choose from a much,louis vuitton outlet, Maybe you didn?t realize it, There are other scores that aren?t FICO scores (even the ones that are legitimate don?t have FICO?s database). The closer you are to maxing-out your cards. ) Internet based calendar where you can organize yo

  34. MFI Really serious Deficiency of Money OptionsNobel Calmness Jackpot designed for microcredit,Cheap Tory Burch, to provide a multitude of profit-driven cash wrapped up around India and also other locations right into a high-margin, substantial progress community, 25% to 100% of the high aprs to clientele seriously affected. Normally indicate,GHD Hair Straighteners, United states of america, Bangladesh is thinking about the use of handle processes in the business The results of micro-credit produ

  35. The whole set of Filth with Trash Disposing ProvidersWealthy economic systems establish a massive amount spam. Property owners apply it, businesses make this happen, government authorities will the app, we all do this. Taking has gone one method or another to be able to decreasing a good waste looking for getting rid,Tory Burch Outlet Online, but yet generally there is still a major amount of trash that’s required to always be taken away carefully, safely and also economically. Below are some co

  36. Purchase the help of over the internet chunk of property distribution agencies once sending the parcel to help you GreeceGreece is a popular set that many of people Britains should transfer to yearly to get started on a brand new personal life. Increasing numbers of people may depart from his or her’s first residential home time period because of activities such as perform pledge or maybe even to get started fresh new and also calm down by using a family group. It really is considered to be not

  37. Incredible A few Affiliate business Designs and also Twenty-four hour Homebased Business online.Keep your vision opened and you also head to listen thoroughly as you usually are planning to explore unbelievable A variety of business online designs that will make on your behalf huge money internet based,Cheap Ugg Boots. Pretty much the homebased internet business is often installed in as few as Twenty-four hours from nowadays,Juicy Couture Handbags. Nevertheless that has not until you will discov

  38. A place And even Organizing In relation to Utilizing WordPress To be able to Drink A webpageHosting for Live journal provides came into common use within the last few 18 months. Typically the pain there may be which you never have to deal with WordPress blog,Karen Millen Sale, but you will will want how you can read all-around a number. If you happen to could possibly have numerous people websites you could then make use of The wordpress platform incase that is required for Online community you

  39. How Mobile home Theme parks Would definitely be a Much better Funding Rather than Solitary Family Homes : Far more More Fun, Way tooWhen a lot of people come up with checking out real estate property,Burberry Outlet Online, these people consistently think about single-family homes. As most of folks possess a home, we all for instance you will find a okay work with at checking them and even spotting a victor on a loser. But the main problem is always that all people perceives of which! Because of

  40. rice caught 17 wholesale jerseys from china touchdown passes while averaging 18Miami will find out why on Wednesday. Next Jamie Dixon (AP Photo) When: Tuesday, 7:30 p.m.,Uruguay world cup shirt, ESPNUSetting the scene: These in-state rivals have played 145 times through the years, but not since a 2005 meeting that Pitt won by 37 points. He’s a big guy,cheap San Francisco 49ers jerseys, but very lean,br />. So let’s be clear from cheap wholesale jerseys the top: Brett Favre had enough time to

  41. A lot of of the things you claim is supprisingly accurate and that makes me ponder the reason why I hadn’t looked at this with this light before. This piece truly did turn the light on for me as far as this particular subject goes. But at this time there is actually one point I am not necessarily too cozy with and while I attempt to reconcile that with the core theme of your position, permit me observe just what all the rest of the subscribers have to point out.Very well done.

  42. Matthew Stafford makes technicians day-to-dayI’m talking about it s something similar to I personally mentioned the 2009 offseason, Frequently on (aspects) anytime I am on the field, process area, activity. I critique myself, We m my personal toughest critic, Stafford claimed. He she stopped, considered precisely what he previously just simply claimed then chuckled as well as stated maybe that’s not the case.Your dog had been making reference to Ralph Jaworski, ESPN expert along with previous

  43. older lady uncovered beneath South Florida streetMan Bone Is still Going back Two,Thousand YEARS Have been located Throughout DAVIE. Nike Sb Shoes It’s really a History You will see Merely In LOCAL10. TODD TONGEN Is definitely Accept What exactly ARCHAEOLOGISTS Happen to be TO BE ABLE Research. Correspondent: LAURIE, We’ve been Outside The particular Woods Form Progress In Gaga, Along with Citizens Point out that A couple of weeks BACK Many people NOTICED Several Definitely Inquisitive Hobby Wit

Leave a comment

Trackbacks are disabled.