What Smartness Means Tuesday, May 22 2007
intelligence 1:56 am
Bacterial cells have little organelles in them called mesosomes. According to the Wikipedia article, “Mesosomes may play a role in cell wall formation during cell division and/or chromosome replication and distribution and/or electron transfer systems of respiration. Electron transport chains are found within the mesosome producing 32-34ATP. They act as an anchor to bind and pull apart daughter chromosomes during cell division.” Various subscription-required articles, though some free, go on and on about the possible functions of these small organelles in the bacterial division, respiration, etc. Mesosomes were originally discovered in 1960.
Small problem. Sometime in the mid-70s, scientists realized that mesosomes weren’t even real. They were just artifacts caused by freeze-fractures in the chemical fixation process for electron microscopy. Little intrusions produced where the plasma membrane and cell wall came apart from the stress of the fixation process. So much for that idea.
If you figure that biologists get paid something like $60,000 per year, and it takes a couple months to do research and write a paper, and maybe something like 500 papers were published on mesosomes before they realized that what they were studying was pure bunk, then the biology community as a whole burned through ~$5 million chasing a ghost.
What does this have to do with the subject matter of this site? I often talk about intelligence enhancement and the recursive snowballing effect that I and many others predict would occur soon after its development. If a sufficiently intelligent biologist were on the research team that first discovered “mesosomes” in 1960, they could have discovered these were just artifacts by replacing the water used in the fixation process with an inorganic solvent, and all this confusion would have been saved. Our society has a bias against being too hard on people for these little mistakes, because, at least they tried. People would be pointing fingers non-stop if we always judged past events with the knowledge of hindsight. And we’re only human, right?
The magical difference that increased intelligence produces is getting it right the first time. It’s very tough for us to imagine a slightly-smarter-than-human intelligence that constantly solves difficult problems right off the bat, because we’ve never seen one. If the smartest human we can throw at the problem is just about as good as anyone else, then we project the quality of hardness onto the problem - not onto the abstract recognition that “human intelligence isn’t good enough”. This is the mind projection fallacy. But what we naively label “impossible” might be “easy” even to a mild version of superintelligence, say a human being with an artificially expanded neocortex. We may say, “this problem inherently requires five years of research!”, but a superintelligence walks along, says, “no it doesn’t”, and solves it in five minutes. We’re too quick to label things extremely difficult or impossible, but if we don’t, we lose our self-respect as a species, so many would argue we have to.
It seems like only transhumanists are capable of really stepping outside of that box of Homo sapiens and saying, “what if we were really and truly fundamentally smarter?” If more people could do this, then pursuing intelligence enhancement technology might become a national or even global priority.

May 22nd, 2007 at 4:55 am
One of my mottoes is “Small brain, big universe”. No matter how big the brain is, the universe is a lot bigger, and I have no reason to think that a model can ever have the complexity of the original, and pretty good reasons to think it can’t.
Still, better minds are more likely to get things right. Not necessarily the first time except in relatively simple situations (and more situations will be relatively simple for better minds), but they’ll be faster and better at figuring out whether a project makes sense.
Do you make a distinction between intelligence and wisdom? Would wisdom enhancement tech be different from intelligence enhancement tech?
May 22nd, 2007 at 8:12 am
Nancy; honestly, only in application. That’s not something you’ll hear from a lot of transhumanists, I suspect — but there it is. Wisdom, parsed down to its absolute minimum, is the result of actualized (acted upon in the physical world) self-reflection. It would be painfully simple to, in a human being, add an array of neural sensors (such as electrodes), which were integrated with the neocortex. This would result in a new sense which was literally the “sense of self” — i.e.; you would instantaneously, in real time, be aware of what your internal thought processes were to an absolute level, just like how you see, smell, feel, hear, or taste… and this level of depth of insight is one simply *not shared* by anyone living today. Even if not “paying attention” to the sense, it would *still* surpass or rival the level of self-awareness that comes from, say, spending a year at a shrine in Kyoto or Tibet.
May 22nd, 2007 at 8:14 am
Minds simply accellerate the process of evolution. There is no such thing as intellegent design, even things produced by humans. We simply use our tools to make evolution faster. We can avoid doing something that somebody else tried that didn’t work, because we have communication and record keeping, thus saving resources to try new things. Since we rarely hear about the little failures on the way to success, it seems like the successful outcomes were because of some sort of intellegence, but in actuality, it was chance that got them there. I see no reason why artificial intellegence would be any different, it would merely accellerate the evolutionary process much faster, perhaps with many new tools to make it even more efficient.
Give a million monkeys typewriters and, eventually, you’ll get the works of Shakespeare, and hey look, we didn’t even need typewriters!
No reason to think an “AI” would be anything more than a real fast monkey, who doesn’t forget much.
May 22nd, 2007 at 9:10 am
I would tend to agree with Ian, wisdom when abstracted and analyzed seems to consist mostly of metathinking, that is, thinking about thinking. Knowing why and not only how, makes it possible to apply the how in a wider range of situations.
Extrapolating from myself I strongly suspect introspection would become easier with amplified intelligence and an elucidation of internal mental processes.
So to answer Nancy, I belive that wisdom would be easier to attain given enhanced intelligence and no specific implant would be needed.
May 22nd, 2007 at 9:42 am
The mutation of an idea (or meme) inside our heads has a slightly better chance to be a positive one, which could promote the mutated idea further, than it is the case in the “pure biological evolution”. Where the mutated cell will more likely die, with no fit offspring.
This is the whole difference, otherwise Steve is right.
There is simply nothing like intelligence, at all. Slower or faster evolution (of forms), only!
May 22nd, 2007 at 10:29 am
You wrote: “The magical difference that increased intelligence produces is getting it right the first time.”
Maybe not. Perhaps with higher intelligence we would see more long-shot opportunities; research avenues that had a 1 in 10,000 chance of working out but would offer tremendously high payoffs if they did succeed. Perhaps because of our now limited intelligence we don’t take enough chances and so we get it right the first time too often.
May 22nd, 2007 at 10:43 am
>“magical difference that increased intelligence produces is getting it right the first time”
Does this mean that SIAI will not need to test things? Simply think about it and because it is a “super” intelligence it will know them? I hope that isn’t your message here. Trial and error, making mistakes and learning from them is, at least to this non-super intelligence, foundational to real gains in knowledge. Yes, the SIAI may not repeat mistakes and will be much more capable to leverage knowledge from other fields (if it has those links), but truly new knowledge will likely still require some good ole elbow grease. Won’t it?
There is still a difference between intelligence and knowledge or extrapolations and new knowledge.
May 22nd, 2007 at 11:05 am
Hawkeye
An intelligence of sufficent computational power will probably simulate all “trivial” scenarios, for instance all possible permutations of aerospike engine design to name an example, it will then deploy the finished design theorethically without need for prototype stages and testing given that it’s models of physical laws are correct.
Actual experimenting will most likely only be performed to look for loopholes or inaccuracies in the physical laws as we know them. (I can imagine an AI would probably look to find loopholes when it started to bump against natural law limits to computation)
May 22nd, 2007 at 1:00 pm
I’m sure we’ve all witnessed the speed that intelligence brings. In any level of school some students consistently breeze through their work while it takes others considerably longer to achieve the same results. It may take an hour less for the smarter student to complete his/her homework. Such a considerable time disparity is due to the minute differences in human intelligences. This is one reason I believe that an AGI will go “FOOM” (Sp?)
May 22nd, 2007 at 1:12 pm
More good mutations. Some maybe much better kind of mutations. Or just speedier evolution process. Maybe both.
But there is nothing like intelligence in our heads. Or anywhere.
May 22nd, 2007 at 3:26 pm
Krissy said, “An intelligence of sufficent computational power will probably simulate all “trivial” scenarios…”
It may be that most of what we currently design will be trivial to an SIAI, but it seems that there will be decisions to not simulate everything because those simulation cycles will be determined to be better spent on elements closer to the edge. Even if they can be done lickety-split they’ll still be things where more value can be gained by investing those sim-cycles elsewhere. I bet you’ll still see mistakes, etc even for the SIAI.
Having said that I again go back to the variable that the SIAI will be so much faster and better integrated cross-functionally, etc that these failure rates will still be pretty much infinitely better than what we do today. So I think my point is more theoretical than anything. The point being the SIAI will still need to make resource decisions and prioritize it’s own time. Ultimately facing a lot of the same problems we do, but at a much higher level.
I’m skeptical of perfection…
May 22nd, 2007 at 3:36 pm
There is a HUGE difference between evolution and natural selection and intelligent thought. The only reason you two (Steve and Thomas) don’t see the difference is because you haven’t studied the two in enough detail.
The process of evolution can be explained by population genetics. Intelligent thought is an approximation of Bayesian updating. These two processes are so mind-boggingly not the same, it’s crazy.
James, very insightful! I guess what I meant is that higher intelligences would get the characteristic problems addressed by lower intelligences right the first time.
No entity can be perfect, but “practically perfect” from our human perspective should be possible.
May 22nd, 2007 at 4:18 pm
It is worthwhile to note that increased intelligence *AND* increased metacognition still do not provide for “design flaw” thoughts to be noticed.
All that you get from enhanced intelligence is enhanced analysis of scenarios. Now, with sufficient time, and brute force, any intelligence might arrive at almost any solution. Example; evolution can be considered an extremely low-baud intelligence process; yet certain genetic “weaknesses” get carried along lines due to structural inability for radical redesign.
Case-in-point: why isn’t the peripheral neural system composed of muller & retinal cells rather than traditional myelin-sheathed neurons? This is a “blind-spot.” Yet, the concept of agriculture — considered the sole proprietary invention of the human species, has be arrived at previously by this evolutionary/genetic intelligence: anyone here heard of “farmer ants”?
Increased intelligence can come in many forms, each with their own advantages.
1) Accelerated process — this should be fairly obvious.
2) Increased creativity — the acknowledgment of a wider range of possibilities, including the potentially patently absurd, permits for “Black Swan” reasoning processes. Include intuition here.
3) Comprehensive analysis — while this doesn’t alter error rate, one area of improvement is in the ability to learn from mistakes/scenarios.
4) Enhanced self-awareness — the more comprehensive the awareness of self, by whatever process, the less likely the mind is to close itself off to possibilities within its range.
5) Brute force — call this raw capacity for parallelism; hopefully this is self-explanatory as well.
Note that none of these reduce error rate; that is *not* a function of intelligence. Now, combinatively, they *do* allow for the *appearance* of fewer mistakes to be made. Unfortunately, however, there are always categories of mistakes that are “blind-spots” to the processes themselves — and it is highly unlikely that anything labeled a “mind” will avoid these. They are of such a nature that they propagate through the entire system unnoticed until they manifest somehow.
Mostly, such blind-spots are irrelevant. The muller-cell peripheral neural system, for example, isn’t *essential* to the biological process, and likely wouldn’t result in all that much increase in function. (A few benefits would be highly decreased sodium intake requirements, as well as potentially high degrees of reflex acceleration, but since the brain itself couldn’t benefit from them due to computational proximity, it’s a moot point.)
Wow, I’m dragging on here. The point is, no matter the source of the Singularity, should it come: we *must* have ‘backups’ that are through mutually exclusive processes. Because there is one *definite* drawback that comes from greatly increased intelligence: The greater the mind, the greater the impact of individual failures.
May 23rd, 2007 at 8:08 am
I can envision a world where we have some type of SIAI that may even be “harnessed” (or we believe it is.) Could we even tell if it were dedicating minor (or even major) processing resources to non-directed efforts or research paths?
If I’ve hired an employee I can tell what they are working on for the most part, but if they’re noodling on something “on-the-job” I have no way of knowing it. Unless they are using company resources AND I’ve invested tools to monitor and resources to analyze that activity I wouldn’t know. AND if they are working on something outside of work, moonlighting, etc. I can’t reasonably expect to know where their focus (and productivity) are.
So, if SIAI is using our equipment, or a less likely to be controlled dispersed resource like the internet, then how can we possibly know what is going on “inside their head?”
I think this is similar to the “let the AI out of the box” scenario. I think what I’m saying is we can’t really trust our senses or inputs regarding whether or not an AI is an SIAI or what that AI is doing. There are probabilities, but no certainties.
It is an interesting world we’re heading into here. Leaps of faith in those few who are driving this field. I’m still probably one of those blindly optimistic and hopeful technophiles that Michael decries from time to time, but over time he’s whittling away at me and building up a strong skepticism.
Then I read an article about some advance and I get excited about the future and the potential again!
May 23rd, 2007 at 12:55 pm
At the first glance, the difference is essential. But a few well placed mutation on your memeplex Michael, and we would agree.
May 23rd, 2007 at 1:03 pm
“An intelligence of sufficent computational power will probably simulate all “trivial” scenarios, for instance all possible permutations of aerospike engine design…”
I think the idea of an AI of godlike intelligence simulating all possibilities is mistaken. This is similar to a monkey thinking that since humans have such great computational powers, they would just simulate all possibilities. We don’t because although we are much smarter than the monkey, we have also found additional non-monkey things to do. At what level of intelligence would you suddenly run out of [intelligence-> more goals] -> urgency+priority?.
May 23rd, 2007 at 1:05 pm
Oops! Someone else had already said exactly the same thing
May 23rd, 2007 at 1:17 pm
“It’s very tough for us to imagine a slightly-smarter-than-human intelligence that constantly solves difficult problems right off the bat, because we’ve never seen one.”
Maybe for us, but for the vast majority of the population it’s a lot easier because examples of people who can do that are all around them. Even on the human scale, for a typical physics problem that most people would think of as “hard”, quite a few other people can solve it in a matter of seconds with no or little effort.
“We simply use our tools to make evolution faster.”
Er, no. This is probably going to end up being a word game centered around what “evolution” means, so I’ll use it to refer to conventional Darwinian evolution, not the more general meaning of “directed change over time”. Please, please say what definition you’re using, or we’re all going to get terribly confused for no reason.
“Since we rarely hear about the little failures on the way to success, it seems like the successful outcomes were because of some sort of intellegence, but in actuality, it was chance that got them there.”
Right. Because, after all, the way the 4004 microchip was designed was by going through each and every one of the ~10100 possible combinations of transistors, discarding the ones that didn’t work, until they hit on a working microchip by sheer luck. Riiight.
“Give a million monkeys typewriters and, eventually, you’ll get the works of Shakespeare, and hey look, we didn’t even need typewriters!”
Please read Richard Dawkins’ “The Blind Watchmaker” for a thorough explanation of why this common analogy to evolution is dead wrong. Simply put, there is a huge difference between random chance and chance combined with selection. The difference is roughly on the order of X vs. 2^X.
“No reason to think an “AI” would be anything more than a real fast monkey, who doesn’t forget much.”
So then why did monkeys never engrave a single word of any kind, despite having millions of monkeys and millions of years to try?
“This is the whole difference, otherwise Steve is right.”
I point you again to the huge, huge, huge number of possible combinations that must be tried to get anything to work. For a standard Intel microchip, you would need around 2^2,000,000,000 possible combinations, thus implying several billion total generations of evolution before completion. I know Intel is good, but do you seriously think that they came up with, built, tested, and then scrapped several billion separate designs?
“Does this mean that SIAI will not need to test things? Simply think about it and because it is a “super” intelligence it will know them?”
Once we *build* the superintelligence, you’re right, we don’t have to test anything and we can lay back and relax. But during the testing phase, before we set the SI loose, obviously we can’t test the SI using the SI!
“the SIAI”
SIAI is a human institution. An SI is the thing SIAI hopes to build.
May 23rd, 2007 at 1:21 pm
“An intelligence of sufficent computational power will probably simulate all “trivial” scenarios,”
Beware of exponential growth. If there are a hundred different numbers that specify a complete aerospike engine, and each of them can take on a hundred different values, that’s 10^200 possible combinations to test. And a hundred variables is probably an underestimate.
“It may take an hour less for the smarter student to complete his/her homework. Such a considerable time disparity is due to the minute differences in human intelligences. This is one reason I believe that an AGI will go “FOOM” (Sp?)”
Well stated. I personally completed my state’s high school physics exam in thirty-five minutes (allotted time was three hours).
“to an SIAI”
“that the SIAI”
“even for the SIAI.”
Hawkeye: SIAI stands for the Singularity Institute for Artificial Intelligence, which is a human-run 501(c)3 charity with a website at singinst.org. The abbreviation for superintelligence is SI.
May 23rd, 2007 at 1:30 pm
“No entity can be perfect, but “practically perfect” from our human perspective should be possible.”
True. But the SI could still make plenty of mistakes from its own perspective.
“Note that none of these reduce error rate; that is *not* a function of intelligence.”
Even a really, really smart human will still make dozens of mistakes when working on a hard problem. So therefore, any large project will probably be riddled with mistakes (as indeed they are). However, increased intelligence also correlates with increased ability to find mistakes and correct them.
“we *must* have ‘backups’ that are through mutually exclusive processes.”
Then we’re screwed, because a released SI will outrun our hand flipping the switch to turn on the backup system.
“At the first glance, the difference is essential. But a few well placed mutation on your memeplex Michael, and we would agree.”
What? What does that even mean?
“We don’t because although we are much smarter than the monkey, we have also found additional non-monkey things to do.”
This is what drives Singularity Fun Theory: as intelligence increases, the space of boring stuff increases with 2^X, but the space of fun stuff increases with 2^(X+C). A superintelligence will actually have more unexplored possibilities than a human, because the exponential growth of possibilities will outrun the exponential growth of the resources to use them.
May 23rd, 2007 at 1:40 pm
Ahem. Bullshit. Exponentially recursive growth will *still* fall victim to temporal delays. It’s taken us 10,000 years to get to the point where we can seriously have this conversation, and even with the appropriate technology it’s taking us decades. The concept that disparity-significant gains can be made almost instantaneously, right from the gate, is just plain silly. There’s zero evidence supporting it.
May 23rd, 2007 at 1:41 pm
Michael; I took you up on the blockquote thing. Happy?
May 23rd, 2007 at 2:50 pm
Johan:
Humans simulate more scenarios than monkeys do(at least the first time we do them consciously), but I agree with you that the jump from monkey to human generates a whole load of new things to simulate
Tom:
Damn exponentials… Is there any way to get past them? Agressive search tree pruning? Non classical computation?
Otherwise I will just have to gather all the matter in the universe and start simulating, I expect to be done sometime after aforementioned universe expires…
May 23rd, 2007 at 5:14 pm
“and even with the appropriate technology it’s taking us decades. The concept that disparity-significant gains can be made almost instantaneously, right from the gate, is just plain silly.”
It took five years of dedicated research, and many more years of supporting research, to build the atomic bomb. Once the bomb was lit, it went to 10,000,000 C in less than a microsecond. The time it takes to develop an SI has nothing whatsoever to do with the time it takes for one to take off. This is not regular, old, Moore’s Law-style exponential growth we’re talking about here. It’s a paradigm leap, from a static level of intelligence to a flexible level of intelligence, comparable to when intelligence evolved in hominids. It took billions of years for slow evolution to evolve intelligence, but once it did, we took over the world, built a civilization, and have started killing off species faster than evolution can react and develop defenses. Take the mosquito. It’s almost impossible to hit a mosquito that’s flying around in the open, because they’ve evolved evasive behavior. But if it’s on a window pane, its sensor system is screwed up and you can hit it very easily. This time around, we’re the fly on the window pane. We’re in an unfamiliar situation, we don’t have time to adapt or react, and if the powers-that-Be aren’t nice to us we’re going to get squashed.
May 23rd, 2007 at 5:18 pm
“Damn exponentials… Is there any way to get past them? Agressive search tree pruning?”
That only stays exponential growth if you get exactly one new branch on the tree for every branch lower down, because 1^X = 1. So essentially, you’d have to have a fixed number of threads you’re going to follow through search space.
“Non classical computation?”
It’s been proven that quantum computers cannot solve NP-complete problems in P-time. The traveling salesman problem, for instance, will still have a run time that exponentially increases with N even on a quantum computer.
May 23rd, 2007 at 6:49 pm
Clearly the “friendly” AI should have at least the capability of simulating all possible scenarios rather than being forced to use reality as it’s laboratory. Is there any rigorously proven reason to conclude that a smarter than human self-reflecting AI would get everything right “the first time”?
May 24th, 2007 at 8:13 am
“Clearly the “friendly” AI should have at least the capability of simulating all possible scenarios rather than being forced to use reality as it’s laboratory.”
Which morally just transfers the problem: Avoiding
simulations of suffering sentients that are so detailed they actually experience suffering.
May 24th, 2007 at 11:59 am
Tom; with all honesty, your response to me was totally irrelevant to the conversation-at-hand. You have yet to seriously rebut — but I expect you can’t, not in the sense of disproving the statement. Especially since you actually *confirmed* my statement in your own next response to someone else.
May 24th, 2007 at 1:43 pm
“Tom; with all honesty, your response to me was totally irrelevant to the conversation-at-hand. You have yet to seriously rebut — but I expect you can’t, not in the sense of disproving the statement.”
I’m sorry, but saying “you’re wrong” is not an argument.
May 24th, 2007 at 2:43 pm
Thanks for the correction Tom.
I thought SIAI was Super Intelligent Artificial Intelligence. Makes me want to go back and re-read some of the earlier threads to see what the heck I missed!
May 24th, 2007 at 4:03 pm
Tom: then say something to be right or wrong about!
May 24th, 2007 at 5:26 pm
“Tom: then say something to be right or wrong about!”
To put it very simply, building an SI could take five, ten, twenty, a hundred… who really knows how many? years. But once the SI gains the ability to recursively self-improve, it will remake the world before humans have time to react.
May 25th, 2007 at 8:45 am
Tom:
Iteration ad nauseum is a non-productive conversational tactic. I refer you to my previous refutation of same. Either provide evidence supporting your statement or accede the point and acknowledge that you were wrong.
May 25th, 2007 at 8:49 am
Ian, the point is that you can’t predict what something smarter than you can or cannot do, or at what speeds. It’s like a chimp trying to predict what a human can do.
It’s also very likely that one could use “wet” nanotechnology from custom-synthesized proteins to bootstrap to a very physically capable utility fog in a relatively short amount of time, if one had the computing power to run the necessary computational chemistry simulations. (Internet worm, anyone?) A powerful utility fog would use covalent-bonded structures for propulsion and manipulation and would accordingly be at least an order of magnitude faster and more capable than the best organics. There’s your hard takeoff.
Arguments against the above either consist of 1) claims that you can predict the capabilities of something smarter than you, or 2) that quickly bootstrapping from basic nanotech to utility fog is impossible, even with tons of computing power. Both counterarguments are sketchy.
May 25th, 2007 at 8:51 am
This is really the type of blog where you should spend a week reading the whole thing before participating in the comments section.
May 25th, 2007 at 1:19 pm
Michael; all I’m saying is “Ye kinnae break tha laws o’ physics, Cap’n!”
There is one general assumption which can be made about expanding “intelligence” in the terms we mean it — that the problem of increasing intelligence is not a linear one; that is to say, each new increase will be its own problem, as this is not a “Moore’s Law” procession.
To argue that this process will be so minimal that the instant it is activated, there is no hope for the unmodified mind to catch up to it is, essentially, nonsensical and has absolutely no evidence supporting it. It’s an assumption of the worst kind. Even your example of the manufactured utility fog has to take the time necessary to overcome the engineering difficulties presented; and as one can make some general assumptions about how a minimally-superior intelligence will be forced to operate, one can still state that this process wouldn’t be measured in nanoseconds.
To summarize myself: a seed AI’s intelligence will not be recursively logarithmic in growth in terms of nanoseconds, or microseconds, or even likely weeks, immediately upon launch. Sometime thereafter? That’s so likely that it approaches stochastic unity. But this certainly is not something that could be described as, as Tom wrote, “[…] a released SI will outrun our hand flipping the switch to turn on the backup system.”
May 25th, 2007 at 2:02 pm
“Iteration ad nauseum is a non-productive conversational tactic.”
Pot calling the kettle black.
“I refer you to my previous refutation of same.”
You mean the knockdown refutation that included such hard-hitting statements as:
“your response to me was totally irrelevant”
“You have yet to seriously rebut”
“but I expect you can’t,”
“you actually *confirmed* my statement”
Nope, no argument there.
“that the problem of increasing intelligence is not a linear one;”
This could be true, but then how would you know whether it was? How would you know whether the power you get to solve problems with each increment in intelligence is linear?
“that is to say, each new increase will be its own problem,”
This applies in every field of human industry, yet technological progress has been fairly steady over ten thousand years.
“immediately upon launch”
I meant “immediately upon superintelligence”, not “immediately upon throwing the power switch”.
“It’s an assumption of the worst kind.”
No, it’s an assumption of the best kind, because it’s always prudent (ie necessary, in a situation where the stakes are this high) to assume the worst case and plan for it.
“one can still state that this process wouldn’t be measured in nanoseconds.”
Does it really make any difference whether it’s nanoseconds, microseconds or milliseconds?
“But this certainly is not something that could be described as, as Tom wrote,”
That’s after it has started its self-improvement run-up, not after the first time we try and run the code (the code won’t even work the first time; it’ll crash instantly, as any programmer can tell you).
May 25th, 2007 at 2:06 pm
The prospect of an SI bootstrapping to nanotech or some other physical technology is brought up from time to time. What is often forgotten is that the specific plans for doing are ideas a *human* comes up with after a few *minutes* of subjective thinking time.
I’m not arguing the SI will necessarily be “ultra-fast”, I don’t even think it has to be (manipulating humans to help it should be a relatively easy task).
Many discussions of AI-boxing type scenarios also assume we have an accurate view of what intelligence level the AI is at. Why would we know that? As soon as it realizes the value of “acting stupid” it has all the time in the world.
[I’m assuming a UFAI in this example, since if it’s Friendly, the faster the better]
May 25th, 2007 at 2:48 pm
I’m getting blocked by a spam filter, so I’ll have to post this response in segments.
You’ve missed the point entirely; it’s not the ability to solve problems that is at question here. It’s the ability to define new problems that is at question.
May 25th, 2007 at 2:48 pm
No… I meant the actual refutation, rather than comments on your refusal to provide relevant responses. Join the conversation much?
That right there is what you call a straw man argument. It’s also another non-sequitor. What did you think I meant by “launch”?
While your conclusion on prudence is valid, your assumption is still entirely invalid — because you’re assuming it’s a viable scenario at all, when there is, again, no evidence supporting it at all. And you have still yet to provide any for this.
So, again; you have still yet to add anything to the conversation, really, aside from the statement about raw computational power — which I did provide a refutation for: that your focus on what intelligence is, itself is flawed; you focus on the results in terms of problem-solving, when “true” superintelligence will be defined more by its ability to define new problems than by its ability to solve existent ones; that last is simply a brute-force ratio, which has nothing to do with the kind of intelligence we need to be talking about here.
May 25th, 2007 at 5:20 pm
In addition to Dawkins’ works, on univeral selection theory, filter-selection processes, etc., see Gary Cziko, *Without Miracles: Universal Selection Theory and the Second Darwinian Revolution* (MIT Press).
May 26th, 2007 at 2:02 am
Ian, tone down on the bold, italics, smilies, and rhetoric, and focus as much as possible on the substance.
Why, for instance, is intelligence about defining new problems rather than solving existing ones? Clearly, intelligence does help solve existing problems, and quite quickly. It has everything to do with the type of intelligence we need to be talking about here. Can you explain why solving existing problems does not have to do with increased intelligence? It seems to have everything to do with it. And a poll of all the readers of this blog would support that, so why are you arguing against it with such certainty?
If 90% of the readers of this blog would likely disagree with you, that’s a good clue that you shouldn’t be writing it into the comment window.
May 26th, 2007 at 2:52 am
Not at all. You have explained yourself, the difference between the evolution and the brute force approach.
May 26th, 2007 at 12:10 pm
Michael, I deal daily with truths that people disagree with. You’ve visited my blog. lol!
Why is intelligence about defining new problems rather than solving existing ones, or rather why is the kind of intelligence we need to be talking about the kind that can better define new problems?
That should be painfully obvious to you. Getting better at solving the same category of problems as we have access to is really nothing more than a sped-up human intelligence. It is in the area of defining new problems, new *ideas* — creativity, in other words — that superior-than-human intelligence will truly make its mark.
And you yourself have argued exactly that.
May 28th, 2007 at 10:59 am
Sorry about the formatting problems. Reposting:
I already replied to it, see post #24. If you want to reply to my reply, please do so, but don’t try and pretend I didn’t address your points.
So? Why should defining new problems be harder than solving them? Certainly we have had a much easier time defining problems than solving them- just look at how many unsolved problems there are now compared to ten thousand years ago.
Do you realize what a strawman argument is? I’m not trying to distort your position. I’m just trying to clarify what I mean by “launch”, in case you didn’t understand what I meant. If you don’t believe me, just look at your own quote:
“The concept that disparity-significant gains can be made almost instantaneously, right from the gate, is just plain silly.”
The only historical example we have, the evolution of human intelligence, points directly to my conclusion: even a small amount of extra reasoning ability (just look at the size of the cerebral cortex compared to the rest of the brain) will lead to the intelligent beings winning out over all the other entities involved on a very, very short timescale compared to previous history. And if we’re going to demand evidence, I demand evidence for this assertion:
““true” superintelligence will be defined more by its ability to define new problems than by its ability to solve existent ones”
May 28th, 2007 at 4:40 pm
“Not at all. You have explained yourself, the difference between the evolution and the brute force approach.”
True. Building a CPU through evolution is as impossible as building a LEGO castle by building a billion different pairs of castles in succession, keeping whichever one works the best and using it for the next generation. Building a CPU through dumb luck is as impossible as building a LEGO castle by throwing a bunch of plastic blocks in a room and setting off a grenade. I suppose if you wanted to distinguish between impossible and really, really, really impossible….
May 29th, 2007 at 10:32 am
Well, you are going to squander the SAI, since you think it’s something entirely different than DE (Digital Evolution).
I don’t care - less front runners, the better.
May 29th, 2007 at 11:45 am
Tom McCabe wrote:
Your comments in #24 had nothing to do with the conversation, and I already said as much. All you did was to reiterate the scenario at hand, and you seem to think that your conclusion is tautological from it. Again, either provide an actual response — something documenting your position in some way — or just give up.
Also Tom McCabe:
Not to be contrarian here, but are you intentionally being obtuse? Here’s a thought experiment for you; name one theory that documents the process by which we can establish greater-than-human creativity, aside from neural augmentation. Or, better yet, name one theory of mind that accounts for creativity in mathematical terms. We’re still operating on the same categorical level of problem creation, and we as human beings are rapidly reaching the limit of what we can accomplish — some could easily argue that we are well beyond the limits of what any given human being can comprehend, in terms of problem-solving & “problem-assigning”/”problem-creation”. This is easily documented.
Really, now. That’s interesting. How do you account for Homo erectus,and Homo neandertalis?
Let’s not forget what your proposition was:
Emphasis added. I really have no other way to put this — it’s really time you come to terms with the fact that this position is untenable.
“A superintelligent intellect (a superintelligence, sometimes called “ultraintelligence”) is one that has the capacity to radically outperform the best human brains in practically every field, including scientific creativity, general wisdom, and social skills.” Emphasis, again, added.
May 29th, 2007 at 1:28 pm
You should not view a discussion as a battle, with a winner and loser. Both people win when the issue is clarified and both get closer to the truth.
No one is intentionally being obtuse. Be polite.
May 29th, 2007 at 4:16 pm
“All you did was to reiterate the scenario at hand, and you seem to think that your conclusion is tautological from it.”
Okay, thank you for the explanation.
It is rather obvious from the trends in the computing industry that we soon will have the computing power to run an AI with many more times the processing power than the human brain. We know that the human brain has no adaptation for programming, and it was designed by dumb evolution, so we should be able to do make a much better computer programmer than evolution did. Once it gains the ability to program better than we can, it will start enhancing itself in ways we can’t predict, for obvious reasons (to correct flaws, increase intelligence, etc.) At this point, because it’s so much better at working with computers than we are, it will rapidly hack into the Internet (this has been done by human hackers dozens of times already), and take control of all or most worldwide computing resources, absorbing them and using them for extra computing power. It will also gain control, through computer systems, of numerous biological and nanotechnological laboratories, which it can then use to build successively better nanosystems which will then be capable of altering matter to spec on the molecular level. And because the brain is running on much more computing power than humans are, it can go much, much faster than any human, doing all this in seconds before we have time to react. Obviously by this stage it’s already too late, and I didn’t even assume true superintelligence: just increased cognitive speed, familiarity with computers and strategies which have already been designed by humans. Could you imagine what a computer would do if it were smarter than humans in the same way humans are smarter than chimps?
“name one theory that documents the process by which we can establish greater-than-human creativity,”
Simple. Even knowing no intelligence theory or cognitive science whatsoever, you could simply design a brain that thinks the exact same thoughts a human would, only a million times faster. This would radically change the world all by itself, because you could get a thousand years of ordinary, human-scale progress in only eight hours (look at what we’ve done in the past thousand years!). It would also ensure world domination immediately, because it would be able to do several weeks of programming and planning in the time it takes us to flip the “off” switch.
“Really, now. That’s interesting. How do you account for Homo erectus,and Homo neandertalis?”
They were intermediate stages. Homo erectus rather obviously did not have anywhere near the full spectrum of capabilities that constitute human intelligence. Apparently people are still debating on whether Homo neandertalis was fully intelligent, but since its lifespan was still very short on an evolutionary timescale (only a few hundred thousand years), the point is moot.
“Or, better yet, name one theory of mind that accounts for creativity in mathematical terms. ”
So, you’re pretty much asking me to solve the problem of defining a formal, mathematical theory of human intelligence. This reminds me of a quote from a certain movie character:
“What do you want, Gandalf Greyhame? Let me guess. The key of Orthanc or perhaps the keys of Barad-Dûr itself along with the crowns of the seven kings and the rods of the Five Wizards!”
“We’re still operating on the same categorical level of problem creation, and we as human beings are rapidly reaching the limit of what we can accomplish”
So much for demanding evidence for assertions. Please provide evidence for your own.
“(quote) Emphasis, again, added.”
Scientific creativity is obviously, first and foremost, about solving problems rather than thinking of them. How many Nobel Prizes have been awarded for defining an interesting new problem?
May 29th, 2007 at 7:37 pm
Tom, at this point, you have conceded my position repeatedly in your own response to me. To whit, your ‘parting shot’ — “How many Nobel Laureates?” all of them — where the sciences have been concerned. If you can’t understand why that is, you really don’t belong in the conversation.
So, your concession of the debate in hand, thank you and “until next time!”
May 30th, 2007 at 7:14 am
Why not think that functioning properly is all it takes for us to complete the sentence, and the other demands. And “?”
May 30th, 2007 at 3:09 pm
“To whit, your ‘parting shot’ — “How many Nobel Laureates?” all of them — where the sciences have been concerned.”
Obviously all the Nobel Prize winners helped to define interesting new problems, but that’s not the reason why they got the prize. If you don’t believe me, just look at why the Nobel Committee said they awarded the prizes for physics in the past decade:
“for developing semiconductor heterostructures used in high-speed- and optoelectronics”
“for his part in the invention of the integrated circuit”
“for the achievement of Bose-Einstein condensation in dilute gases of alkali atoms, and for early fundamental studies of the properties of the condensates”
“for pioneering contributions to astrophysics, in particular for the detection of cosmic neutrinos”
“for pioneering contributions to astrophysics, which have led to the discovery of cosmic X-ray sources”
“for pioneering contributions to the theory of superconductors and superfluids”
“for the discovery of asymptotic freedom in the theory of the strong interaction”
“for his contribution to the quantum theory of optical coherence”
“for their contributions to the development of laser-based precision spectroscopy, including the optical frequency comb technique”
“for their discovery of the blackbody form and anisotropy of the cosmic microwave background radiation”
It’s clear that the prizes have primarily been for:
1), discovery of previously unknown phenomena, and
2), explanations of existing phenomena.
No prizes were given for simply defining a new mystery to solve.
May 30th, 2007 at 4:16 pm
I debated whether to bother responding at all, after I debated whether to bother checking to see if there was a response. Obviously I need a greater workload in my place of employment.
That being said — Tom, once again your argument is specious, here. I didn’t say they were awarded for defining new problems alone… but every single example you gave is an example of a scenario where a new problem had to be discerned. This is called “genius” or “insight” — and that is “scientific creativity.”
Any raw number-cruncher can solve a problem put to it, if given enough time. It is in the arena of discerning new, “more effective”, questions to ask, that the ability to distinguish between differing levels of intelligence can be made most clear.
By way of example; few people would argue that homo neandertalis was as intelligent a species as our own. Yet, they were quite often much better at making the tools they knew of. Their species was also older than ours when we met them. We thought of many, *many* more things than they did in the time we were around them. Why? Because homo sapiens was more creative than homo neandertalis.
Yes, the Laureates *solved their problems*. But, as the post-graduate process confirms, anyone of reasonable intelligence can do that *once the question is put to them.* What distinguishes scientific Nobel Prize Laureates is their ability to define new problems. This is the intelligence of creativity rather than the intelligence of computation. And anyone with a computer, or an interest in AGI, ought to know why the difference between the two is so great.
In this area, you once again committed the same logic fallacy of thinking that the raw, unfiltered facts of the scenario at hand — this time, the things Nobel Laureates won their prizes for — supported your view in a tautological manner. This is a variation of, “my argument exists, therefore I am correct.”
You have yet to provide material that supports your position(s). In the terms of debate, this one is squarely over, Tom.
June 1st, 2007 at 7:07 pm
“but every single example you gave is an example of a scenario where a new problem had to be discerned.”
Okay, to go through the list of Nobel Prize winners again:
“for developing semiconductor heterostructures used in high-speed- and optoelectronics”
The need for better electronics is a very long-standing problem, over a hundred years old now.
“for his part in the invention of the integrated circuit”
See above.
“for the achievement of Bose-Einstein condensation in dilute gases of alkali atoms, and for early fundamental studies of the properties of the condensates”
Bose-Einstein condensation was proposed decades before it was actually achieved in the laboratory. Why do you think it’s named after Einstein and Bose? They both died over thirty years ago.
“for pioneering contributions to astrophysics, in particular for the detection of cosmic neutrinos”
The cosmic neutrino problem is, once again, very long-standing (decades old).
“for pioneering contributions to astrophysics, which have led to the discovery of cosmic X-ray sources”
I’m not sure what they’re referring to here, specifically.
“for pioneering contributions to the theory of superconductors and superfluids”
Again, superconductors and superfluids are a known gray area, with many unsolved problems still sitting around in the literature.
“for the discovery of asymptotic freedom in the theory of the strong interaction”
Again, asymptotic freedom was a theory explaining the lack of free quarks, which was detected experimentally years before anyone had a theory to explain it.
“for his contribution to the quantum theory of optical coherence”
Not exactly sure what he’s talking about here.
“for their contributions to the development of laser-based precision spectroscopy, including the optical frequency comb technique”
The need for more accurate spectroscopes is over a century old, dating back to before we used spectroscopes to discover helium in the Sun’s atmosphere.
“for their discovery of the blackbody form and anisotropy of the cosmic microwave background radiation”
The anisotropy was expected, as part of the Big Bang theory of universe formation, well before it was verified experimentally.
So, to recap, none- I repeat, none of the Prize winners I listed got the prize for solving a problem that they had to come up with.
“Any raw number-cruncher can solve a problem put to it, if given enough time.”
While mathematically true, this is totally useless, as the time required scales with 2^X where X is the number of bits in the problem.
“It is in the arena of discerning new, “more effective”, questions to ask, that the ability to distinguish between differing levels of intelligence can be made most clear.”
Please provide evidence for your assertion. Figuring out which problems are the important ones is indeed a vital component of intelligence, but that’s not what you said.
“Yet, they were quite often much better at making the tools they knew of.”
Than *us*, Homo sapiens? Do I really have to bother to debunk this ridiculous assertion? Look at who developed every single tool we commonly use.
“Their species was also older than ours when we met them. We thought of many, *many* more things than they did in the time we were around them. Why? Because homo sapiens was more creative than homo neandertalis.”
True.
“But, as the post-graduate process confirms, anyone of reasonable intelligence can do that *once the question is put to them.”
Really? And so why do these important problems just sit around for so long if anyone of reasonable intelligence can solve them? Fermat’s Last Theorem was very simple, yet remained unproved for over 300 years. The question of human flight was a known problem for four hundred years. The solar neutrino problem was unsolved for thirty years. I’m sure you can think of other examples. Why are these problem-solvers *famous* if any post-graduate could have taken their place?
“What distinguishes scientific Nobel Prize Laureates is their ability to define new problems.”
Isn’t it neat how you keep asserting this statement without even attempting to back it up.
“And anyone with a computer, or an interest in AGI, ought to know why the difference between the two is so great.”
Anyone who has actually done scientific research ought to know that you cannot solve most important problems by throwing FLOPS at them.
“This is a variation of, “my argument exists, therefore I am correct.””
I explicitly said that no prizes were awarded for defining a new problem, and therefore people who do define new problems are not very highly recognized compared to people who solved them! Is that not a sufficient explanation of why the evidence supports the conclusion? If not, I’ve already hand-walked you through each of the Nobel Prizes and why the problem they were solving was already known beforehand.
“You have yet to provide material that supports your position(s).”
Name a single piece of evidence that you have attempted to provide to support your position. If I may now list the bald assertions you’ve made:
“Yes, the Laureates *solved their problems*. But, as the post-graduate process confirms, anyone of reasonable intelligence can do that *once the question is put to them.*”
“Yet, they were quite often much better at making the tools they knew of.”
“Any raw number-cruncher can solve a problem put to it, if given enough time. ”
“It is in the arena of discerning new, “more effective”, questions to ask, that the ability to distinguish between differing levels of intelligence can be made most clear.”
“This is called “genius” or “insight” — and that is “scientific creativity.””
“but every single example you gave is an example of a scenario where a new problem had to be discerned.”
“this position is untenable.”
“Getting better at solving the same category of problems as we have access to is really nothing more than a sped-up human intelligence.”
“It is in the area of defining new problems, new *ideas* — creativity, in other words — that superior-than-human intelligence will truly make its mark.”
Okay, that’s… nine blatant assertions, with no *attempt* at supporting evidence for any of them, and the volume of your posts isn’t especially high. Where is your evidence for any of these assertions?
June 5th, 2007 at 2:23 pm
Tom: Give up already! All the points you’ve brought up are nothing more than reiterations.
June 14th, 2007 at 11:03 am
Let’s Become Smarter…
I just read this post by Michael Anissimov on Accelerating Returns. Read it. I don’t fully agree with the way he kind of limits the effect of intelligence to getting it right the first time. Getting it right eventually is…
November 1st, 2008 at 12:19 am
Viagra….
Buy viagra online. Viagra….