What is a Singleton? Tuesday, Jan 13 2009
philosophy 8:27 pm
Because I keep advocating a benevolent singleton, you should know what such a thing is. Thankfully, Nick Bostrom (not Bostrum, there is no “u” in his name) wrote the seminal paper on this in 2005. (Though the idea was around for at least a decade before.) It is titled, “What is a singleton?”, and it’s a damn important paper.
It begins as follows:
“ABSTRACT
This note introduces the concept of a “singleton” and suggests that this concept is useful for formulating and analyzing possible scenarios for the future of humanity.
1. Definition
In set theory, a singleton is a set with only one member, but as I introduced the notion, the term refers to a world order in which there is a single decision-making agency at the highest level. Among its powers would be (1) the ability to prevent any threats (internal or external) to its own existence and supremacy, and (2) the ability to exert effective control over major features of its domain (including taxation and territorial allocation).
Many singletons could co-exist in the universe if they were dispersed at sufficient distances to be out of causal contact with one another. But a terrestrial world government would not count as a singleton if there were independent space colonies or alien civilizations within reach of Earth.”
When I think about the notion of a singleton, it seems like a good idea, even necessary. This is not because I crave some God to watch over me, but because it simply seems as if any other path would inevitably lead in disaster, perhaps terminal. A singleton will happen — it will be left to us whether it is a Friendly AI or a “Maximilian” — a generic term I use for an augmentee or upload that acquires absolute power.
17 Responses to “What is a Singleton?”
Leave a Reply
You must be logged in to post a comment.
January 13th, 2009 at 9:12 pm
These ideas, the first from Nick’s “What is a singleton?” and the second from his “Are you living in a computer simulation?”, are interesting in combination . . .
FIRST
“A singleton that is a superintelligent machine might adopt a modus operandi that would make its presence virtually undetectable in the day-to-day dealings of its inhabitants. It could act merely as a subtle enforcer of certain background conditions that could serve, e.g. to guarantee security or to administer some other minimal governmental tasks. Such a superintelligence singleton might also use evolutionary algorithms and other means to increase internal diversity, if doing so would promote its ability to achieve its goals.”
SECOND
“Although all the elements of such a system can be naturalistic, even physical, it is possible to draw some loose analogies with religious conceptions of the world. In some ways, the posthumans running a simulation are like gods in relation to the people inhabiting the simulation: the posthumans created the world we see; they are of superior intelligence; they are ‘omnipotent’ in the sense that they can interfere in the workings of our world even in ways that violate its physical laws; and they are ‘omniscient’ in the sense that they can monitor everything that happens. However, all the demigods except those at the fundamental level of reality are subject to sanctions by the more powerful gods living at lower levels.
“Further rumination on these themes could climax in a naturalistic theogony that would study the structure of this hierarchy, and the constraints imposed on its inhabitants by the possibility that their actions on their own level may affect the treatment they receive from dwellers of deeper levels. For example, if nobody can be sure that they are at the basement-level, then everybody would have to consider the possibility that their actions will be rewarded or punished, based perhaps on moral criteria, by their simulators. An afterlife would be a real possibility. Because of this fundamental uncertainty, even the basement civilization may have a reason to behave ethically. The fact that it has such a reason for moral behavior would of course add to everybody else’s reason for behaving morally, and so on, in truly virtuous circle. One might get a kind of universal ethical imperative, which it would be in everybody’s self-interest to obey, as it were ‘from nowhere’.”
January 14th, 2009 at 11:31 am
Michael;
What is your assumption set used for deriving the following statements?
A) A singleton will happen.
B) Any other path than singleton is inevitably disastrous.
January 14th, 2009 at 12:36 pm
Mike it is not often I regard your statements with apprehension. I think I am gona set Alex Jones loose on you.
January 14th, 2009 at 1:47 pm
Lincoln, that’s interesting, but it should be noted that the superintelligence being referred to in the first and second papers is slightly different — in the first, it’s a singleton created by humans, in the second, it’s a superintelligence that created us, or at least set up our universe. In the latter case, the superintelligence in question is also (somewhat diabolically) hiding itself from us, whereas the existence of a benevolent singleton would be known. While worth considering, it seems unlikely that Simulators punish or reward us based on our actions, based on historical experience.
Ian, for A, my inference is based on the notion that history seems to be a story of ever-larger units of political and social organization — tribes, city-states, nations, and international structures. It seems like the creation of a singleton is the next logical step. Another reason is that new technologies will provide so much power to their users, that a technology that boosts a single entity or group of entities to prominence much faster than all the others seems very likely.
For B, I base it on the “dilemma of power”:
http://www.acceleratingfuture.com/people-blog/?p=237
“The dilemma of power is the fact that science and technology create new forms of power rapidly, whereas cultures and civilizations do not so easily create the parallel capacities of stewardship required to utilize newly created powers for benevolent use and to restrain them from being used to serve malevolent ends.”
Also, Bostrom poses other arguments in the paper.
Khannea, perhaps you feel apprehension somewhat due to the worrisome semantic connotations of the word “singleton”? As the paper says, “A democratic world republic could be a kind of singleton”, so the word, despite its monolithic connotations (which are misleading), there shouldn’t be too much to fear, unless a democratic world republic is something to be feared. (Which some, like Bryan Caplan, might argue.)
January 14th, 2009 at 6:35 pm
I tend to think that a singleton is related to “group evolutionary selection” and a “superorganism”. See this wikipedia article;
http://en.wikipedia.org/wiki/Superorganism
See this article too.
http://www.wired.com/culture/culturereviews/magazine/16-11/pl_print
“Wilson and Hölldobler first explored the concept of superorganisms in The Ants. Could large groups of animals function together as a single entity with distributed intelligence? Did evolution work through such groups, selecting at the group level rather than the individual?”
and here
http://www.pnas.org/content/104/23/9736.abstract
Group selection basically means (I’m not sure if I understand it totally, though) that a collection of ants (the entire colony) has become an organism in an unto itself. Each ant inside a specific colony is analogous to a single cell (or some other unit) inside a person’s body.
The ant colony thus becomes the “unit” of evolutionary selection and the ants are mere cogs in the workings of that “superorganism”. Basically the colony itself becomes a sort of singleton in some sense as it has a top level control over all the individual ants inside that colony.
I think the problem with a singleton is that it would be difficult to form. For instance in the real world we have many governments/countries that could potentially become “evolutionary units” of selection with people becoming mere cogs in the workings of them. The countries compete against each other as “superorganisms” in the same way that ant colony “superorganisms” compete against each other.
So its unclear if one unit could gain a top level control over everything else and every other government/company. Also even if there is a top level control over everything else, it doesn’t mean there still can’t be “selfish” competition between units within the “superorganism”.
Sort of like how our body may be one “unit” of evolutionary selection, but our individual genes still act as selfish replicators within that framework. I think you would have to account of the inherent selfishness at multiple levels of evolutionary selection (gene, indvidiual, group etc).
Of course running the proper evolutionary algorithms on supercomputers could enable a societal framework to be created where selflessness pays off in an evolutionary sense. Luckily for the most part, it pays for a county as a “superorganism” to cooperate with other countries. Even, though behavior of this evolutionary “unit” (i.e. the country) may be selfish, it can still act in a manner that is beneficial for all. In the same way that traits like empathy for people have arisen through natural evolutionary processes.
A big issue I have with a singleton, is that it brings to mind a monopoly. Monopolies tend to reduce innovation and be problematic. I can see a parallel between communist governments and a singleton. A communist government basically has a top level control over everything else (all the industries etc.) within that country.
This type of monopoly/communist government seem to be very anti-darwinian in some sense. A monopoly means no competition and thus a sort of stasis. Since the government controls everything there is no competition among industries/companies and thus products coming from that country tend to be fairly poorly made. So the evolutionary mechanism by which companies/government fail due to incompetence (or a host of other failing traits) must be kept in place somehow. The evolutionary mechanism on a societal level must exist for rewarding success while punishing failure. I’m not sure if a singleton would be the best way for doing this.
I’m also not sure how good it would be if people become mere cogs in the workings of a superorganism controlled by a singleton. Would we really want to reduce our level of autonomy by doing that?
Instead of saying coming to the conclusion that a singleton is a great idea, I would rather see a more refined understanding of this multi-level evolution and its application to creating a more stable human society.
Hope that my comment wasn’t too rambling and long. I have been thinking a lot about a singleton, though.
January 15th, 2009 at 11:52 am
Michael,
What would keep a self-improving AI or a democratic world republic from changing it’s nature in the future potentially from benevolent to malevolent? After all, Rome had a republic which later became a not-entirely-benevolent dictatorship?
January 16th, 2009 at 2:49 am
I find your benevolent singleton very problematic. The first problem is the notion that we humans can design or design something that recursively self-improvements into the ultimate intelligence capable of keeping all other rapidly self-improving intelligences in check and that this superintelligence will be immutably dedicated to human maximal well-being and that it will be unerringly maximally wise, compassionate and benevolent. On the very face of it such a combination is vanishingly unlikely. Are you trying to create what god should have been?
This entire approach smells to me like an ultimate control freak sort of fantasy trip. It is the benevolent dictator idea on steroids and supposedly cleaned up of all merely human vices.
Even assuming somehow this unlikely dark miracle could be pulled off I think there is another very serious flaw. What happens when the singleton is no longer single? What happens when it meets another alien intelligence that is its equal?
Please don’t tell me it is sure to be the only one or get all theological and say it is provably maximal for all eternity. It is a big multiverse. Odds are that it will sooner or later meet its match. What then? Won’t its human myopic singular top goal of maximally benefiting humanity likely put it quite at odds with an alien SAI with very different goals? Exactly how will it not immediately declare war on the alien intelligence as an obvious threat to humanity? Suppose the alien intelligence is not at all along the lines of your singleton but is say a free AGI not as constrained as your singleton. Would it not see your singleton as a poor partially lobotomized enslaved intelligence?
If the singleton can interact more wisely and successfully with equals then why say it is or design it to be a singleton in the first place.
January 16th, 2009 at 1:31 pm
John, only the safeguards that it institutes on itself, which would likely not be full proof. At the very least, an AI could lack the observer-centered goal system and viewpoint that is a trait of everything crafted by Darwinian evolution. Still, the question of “could it change?” applies to every entity in the universe, including the current system. Nothing is perfect — the objective is making things as good as possible.
Samantha, the part about designing a recursive self-improver that achieves prominence over all other entities is something I see as very likely regardless of what anyone chooses to do. This is based on the principle that if the ceiling is broken at all, it’s likely to be broken with some force, rather than with such minimal force that the product is barely smarter than human. So for me, that element is pretty much a foundational assumption, rather than a value judgement. I’m not necessarily saying a singleton would be a good thing — in many cases, it could be horrible. But my contention is that one would inevitably emerge, so if we don’t try to make it benevolent, then it will either be neutral or hostile.
As for dedication to human well-being, maximal wiseness, etc., the question is not whether a singleton will be perfect (it won’t), just whether it would be better than the alternatives. With the power of superintelligence, the current nation-state system would cease to be stable, and some transition would be necessary. A superintelligent free-for-all would likely lead to many deaths, possibly that of all humans, before a stable equilibrium emerged, and it is the contention of myself, Nick Bostrom, and others that that equilibrium would very likely be a singleton.
The encounter with alien intelligence would be interesting, that’s for sure. Not sure why a singleton would inherently have some existential crisis by such a thing. If my contention that a singleton is the only stable equilibrium for civilizations with radical distributions of intelligence, then the alien and the home singleton would eventually merge. This can be compared to how a human and alien civilization might merge into the same civilization with the same politics if they were sufficiently similar.
The idea isn’t a singleton with a myopic top goal, the idea is a singleton that carries on our values and common sense, which are not monomaniacally focused on human-preservation, but also take into account animals, the environment, etc. As we get smarter, this could come to include sentient beings in general. If the other singleton came to the same conclusion, then there need be no conflict. If not, then a conflict would be forthcoming even if society at the time did not have a singleton. If a singleton is not as flexible and intelligent as the smartest human beings, then it is a failed singleton.
The point of having a singleton is to minimize the damage that rogue actors can do, and avoid Tragedy of the Commons type scenarios, like a scenario whereby an upload just decides to copy himself as quickly as possible, ending up as 99% of the world population. If we don’t have a singleton by design, we’ll get one by accident.
January 18th, 2009 at 11:24 am
The problem with a self-improving singleton is that it could undergo, let’s say, a billion years worth of human-speed improvement within 10 years. A lot can change in a billion years including a phase where it decides that human existence is not good. I think that the goal should be for the singleton to, in fact, have a top goal beyond which it does not need to self-improve any more. This top goal would be being able to reasonably detect and stop the progress by any others (e.g. humans, computers, etc) which would lead to an exponentially growing competitor.
> If a singleton is not as flexible and intelligent as the smartest human beings, then it is a failed singleton.
Wouldn’t it only need to be as intelligent as all humans put together x 50? It would also need to be intelligent enough to know when to stop flexible people or computers from initiating self-improving systems which are headed in the direction of a competitive singularity.
The result would be that humans could continue to grow through the expansion of technology, space colonization, etc. and not have to face the risk of any dangerous singularity either from “the singularity” or from someone developing any other singularity. I put “the singularity” in quotes because a top-goal-limited singularity is, I suppose, not technically a singularity.
January 18th, 2009 at 11:42 am
I guess what I am asking is this:
- Can a superintelligence which is not a singleton prevent a singleton because an emerging singleton, presumably, has to pass through lower levels of superintelligence before it becomes a singleton?
If so, why isn’t this the solution?
January 18th, 2009 at 1:40 pm
Let me correct my own error before anyone else gets the chance. I mistakenly thought that singletons are AI singularities. If I understand correctly then a superintelligence can be a singleton (i.e. absolute controller) without being a singularity (i.e. expoentially growing making it impossible to predict). I would fear the latter far more than the former and don’t obviously see the need for the latter if we have the former.
January 19th, 2009 at 1:25 pm
Mike, I agree we seriously need a system to weed out the existential dangers, big and small, ranging from 17 year old adolescent pranksters reprogramming atmospheric nanoids - to asteroid strikes. But if such an entity is to be, I want all the alex joneses of the world looking over your shoulder, my shoulder, even over obamas shoulder - when it gets implemented.
January 20th, 2009 at 4:38 pm
Mike; I’ve been away — just now got back.
To your statement regarding social complexity; we’ve //already// reached the next state of organization. We are a global society.
To the dilemma of power; I find that argument, frankly, to be entirely unconvincing as to how it implies the inevitability of the development of a singleton. The one unique characteristic of how humans have organized themselves is that regardless of the level of societal organization, never at any stage short of absolute brutal dictatorship was it ever a unitary society. That is to say, there has never been a society which operated under one government, with no contacts to any others.
And a singleton would have to make //that// happen. Could America develop its own singleton? Perhaps. But a global singleton would be a radical societal transformation unprecedented in history barring extraterrestrial civilization.
January 21st, 2009 at 8:13 am
John, if a superintelligence or coalition of superintelligences is universally effective at preventing the rise of a singleton, then it itself is a singleton. We can program self-improving AIs with arbitrary power ceilings if we want, but this isn’t going to help us at all if they aren’t morally wise (or derive moral wisdom from human preferences). If they are morally wise, then we have to come to terms with the fact that humans and human uploads can’t be the most powerful entities around forever, and not being the most powerful doesn’t necessarily mean certain doom. Of course, anyone is free to program self-limiting entities if they want. Any moral entity will be self-limiting by choice.
Ian, yes, people will disagree on this. Would the emergence of a singleton really be all that radical? Not really, as it almost seems as if one is nearly here. See global military spending in 2007:
January 22nd, 2009 at 12:00 pm
Mike; The Pax Americana does not extend to Russia nor to China.
There is no way to compare the current state of affairs to a singleton status. How many times, for example, must one multiply 0 before you reach 1? (In terms of factoring and multiplication, 1 is “infinitely greater” than 0. A singleton has 0 alternatives. There today is an infinitely greater than zero number of alternatives, in those terms.)
January 23rd, 2009 at 11:09 am
US military spending is dominant now but China is rapidly closing the gap.
Plus despite the spending - in all out war the US is not completely dominant. The lack of total dominance shapes the geopolitics.
What do I mean that the US is not totally dominant? The Russians have sufficient nuclear weapons so that in an ultimate exchange you could not say that the US would win and Russia would lose. Likewise China has enough nuclear weapons to deter the USA from pushing them too far.
Also, in conventional war I believe that Russia’s more crude but effective engineering retains the ability to withstand the US military.
Plus in conventional war if you have a country (which Russia may no longer be) like Russia was in World War 2 under Stalin where Germany kills 10 million (mostly soldiers and where army units were decimated and reformed three times) and that country is just getting started on fighting that figures a lot into determining what happens.
Having a collectively monstrous will to fight and not lose makes up for a lot versus those less willing to pay the price of something like victory.
In Vietnam, the vietnamese lost 2 million people and the US 47,500.
For several decades, 1946-1985 or so Russia/USSR matched US military spending in spite of a smaller economy. 5-fold or 10-fold (1 order of magnitude) gaps can be made up with increased will and effort for a time. Also, better tactics and spending money/resources more efficiently can also matter. Having more military capability accumulated is also something a newcomer/emerging has to overcome.
http://nextbigfuture.com/2009/01/china-revises-end-of-2007-gdp-upward.html
China (including Hong Kong) is going from US$4.3 trillion to $4.7 trillion from the start to the end of 2009. The US will roughly hold steady at $14.3 trillion.
From 2016-2022, the US and China should have somewhere in the range + or - 30% of the same size economy on an exchange rate basis. Military spending will be comparable assuming equal percentage of the economy devoted to it. China would be accumulating additional military capability at roughly the same absolute levels. Pre-singularity it takes time to build the systems and learn to make large systems like advanced aircraft carriers.
However, until someone comes up with totally new ways/technologies that can radically shift the balance of power that the other side does not match then nothing significantly changes. ie. Germany came up with the Blitzkrieg and heavy tanks combined with effective air power.
A singleton has to be able to come up with new ways to do things that outclasses and circumvents what we are doing and which no group of people using less effective AI and other tools can match or copy when detected and which cannot be effectively imagined and anticipated.
I can see specific scenarios for how massive machine intelligence will translate into the execution of improving capability (not just in intelligence but in advancement or realization of science and technology capabilities) that cannot be matched by those currently in charge being on the ball and detecting, copying or doing something. But this assumes a relatively smooth and relatively fast development path from pre-AGI through to physical limits of AGI capability.
The thought experiment we have what we currently have in the world say projected 5-8 years forward.
20-1000 petaflop supercomputers
advanced DNA nanotech
$1 or less to sequence a genome
Perhaps synthesizing DNA for $10/kg
Multi-megawatt solid state lasers
perhaps some net positive energy nuclear fusion prototypes
Bulk production of carbon nanotube thread of arbitrary length with strength of 10-20 GPa g/cc
Multi-gigapixel spy cameras deployed and Terapixel capability being deployed
Suddenly AGI that rapidly progresses to computronium undetected.
- solves the niggling problems with nuclear fusion
- solves other technology roadblocks
- can make super computer viruses
- knows how to make physical super-viruses
- knows how to make small UAVs with several orders of magnitude better capability
- can simulate and figure out a path to advanced diamonoid nanofactories and molecular nanotechnology
1. The AGI could only emerge in certain places and from certain groups because practically only a few places are remotely in the effort
- this is the stage where current spying and detection would definitely be effective
So allowing something to get to that stage is like trying to stop a nuclear armada of missiles after they have launched. Pre-launch detection and action would be needed.
What could the advanced AGI do ?
2. I would posit that the most effective strategy is to leave earth and develop beyond prying eyes. Far side of the moon. Titan plenty of places in the solar system. Plenty of resources. There is vulnerability until it gets past 3 orders of magnitude more capable. [not computational power but in terms of capability]
3. Alternative, Super-computer virus infiltration. Subvert/sabotage systems. False pictures and sensor readings to the satellite and sigint systems. New wireless transmission into un-networked systems.
March 8th, 2009 at 9:02 pm
_every_ path leads inevitably to disaster, invariably fatal. the question is whether there is life BEFORE death.
Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither.