Read Aubrey’s 8-page paper “The singularity and the Methuselarity: similarities and differences” at the SENS Foundation website. The arguments are quite subtle and complex at points, providing a lot to chew on. Here’s a quote:
Let us now consider the aftermath of a â€œsuccessfulâ€ singularity, i.e. one in which recursively self-improving systems exist and have duly improved themselves out of sight, but have been built in such a way that they permanently remain â€œfriendlyâ€ to us. It is legitimate to wonder what would happen next, albeit that to do so is in defiance of Vinge. While very little can confidently be said, I feel able to make one prediction: that our electronic guardians and minions will not be making their superintelligence terribly conspicuous to us. If we can define â€œfriendly AIâ€ as AI that permits us as a species to follow our preferred, presumably familiarly dawdling, trajectory of progress, and yet also to maintain our self-image, it will probably do the overwhelming majority of its work in the background, mysteriously keeping things the way we want them without worrying us about how itâ€™s doing it. We may dimly notice the statistically implausible occurrence of hurricanes only in entirely unpopulated regions, of sufficiently deep snow in just the right places to save the lives of reckless mountaineers, and so on â€“ but we will not dwell on it, and quite soon we will take it for granted.
Shades of The Future Might Be Like the Past… At First.