EXCERPT ESSAY NICK BOSTROM It is difficult to predict how long it will take to develop human‐level artificial general intelligence. The prospect does not seem imminent. But whether it will take a couple of decades, many decades, or centuries, is probably not something that we are currently in a position to know. We should acknowledge this uncertainty by assigning some non‐trivial degree of credence to each of these possibilities. However long it takes to get from here to roughly human‐level machine intelligence, the step from there to superintelligence is likely to be much quicker. In one type of scenario, ʺthe singularity hypothesisʺ, some sufficiently advanced and easily modifiable machine intelligence (a ʺseed AIʺ) applies its wits to create a smarter version of itself. This smarter version uses its greater intelligence to improve itself even further. The process is iterative, and each cycle is faster than its predecessor. The result is an intelligence explosion. Within some very short period of time — weeks, hours — radical superintelligence is attained. Whether abrupt and singular, or more gradual and multi‐polar, the transition from human‐level to superintelligence would of pivotal significance. Superintelligence would be the last invention biological man would ever need to make, since, by definition, it would be much better at inventing than we are. All sorts of theoretically possible technologies could be developed quickly by superintelligence — advanced molecular manufacturing, medical nanotechnology, human enhancement technologies, uploading, weapons of all kinds, lifelike virtual realities, self‐replicating space‐colonizing robotic probes, and more. It would also be super‐effective at creating plans and strategies, working out philosophical problems, persuading and manipulating, and much else beside. It is an open question whether the consequences would be for the better or the worse. The potential upside is clearly enormous; but the downside includes existential risk. Humanityʹs future might one day depend on the initial conditions we create, in particular on whether we successfully design the system (e.g., the seed AIʹs goal architecture) in such a way as to make it ʺhuman‐friendlyʺ — in the best possible interpretation of that term. |