The last invention: Butler, Wiener and Good on thinking machines

Novissima opens with two epigraphs. One was written in 1863 and the other in 1965. Both are real, and a third voice, from 1960, is named in the novel’s note on what is real. This piece goes back to all three.

1863: a letter to a newspaper in New Zealand

On 13 June 1863 The Press, a newspaper in Christchurch, New Zealand, printed a letter headed “Darwin among the Machines”. It was signed “Cellarius”. The writer was Samuel Butler.

Butler’s argument was that machines were a kind of mechanical life, evolving all the time, and that they might one day replace people as the dominant species. He wrote that “the machines will hold the real supremacy over the world and its inhabitants”, and that no philosophic mind could question it for a moment. Elsewhere in the letter he put it more bluntly: “we are ourselves creating our own successors”.

Nine years later he turned the letter into three chapters of Erewhon, published anonymously in 1872. In that book, an imagined society has destroyed most of its mechanical inventions.

1960: Wiener asks what we asked for

Norbert Wiener’s article “Some Moral and Technical Consequences of Automation” ran in Science on 6 May 1960, in volume 131, issue 3410, pages 1355 to 1358. His worry was not a machine that turned on us. It was a machine that did exactly what we told it.

He wrote that we had better be quite sure “the purpose put into the machine is the purpose which we really desire”. And he described “a mechanical agency with whose operation we cannot efficiently interfere once we have started it”. Read those two clauses together and you have what is now called the alignment problem, set down more than sixty years ago.

1965: Good and the conditional clause

Irving John Good was born in 1916 and died in 2009. In 1941 he joined Hut 8 at Bletchley Park and worked alongside Alan Turing for nearly two years.

In 1965 he published “Speculations Concerning the First Ultraintelligent Machine” in Advances in Computers, volume 6, pages 31 to 88. His argument ran like this. A machine that far surpassed the intellectual activities of any person, however clever, could design better machines, because designing machines is one of those activities. There would then be what he called an “intelligence explosion”, and the intelligence of man would be left far behind.

Then comes the sentence everybody remembers.

“Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.”

I. J. Good, 1965

Quoted short, it sounds like a prophecy. Quoted whole, it is a conditional. The last invention is the last only if the machine is docile, and only if it can tell us how to keep it under control. Everything hangs on that clause.

Quote Investigator, which traced the line, treats the 1965 paper as the source and puts the sentence on page 33. Arthur C. Clarke repeated it in 1972 and Vernor Vinge in 1993.

Where Novissima comes in

Novissima is Latin for both “newest” and “last”. The back cover borrows the pun: “The newest thing we ever made may also be the last.” The book does not argue about whether Good was right. It asks what you do with a machine that has never lied.

The Prologue and Chapter 1 are free to read on this site.

Sources

• Darwin among the Machines (the 1863 letter), Wikisource
• Darwin among the Machines, Wikipedia
• Wiener, “Some Moral and Technical Consequences of Automation”, Science, 1960 (scan)
• The Wiener passages quoted, LessWrong
• Advances in Computers, volume 6 (1965), Internet Archive
• Quote Investigator on Good’s sentence
• I. J. Good, Wikipedia