This is really good , and the physicalist vs singularist division is a framing I suspect I’ll find myself using in the future. I made similar but much less coherently-expressed complaints here . There are two things I’d now like to add.
First, the nanotech argument is more ridiculous than Tim acknowledges. Not only, as he notes, is no serious scientist investigating it; not only is King Charles the closest thing to a public intellectual the movement has; but we have strong existence proofs of its implausibility: bacteria. The world is blanketed in self-assembling nanomachines that diligently harvest environmental energy sources to replicate themselves. There are an estimated five million trillion trillion of them, competing under constant
Second, and more nascently: I’m less sure that we’re on the cusp of AI than I used to be. Generative transformers are very impressive. They can do things that humans can’t. They’re improving very rapidly–not only in quality but training costs–and well-known problems like hallucination seem tractable. I’m even cheered by the analytic work surrounding them, as teams compare different models using rigorous procedures that often encompass aspects of the Alignment Problem and which, while perhaps incomplete, see