Sometimes you find the perfect paragraph in the most unexpected of places. Kevin Drum summarizes very succinctly why AI isn’t this magically impossible thing.

This, by the way, is why I’m so generally bullish on artificial intelligence. It’s not because I have such a high opinion of computers, but because I have such a low opinion of humans. We really are just overclocked chimpanzees who have convinced ourselves that our weird jumble of largely Pavlovian behaviors—punctuated by regrettably rare dollops of intelligence—is deeply ineffable and therefore resistant to true understanding. Why do we believe this? Primarily for the amusingly oxymoronic reason that we aren’t smart enough to understand our own brains. The silicon crowd should be able to do better before long.

“Largely Pavlovian behaviors” sums it up pretty well. We are complex, but not as complex as we think we are. And a lot of human behavior is easily predictable, despite what you’ve read.

The main problem (in my view as a non-expert, ahem) is that AI research has been math-proof focused rather than evolutionarily focused.  No one will write an AI. But it might write itself.

Math-proof focused is narrow and hopeless and will never lead to AI. Other paths will.