People assert without evidence that human speech operates completely differently from how an AI generates words and phrases. But that is not at all clear, as we do not actually know how human speech functions at the level of word/meaning synthesis and expression.
Sure, humans are not operating on silicon in a datacenter, but I am not talking about the hardware substrate here.
It’s possible that AI could function completely differently, but it’s possible that it does not. We simply do not know. I suspect it’s more similar than most would want to believe, though.