I think the point is that LLMs are not and cannot be an "intelligence" as commonly understood. They can string together words that might, or might not, be a good approximation of information. Since the supposed point of them is to approximate knowledgeable humans ("they can pass the bar exam!!"), the fact that they cannot be hallucination-free seems to raise red flags. What good is an "expert" that you have to second-guess and double-check when it comes to anything important?

Hacker News thread where someone makes similar points about Google's "AI Overviews".

Yeah, humans make mistakes. But everyone knows that, and that's why "I want to speak to the manager" exists. What "manager" are we going to talk to when everyone we attempt to interact with is just a LLM instance?? We all know the ancient "computers cannot make mistakes" trope that gets trotted out when there are problems. There's a rather infamous example of that still playing out in the UK.

I'm sure there are examples of LLMs that do a decent job (what used to be called "expert systems" seems related, I think), but they didn't crawl TheOnion and Reddit in an attempt to get huge and crush their competitors. The good expert systems builders fed it known-good, or at least best-effort good, information. But even then, humans always have to check the work.

Didn't someone say that once garbage info - like putting glue on pizza, or eating rocks - gets into these LLMs, then it's impossible to get it out? They basically have to retrain it with new data?

We'll see.

Cheers,
Scott.