Anthony on Nostr: I don't know for sure, but the large number of parameters and large training sets of ...

I don't know for sure, but the large number of parameters and large training sets of the popular LLMs strongly suggests they train them past the interpolation threshold, which would mean they cannot be overfit.

If you trained an LLM with only demonstrably true statements, they would still output false statements. They have no representation of truth at the level of the sequences they emit.

"Hallucination" is therefore a human judgment about the human user's reaction to LLM output and is not reflective of any semantic content of that output. All LLM outputs, whether they appear to have semantic content or not and regardless of whether that aligns with the user's expectations, are thus hallucinations in a sense.