A contact just told me that my old "LLMs generate nonsense code" blog post from 2 years ago is now very outdated with GPT5 because it's so awesome and so helpful. So I asked him to give it a test for me, and asked it my favorite test question based on a use-case I had myself recently:
> Without adding third-party dependencies, how can I compress a Data stream with zstd in Swift on an iPhone?
and here is the answer from ChatGPT 5: https://chatgpt.com/share/68968506-1834-8004-8390-d27f4a00f480
Very confident, very bold, even claims "Works on iOS 16+".
Problem with that: Just like any other LLM I've tested that provided similar responses, it is - excuse my language but I need to use it - absolute horseshit. No version of any Apple SDK ever supported or supports ZSTD (see https://developer.apple.com/documentation/compression/compression_algorithm for a real piece of knowledge). It was never there. Not even in private code. Not even as a mention of "things we might do in the future" on some developer event. It fundamentally does not exist. It's completely made up nonsense.
This concludes all the testing for GPT5 I have to do. If a tool is able to actively mislead me this easy, which potentially results in me wasting significant amounts of time in trying to make something work *that is guaranteed to never work*, it's a useless tool. I don't like collaborating with chronic liars who aren't able to openly point out knowledge gaps, so I'm also not interested in burning resources for a LLM that does the same.
Dennis Schubert
npub10v…axgw8
2025-08-08 23:23:47 UTC
Author Public Key
npub10v3kcyq0ey5tymuxew397qgqcvwqlyft5yw4ra6x3relq3jmaufqkaxgw8Seen on
wss://relay.momostr.pinkPublished at
2025-08-08 23:23:47 UTCEvent JSON
{
"id": "ee4c0dcb269fe11baec3dad1456981190a42e9ead2a4080279dadf5fe5042434",
"pubkey": "7b236c100fc928b26f86cba25f0100c31c0f912ba11d51f74688f3f0465bef12",
"created_at": 1754695427,
"kind": 1,
"tags": [
[
"proxy",
"https://mastodon.schub.social/@denschub/114995719518766451",
"web"
],
[
"proxy",
"https://mastodon.schub.social/users/denschub/statuses/114995719518766451",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mastodon.schub.social/users/denschub/statuses/114995719518766451",
"pink.momostr"
],
[
"-"
]
],
"content": "A contact just told me that my old \"LLMs generate nonsense code\" blog post from 2 years ago is now very outdated with GPT5 because it's so awesome and so helpful. So I asked him to give it a test for me, and asked it my favorite test question based on a use-case I had myself recently:\n\n\u003e Without adding third-party dependencies, how can I compress a Data stream with zstd in Swift on an iPhone?\n\nand here is the answer from ChatGPT 5: https://chatgpt.com/share/68968506-1834-8004-8390-d27f4a00f480\n\nVery confident, very bold, even claims \"Works on iOS 16+\".\n\nProblem with that: Just like any other LLM I've tested that provided similar responses, it is - excuse my language but I need to use it - absolute horseshit. No version of any Apple SDK ever supported or supports ZSTD (see https://developer.apple.com/documentation/compression/compression_algorithm for a real piece of knowledge). It was never there. Not even in private code. Not even as a mention of \"things we might do in the future\" on some developer event. It fundamentally does not exist. It's completely made up nonsense.\n\nThis concludes all the testing for GPT5 I have to do. If a tool is able to actively mislead me this easy, which potentially results in me wasting significant amounts of time in trying to make something work *that is guaranteed to never work*, it's a useless tool. I don't like collaborating with chronic liars who aren't able to openly point out knowledge gaps, so I'm also not interested in burning resources for a LLM that does the same.",
"sig": "79eb9a274d6b50acd147a202c03880833c78505b6b18bdef3398351c699f688ea85501d4ed7dbfe0cf4a55851ac197a15e2c8c53c4656091f100f834903c0271"
}