Agreed about LLMs reinforcing errors. They aren't grounded the same way software like Mathematica are and can lead students astray.
For now I think they could be useful for people to explore ideas that lend well to verifiable feedback. For example, writing software that has a compiler or the user can manually inspect the output to see if it aligns with their vision. It opens the doors to try out things that we may otherwise forego because the expected payoff is too low.
