Arguably the main source of the annoyance is actually Windows. Unlike every other OS (except Chrome OS), it doesn't have a standalone preference for high contrast. It's intrinsically tied to Contrast Themes (which are what provide colours for forced-colors mode).
forced-colors itself is also tricky to fully understand because it is in practice a Windows only feature.