USB3 is known to be kind of dodgy requiring replugs sometimes. Sven Peter (npub1jzz…687j) is on it and it's a known bug. Type C stuff is... quite an adventure to fully support properly and reliably, because it requires getting 6+ kernel subsystems coordinating with each other correctly (Type C PD, PHY, USB, DP, TB, PCIe).
There's also the tangential problem of the U-Boot USB stack being rather... questionable, but that rather manifests as hangs on boot if you have an unlucky USB device connected.