we verify the signature using other sources on the web, that it was signed by the key we expect. that's how we verify.
ultimately youre right though theres some level of trust like, did you manufacture each processor and modem/interface in your device yourself? and even in some bizarre world where you did, can you trust the machine which manufactured them
asking questions, verifying, is great. more of it!