"servers are reduced to dumb store-and-forward relays".
But since it is SMTP transport, hacked server can become to store-record_sender+recipient_pair-and-forward.
Look. Signal has server code open. How do I know that right now my client side connects to server running this code?
Hence, what client tells server becomes important. In SMTP client fills to and from fields, plus authentication to sending server.
Next step would be an option to use better transport.