interesting, I have all my devices synced and I don't get any nags even though I don't believe I gave Signal access to notifications nor the contact list.
Although I am using Molly, which is an independent build of Signal. Maybe check it out?
https://molly.im/
Signal (npub1rkw…5q62)