The problem is that all of your Android contacts already have you in their Google could. Algorithm can easily connect the dots and locate you by any bit of private information.
That being said, end to end encryption is done only while you’re transmitting that data somewhere, and by default, communication with Google servers is encrypted, as would be any other where you’re required to enter your credentials.
My point is that it’s easier to get your device physically and get the contacts out than it is for someone to hack your account and/or intercept your data. And even if you hide your stuff well, your personal data is still floating around dozens if not hundreds of phones.
What would be your angle on the issue?