Try them! When they're unripe they're very astringent, so in that case wait a few days and try them again.
We've also found that there's a lot of variation in the flavor between different trees. Sometimes even neighboring trees will taste very different.