If you own the book, I don't see what the issue with annotation is. While I don't do this often, I do if it's a book I am studying and might want to quote.
I also have books from my parents and grandparents that have been annotated, and I think that's wonderful!
Marginalia are a glorious window into the past, and I find it far less obstructive than post-its. It's only a problem if you do it to someone else's book/a library book.