I love autumn olive. I prefer to use opportunistic, invasive gives a negative connotation. I am noticing more insect damage to the leaves here, suggesting it is naturalizing. The only place I see it as a big 'problem' here is abandoned farmland.

I also like to plant trees and shrubs around autumn olives for shelter. You can prune them heavily to release nitrogen and allow more sun in.