Yeah most say start at your back door and move out, but I get the logic of starting with the trees.
Discussion
It isn't quite just about the trees, I was too zoomed in first advice. Use big rocks planning. Physical size of layout and highest return on investment items should be used to prioritize.
Easy to squeeze smaller plants in around trees but not the other way around. Schedule big projects by the seasons and let smaller projects fall in around them.
This link explains big rocks as a project and program management system.