I've grown some fungus to eat for while and learned a lot from it:

Mushrooms are the reproducing system of the fungus. They come out of the soil if the fungus has not enough space or if it is slightly dying; I would leave them there safely because if they are growing mushrooms and your flowers are ok, then they are likely to be giving proteins to your flowers and not really winning againt their roots.

Fungus that can be harmful to plants doesn't really show up in mushrooms, they likely would make leaf by leaf to die and get yellow or dry; or make the body of the tree to go black putrid and rotten. If you see any of this signs then you should squish some antifungus meds around your camps.