They may be meat bees. Miserable.. half of them sting you and the other half just want to saw off a bite of your flesh. Generally I don’t mind wasps and hornets, but they don’t need to be IN the garden. Gardens should bring joy.
There are many pollinators. Some wasps pollinate. Other wasps don’t pollinate at all. The worst kind of wasps are the ones that rip the flower open to get the pollen before other pollinators. Those kinds use the pollen, but don’t pollinate because they destroy instead for create.