You can at any time, even after accepting them, hide them at the badge site. As for slowing it down, I don't think it actually causes the rest of the app to load slower, even if a few badges load particularly slowly; the individual images for the badges are hosted on a specific relay. And sometimes the relay that happened to accept that badge image is now super slow.

Badges.page is the site url.