It would indeed be nice if accounts owned their replies collections in the same way that comments on blogs used to work. i.e. the blogger could moderate or remove comments as needed.

But Actually Existing #ActivityPub is the culmination of a lot of tradeoffs and battles that the contemporaneous tooters have never witnessed. Some things which seem to be unreasonable design choices to the newcomer are because of the events that went before. The exoduses, purges and brigading on the silo sites.

In the context of replies, with the traditional blog the commenter is someone with a browser and the blogger is the publisher fully in control of the site content. In the federated model both are publishers. So one cannot necessarily determine what version of events the other publishes. But things like object capabilities may help, and the pixelfed stuff which I've been adding support for also helps to communicate intentionality around replies and boosts.