Not supporting outbox in one of the most popular clients makes everyone else's lives harder, not just your users. Adding it will have a much bigger impact on nostr's success and user experience across the network than improving performance (damus is already really fast).