Too small indicates he doesn't see his wife for who she is. Too big means he held it up in a store, thought "yeah, that's about how big her ass/chest/thighs is/are" and insinuates that he thinks she's bigger than she is. Both can be triggering for someone with body issues.