Yes, agriculture doesn't scale down well.
It's too much like manufacturing.
You need to at least acheive a size where some mechanization is worth investing in, and where you can specialize.
Or just abandon the goal of self-sufficiency and focus on creating beauty and resiliency/diversity.