Yes, agriculture doesn't scale down well.

It's too much like manufacturing.

You need to at least acheive a size where some mechanization is worth investing in, and where you can specialize.

Or just abandon the goal of self-sufficiency and focus on creating beauty and resiliency/diversity.