I had a client try exactly that about a year ago. He decided to do something else, but its all in the marketing, you could succeed.

I'd suggest you need a direct line to local people who are into gardening (FB? Instagram?), a good, flexible, low-cost system for packaging and delivery, and a lot of patience.