Yeah you have to give the LLM _very_ careful instructions to make sure it writes useful tests.

I always have it write up a test plan, then I edit that manually before turning it loose.