not sure exactly about the details but have you seen the new stuff that Home Assistant is putting out
Discussion
No, what’s that?
they are just now launching their "voice edition", with devices that do processing locally https://www.home-assistant.io/voice-pe/