I don't know about women. I'm not a women.

For men, It's only hard to keep muscle if you develop your muscles beyond what is normal. I see a lot of guys go crazy with their muscles and then yes they will lose it if they don't keep up on it.

You don't really need extra big muscles other then for show. You can live perfectly fine and stay fit and healthy as long as you don't eat like a pig without even working out.