yes it will degrade, only if you don’t separate the walls from the ground with stone or a barrier, where it will soak up all the rain and if there’s no roof overhang to stop the rain hitting the house, that could degrade it a bit, but i’ve seen a few mud homes that don’t have any overhang and they seem to be fine as the walls don’t retain the moisture

and if the lime or mud plaster is damaged, it can be replaced without much effort