Nope when you add a contact you also store the relay your contacts are posting to, your client automatically fetches posts from those relays, you yourself don't need to add relays if all you want to do is to read posts from your contacts.

They're more for posting your events, loading global feeds and a few other things.