The number of people you follow won't affect whether your replies are seen as spam. Your follow list is only used to help decide if other people might be spammers.
For example, because I follow you, the people you follow are generally considered not spam. However, if you follow a huge number of people (the Dunbar threshold), then your follow list is no longer consider useful for this purpose.