that really depends on the language you're speaking. In german, greek or russian, you've got 3 genders, masculine, feminine and neutral. In some languages you can have even more, like in Dyirbal (Australian Aboriginal language) where you have four genders: masculine, feminine, neuter, and vegetable.
English has no gramatical gender, but instead has "gender pronouns" (he, she and it). In my opinion, that's why there is such a confusion between gender and "sex" in that language.